BigQuery is a server-less data warehouse provided by Google that allows you to ingest, store, analyze and visualize large amounts of data, quickly and easily, at a low cost.
The CloudM / BigQuery integration allows you to send auditable logs from CloudM Automate to BigQuery.
In order to set up the integration, you will need to:
- Select a Project within the Google Cloud Console,
- Enable the BigQuery API for the Project,
- Create a Service Account and IAM policy so that CloudM can access BigQuery. We do that using a JSON file as authentication.
Enable the BigQuery API in the Google Cloud Console
- Sign into Google Cloud Console (as an admin),
- Select the relevant Project (using the drop down arrow next to the currently selected project name),
- Use the menu to go to APIs and Services > Library,
- Search for BigQuery API,
- Select the Enable option.
- If the API is already enabled, the option will say Automate instead and you will see a tick icon confirming the API is enabled.
Create a Service Account
- In the Google Cloud Console, and with the same project that you previously enable the API for selected, use the menu to go to IAM & admin > Service Accounts,
- Click on + Create Service Account,
- Give the Service Account a unique name
- Copy the Service Account ID (it will appear like an email address).
- Select Create and Continue.
- Skip the two optional sections and select Done.
- The Account will appear on the Service Account screen.
- Click on the Service Account name.
- Select the Keys tab
- Select Add Key > Create New Key
- Select JSON
- Select Create
- Save the JSON file to your desktop.
- Go to IAM & admin > IAM,
- Select ADD,
- Paste the Service Account ID in the New Principals field,
- Add BigQuery Admin as the Role (Project Browser role must also be set, if not already enabled)
- Select Save.
How to get the Project Id
To obtain the Project ID (that you will need to configure BigQuery in CloudM Automate), navigate to your Google Cloud Console's Automate Resources List.
Then, choose the Project ID of the project where your BigQuery is located.
Configure BigQuery in CloudM Automate
- As an admin (with permissions to see BigQuery - Edit Global Settings), select Settings > BigQuery
- Select the Enable button,
- Add the BigQuery Project ID
- Upload the JSON Key - This will populate the Service Account field.
- Select Test Connection
- Select Populate. CloudM will populate the data sets from BigQuery
- Select the Default Dataset that you want to export to from the drop down menu.
- Select Update
- Go to Logs > BigQuery Export
- Use the Default Dataset drop down to choose which dataset the logs will be exported to.
- If Force Overwrite is checked, then the table will be truncated and the whole audit log will be exported. Otherwise, only the recent log events will be appended to those already exported.
Select Export to export the logs to BigQuery.
- Please note that this is a manual process only. You will need to repeat this process periodically to keep the exported results up to date.