BigQuery

🚧

In order to send datasets to BigQuery, a BigQuery Dataset must already be set up in your Google Workspace.

Connecting to the BigQuery App

In your DDx API Console account:

1. Click through “View & Manage Datasets”.
2. Select the “Apps & Workspace ID” tab.
3. Select “+ Connect a New App”.
4. Select “BigQuery” from the App options.
5. A service account will be generated for you. This service account is the account the Dataset Exchange API (DXAPI) will use to load datasets into tables in your BigQuery Dataset.

Review each steps with visual aids below!

1. In your Google Cloud Console, open IAM & Admin > IAM.

2. Add the generated service account to your project with the role “BigQuery Job User”. The service account must be given the role BigQuery Job User at your GCP project level, not the BigQuery dataset level.

   
   

3. Go to BigQuery, on your Dataset click Share > Manage Permissions > Add Principal.

4. Add the service account as a principal with role “BigQuery Data Editor”

The service account must be given the role “BigQuery Data Editor” at the BigQuery dataset level, not at the GCP project level to practice the principle of least privilege.

1. Return to your DDx API Console account and click "Continue".
2. Submit the GCP Project ID, your BigQuery Dataset name and the location of your Dataset. If you did not set a location when you created your Dataset the default location is “Multi-Region US".

12. Click 'Test Connection'.

    • DXAPI will run a test job to verify that it has the correct permissions to create tables on the Dataset before enabling delivery. No records will be loaded during the test.

13. Upon a successful tested connection click 'Finish Setup'.

14. Done!

Once you send datasets to BigQuery the tables in your BigQuery Dataset will be named after your DXAPI Dataset name.

Troubleshooting

If the test results are unsuccessful, confirm the BigQuery Dataset details are correct and that the service account has been given the correct IAM permissions as detailed above.

• Make sure there are no trailing whitespaces on the Dataset Name or Project ID.
• Confirm the selected location is correct.
• Confirm the generated service account has been given the BigQuery Job User and BigQuery Data Editor roles and that they have been saved. It can sometimes take a couple minutes to propagate.

If errors continue please reach out to [email protected] for further assistance!

You can leave and finish the set up any time; a ‘Pending Setup’ message will show on the app and datasets will not be sent to BigQuery until the connection has been successfully tested.