Integrate with BigQuery

This guide explains how to integrate the Treasure AI-managed Glue HMS catalog and Iceberg tables with Google BigQuery using BigQuery Omni and an AWS Glue federated dataset.

After provisioning Iceberg catalog resources for your Treasure AI account, you can expose the Glue HMS catalog to BigQuery as a federated dataset. This allows you to query Treasure AI-managed Iceberg tables directly from BigQuery without copying data.

The integration uses:

• A BigQuery Omni AWS connection to access the Glue HMS catalog and the S3 bucket via the Treasure AI reader IAM role
• A federated dataset that exposes the Treasure AI Glue database and its Iceberg tables in BigQuery

Unlike an IAM-role-based service, BigQuery authenticates to AWS with a BigQuery Google identity using AWS web identity federation (sts:AssumeRoleWithWebIdentity). Because of this, a single AWS connection provides both Glue catalog and S3 read access — you do not register separate service or storage credentials.

• Iceberg catalog resources provisioned and active for your account (see Provision Resources)
• A Google Cloud project with BigQuery enabled, in a region where BigQuery Omni supports the Treasure AI aws_region

For authentication details and API endpoints by region, see Use Iceberg Catalog Management API.

Retrieve these values from the Get Resource Status endpoint:

curl "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources" \
  -H "Authorization: TD1 <admin_api_key>" \
  -H "Accept: application/json"

The BigQuery Omni location is derived from aws_region by prefixing aws- — for example, us-east-1 becomes aws-us-east-1. Use this same location for both the connection and the federated dataset.

Create a BigLake on AWS (via BigQuery Omni) connection that uses the Treasure AI reader IAM role. Follow the Create connections section in Connect to Amazon S3.

• For AWS role ID, use the iam_role_arn from the Treasure AI resource response.
• Set the connection location to the BigQuery Omni location that matches aws_region (for example, aws-us-east-1).

After creating the connection, open its Connection info and record the BigQuery Google identity. You will use it in the next step.

Update the trust policy of the IAM role for reading tables so BigQuery can assume it via web identity federation. This step corresponds to Add a trust relationship to the AWS role in the BigQuery documentation.

Call the trust policy update endpoint with service set to bigquery and the BigQuery Google identity from Step 1 as external_ids:

curl -X POST "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources/reader_role/trust_policy" \
  -H "Authorization: TD1 <admin_api_key>" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "entries": [
      {
        "service": "bigquery",
        "external_ids": ["<bigquery_google_identity>"]
      }
    ]
  }'

The external_ids value is matched against the accounts.google.com:sub condition in the IAM role trust policy, so only the specified BigQuery connection can assume the role. This value is generated by BigQuery and is only known after creating the connection.

For BigQuery, iam_principal_arns is not used and must be omitted — BigQuery assumes the role through the fixed accounts.google.com federated principal, not an IAM role ARN.

The update replaces the entire trust policy. If the reader role already trusts other integrations, append the BigQuery entry to the existing policy instead of posting only the entry above — see Manage the Reader Role Trust Policy.

Create a federated dataset that points at the Treasure AI Glue database. Follow: Create a federated dataset for AWS Glue.

Run the following in BigQuery, substituting the values from the Treasure AI resource response:

CREATE EXTERNAL SCHEMA my_dataset
WITH CONNECTION `<project_id>.aws-<aws_region>.<connection_name>`
  OPTIONS (
    external_source = 'aws-glue://arn:aws:glue:<aws_region>:<aws_account_id>:database/<db_name>',
    location = 'aws-<aws_region>');

• Use the connection created in Step 1 for the WITH CONNECTION clause.
• The dataset location must match the connection location (for example, aws-us-east-1) and cannot be changed after creation.
• Treasure AI's reader role already grants the Glue (glue:GetDatabase, glue:GetTable, glue:GetTables, glue:GetPartitions) and S3 read permissions that the federated dataset requires. No additional AWS permissions are needed.

After creating the federated dataset, it appears in the BigQuery Explorer, and the Iceberg tables from the Treasure AI Glue database are exposed as tables you can query.

• The trust policy update in Step 2 is a full replacement. To add another integration later without dropping BigQuery, use the read-modify-write flow in Manage the Reader Role Trust Policy.
• The reader IAM role is read-only. BigQuery can query tables but cannot modify data in the Treasure AI-managed S3 bucket.
• Databases created via the Database Management API are automatically accessible as federated datasets without additional configuration.