# Integrate with Databricks

info
This feature is not enabled on accounts by default. Contact Technical Support or your Customer Success representative to enable it.

This guide explains how to integrate the Treasure AI-managed Glue HMS catalog and Iceberg tables with Databricks using Unity Catalog federation.

## Overview

After provisioning Iceberg catalog resources for your Treasure AI account, you can federate the Glue HMS catalog into Databricks Unity Catalog. This allows you to query Treasure AI-managed Iceberg tables directly from Databricks without copying data.

The integration uses:

- A **service credential** to access the Glue HMS catalog
- A **storage credential** to access the S3 bucket containing Iceberg table data
- A **foreign catalog** to federate the Treasure AI catalog into Unity Catalog


## Prerequisites

- Iceberg catalog resources provisioned and `active` for your account (see [Provision Resources](/ja/products/customer-data-platform/composable-publish/resource-provisioning))
- Unity Catalog enabled on your Databricks workspace and compute ([Databricks docs](https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspaces))


### Authentication

For authentication details and API endpoints by region, see [Use Iceberg Catalog Management API](/ja/products/customer-data-platform/composable-publish/use-iceberg-catalog-api).

### Required Information

Retrieve these values from the [Get Resource Status](/ja/products/customer-data-platform/composable-publish/resource-provisioning#get-resource-status) endpoint:

```sh
curl "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources" \
  -H "Authorization: TD1 <admin_api_key>" \
  -H "Accept: application/json"
```

| Response Field | Used As | Example |
|  --- | --- | --- |
| `aws_region` | AWS Region | `us-east-1` |
| `aws_account_id` | AWS Account ID | `123456789012` |
| `iam_role_arn` | IAM Role ARN | `arn:aws:iam::123456789012:role/zcpo-hms-tenant-us01_td10000` |
| `external_location_url` | External Location URL | `s3://zcp-us01-td10000-a1b2c3d4/iceberg` |
| `db_name` | Database Name | `td10000_us01_export` |


## Setup

### Step 1 (Databricks): Create a service credential and a storage credential

Create a service credential and a storage credential for `iam_role_arn`. Both credentials point to the same IAM role provided by Treasure AI.

- **Service credential** — used to access the Glue HMS catalog. Follow: [Create service credentials](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-services/service-credentials#create-service-credentials-2)
- **Storage credential** — used to access the S3 bucket. Follow: [Give Databricks the IAM role details](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#step-2-give-databricks-the-iam-role-details)


After creating the credentials, record the **External ID** displayed by Databricks. You will need it in the next step.

### Step 2 (Treasure AI): Update Trust Policy of IAM role for reading tables

Update the trust policy of the IAM role for reading tables.
This step corresponds to [this step in the Databricks documentation](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#step-3-update-the-iam-role-trust-relationship-policy).

Call the trust policy update endpoint with the External ID and the IAM role ARNs from Step 1:

```sh
curl -X POST "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources/reader_role/trust_policy" \
  -H "Authorization: TD1 <admin_api_key>" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "entries": [
      {
        "service": "databricks",
        "iam_principal_arns": [
          "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
          "<iam_role_arn>"
        ],
        "external_id": "<external_id_from_databricks>"
      }
    ]
  }'
```

The External ID secures the IAM role so it can only be assumed when the correct External ID is provided.
This ID is generated by Databricks and is only known after creating the credentials.

For `iam_principal_arns`, 2 IAM roles are required.

- Unity Catalog's IAM role
- This IAM role (i.e. `<iam_role_arn>`)
  - This is required for [assuming itself](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#self-assuming-role-enforcement-policy)


Important
The catalog federation will not work until this step is completed. Poll the [Get Resource Status](/ja/products/customer-data-platform/composable-publish/resource-provisioning#get-resource-status) endpoint and wait for the status to return to `active` before proceeding.

### Step 3 (Databricks): Create an external location for provided S3

Create an external location for `external_location_url` by following: [Create an external location for an S3 bucket](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#-create-an-external-location-for-an-aws-s3-bucket).

- Use the storage credential created in Step 1
- Databricks may warn about missing permissions (e.g., "Write" or "File Events Read"). You can ignore these warnings and force-create the external location — Treasure AI's reader role is read-only by design.


### Step 4 (Databricks): Create an external location for Databricks metadata

Create a separate storage credential and external location for Databricks to store its own metadata. Follow: [Create an external location for S3](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/).

This location is used by Databricks internally to read external Iceberg tables. It does **not** store your table data.

Note
This external location is different from the one in Step 3. You must create a separate S3 bucket and IAM role in your own AWS account (the account where Databricks is deployed).

### Step 5 (Databricks): Create a connection and a foreign catalog

Create a connection to Treasure AI's Glue HMS catalog and a foreign catalog to federate it into Unity Catalog. Follow: [Create the connection (Glue HMS federation)](https://docs.databricks.com/aws/en/query-federation/hms-federation-glue#create-the-connection).

**Connection parameters:**

| Parameter | Value |
|  --- | --- |
| AWS Region | `aws_region` from the Treasure AI resource response |
| AWS Account ID | `aws_account_id` from the Treasure AI resource response |
| Credentials | The service credential created in Step 1 |


**Foreign catalog parameters:**

| Parameter | Value |
|  --- | --- |
| Authorized paths | The external location created in Step 3 |
| Storage location | The external location created in Step 4 |


## Validation

After creating the foreign catalog, it appears in the Databricks Catalog Explorer.

## Troubleshooting

| Symptom | Possible Cause |
|  --- | --- |
| Access denied when querying | IAM role ARN or External ID mismatch — verify the value registered in Step 2 |
| External location creation warns about permissions | Expected — Treasure AI's reader role is read-only; force-create the location |


## Important Notes

- The trust policy update in Step 2 is a full replacement. If you need to add a second Databricks workspace later, include all existing entries along with the new one in the request.
- The reader IAM role is read-only. Databricks can query tables but cannot modify data in the Treasure AI-managed S3 bucket.
- Databases created via the [Database Management API](/ja/products/customer-data-platform/composable-publish/database-management) are automatically accessible through the foreign catalog without additional configuration.