Composable Publish Tutorial

Composable Publish shares tables managed in Treasure AI with your cloud data warehouse (CDW) without copying data. You publish tables in Apache Iceberg format to an export database, and the warehouse queries them in place through the Iceberg catalog — no export jobs, no transfer delays, no second copy to govern. This tutorial walks the complete path using Databricks as the warehouse: a one-time setup workflow performed by admins, followed by the recurring publish workflow performed by a data engineer.

Each Treasure AI account gets a dedicated set of AWS resources: an S3 bucket that stores Iceberg table data, a Glue database (the default export database) that holds table metadata, and a read-only IAM role that external warehouses assume to read the data. Data engineers write Iceberg tables from Data Workbench using Trino. The warehouse connects to the catalog by assuming the reader role and reads table data directly from S3.

The tutorial covers two workflows:

• The Composable Publish feature is enabled on your account
• A Treasure AI admin API key for Workflow 1; the publishing user's API key for Workflow 2. Write-only API keys are rejected. See Use Iceberg Catalog Management API for authentication and the endpoint for your site — examples below use the US endpoint
• Unity Catalog enabled on your Databricks workspace and compute (Databricks docs), with privileges to create credentials, external locations, connections, and catalogs
• Familiarity with running Trino queries in Data Workbench (see Data Workbench Trino Quickstart)

A Treasure AI admin provisions the account's resources and grants access, then works with the warehouse admin to connect Databricks. The workflow ends by reading a sample table from Databricks, so the integration is validated before any real data is published.

Provisioning creates the S3 bucket, the default export database, and the reader IAM role. It is a one-time, asynchronous operation.

1. Start provisioning:

   curl -X POST "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources" \
     -H "Authorization: TD1 <admin_api_key>" \
     -H "Accept: application/json"

2. Poll until status becomes active (typically 1–2 minutes):

   curl "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources" \
     -H "Authorization: TD1 <admin_api_key>" \
     -H "Accept: application/json"

3. From the response, record aws_region, aws_account_id, iam_role_arn, external_location_url, and db_name (for example td10000_us01_export) — Step 4 uses all five values.

If the status becomes failed, retry the POST request. See Provision Resources for status semantics and error responses.

Users have no access to the Iceberg catalog until an admin grants it. Grant the data engineer who will publish tables FULL access to the export database:

curl -X PUT "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/permissions" \
  -H "Authorization: TD1 <admin_api_key>" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": 12345,
    "permissions": [
      {
        "resource_type": "DATABASE",
        "resource_names": ["td10000_us01_export"],
        "operation": "FULL"
      }
    ]
  }'

Replace 12345 with the user's ID (find it with the get user list API) and td10000_us01_export with your db_name. To grant your own user — for example, to run Step 3 yourself — omit user_id.

The endpoint replaces the user's entire permission list, and finer-grained operations (READ, WRITE) are available. See Database-Level Access Control for Iceberg Catalog.

A table must exist before the Databricks connection can be verified end to end. As a user granted access in Step 2, run a Trino query in Data Workbench:

CREATE TABLE iceberg.td10000_us01_export.sample_products AS
SELECT * FROM (VALUES (1, 'apple'), (2, 'banana')) AS t(id, name);

Confirm the table is readable:

SELECT * FROM iceberg.td10000_us01_export.sample_products;

The query returns 2 rows. See Reading and Writing Iceberg Tables from Data Workbench for the table naming convention and supported SQL commands.

The warehouse admin federates the Treasure AI catalog into Unity Catalog, and the Treasure AI admin allows Databricks to assume the reader role. The full procedure with links to the Databricks documentation for each sub-step is in Integrate with Databricks; the essential steps are:

1. In Databricks, create a service credential and a storage credential, both pointing at the iam_role_arn recorded in Step 1. Record the External ID that Databricks displays.

2. In Treasure AI, register Databricks in the reader role trust policy, with the Unity Catalog IAM role, the reader role itself (required for self-assumption), and the External ID from the previous step:

   curl -X POST "https://api-iceberg-mng.us01.treasuredata.com/v1/iceberg/catalog/resources/reader_role/trust_policy" \
     -H "Authorization: TD1 <admin_api_key>" \
     -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -d '{
       "entries": [
         {
           "service": "databricks",
           "iam_principal_arns": [
             "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
             "<iam_role_arn>"
           ],
           "external_ids": ["<external_id_from_databricks>"]
         }
       ]
     }'

   The update applies asynchronously — poll the resource status endpoint (Step 1) and wait for status to return to active.

   Important

   The update replaces the entire trust policy. If your account already has an integration (for example Snowflake), include its existing entry in the request, or it loses access. See Manage the Reader Role Trust Policy.

3. In Databricks, create an external location for the external_location_url recorded in Step 1, using the storage credential from the first sub-step. Databricks may warn about missing "Write" or "File Events Read" permissions — the reader role is read-only by design, so force-create the location.

4. In Databricks, create a second storage credential and external location in your own AWS account for Databricks to store its metadata. This location is internal to Databricks and does not hold your table data.

5. In Databricks, create a connection to the Treasure AI catalog (using aws_region, aws_account_id, and the service credential) and a foreign catalog — this tutorial names it treasure_ai — with the location from sub-step 3 as its authorized path and the location from sub-step 4 as its storage location.

The foreign catalog appears in the Databricks Catalog Explorer with the export database inside it.

In a Databricks notebook or the SQL editor, query the sample table through the foreign catalog:

SELECT * FROM treasure_ai.td10000_us01_export.sample_products;

The query returns the 2 rows created in Step 3. The integration is now validated end to end, and setup is complete. You can drop the sample table from Data Workbench (DROP TABLE iceberg.td10000_us01_export.sample_products) once real tables are published.

A data engineer with WRITE or FULL access to the export database publishes data by writing Iceberg tables to it. Anything the warehouse should see — a refreshed segment, a new dataset — is one Trino query away; the Databricks side needs no changes.

Copy data from a Data Workbench table into the export database with CREATE TABLE AS SELECT:

CREATE TABLE iceberg.td10000_us01_export.customer_segments AS
SELECT * FROM my_database.customer_segments;

New tables in the export database appear in the foreign catalog automatically — no reconfiguration on the Databricks side:

SELECT * FROM treasure_ai.td10000_us01_export.customer_segments LIMIT 100;

Databricks reads the table's current state on every query, so refreshing the data in Treasure AI is all that is needed. Choose a refresh pattern:

• Append new rows incrementally, for example the last day of events:

  INSERT INTO iceberg.td10000_us01_export.customer_segments
  SELECT * FROM my_database.customer_segments
  WHERE td_interval(time, '-1d');

• Rewrite changed rows with UPDATE, DELETE, or MERGE, then compact the table (requires FULL permission):

  ALTER TABLE iceberg.td10000_us01_export.customer_segments EXECUTE optimize;

  Trino writes row-level changes as Iceberg position delete files, which Databricks does not support — until the table is optimized, it may not be readable from Databricks. See the DML compatibility warning.

To refresh on a recurring basis, run the query as a scheduled query in Data Workbench.

To stop publishing a dataset, drop its table (requires FULL permission):

DROP TABLE iceberg.td10000_us01_export.customer_segments;

The table and its data are deleted, and it disappears from the foreign catalog.

• Organize published data into multiple databases with the Database Management API — new databases are accessible through the foreign catalog without additional configuration
• Fine-tune who can read and write each database with Database-Level Access Control for Iceberg Catalog
• Connect additional warehouses: Snowflake or BigQuery, adding their trust policy entries with the read-modify-write flow in Manage the Reader Role Trust Policy
• Review the full SQL command support and restrictions in Reading and Writing Iceberg Tables from Data Workbench