{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":["admonition"]},"redocly_category":"Products","product_name":"Real-time","type":"markdown"},"seo":{"title":"Lookup Catalog Sync Workflow","description":"Learn how to configure and run the Treasure Workflow sync workflow that uploads cdp_lookup_catalog table data to RT 2.0 internal storage for use in RT 2.0 personalization.","siteUrl":"https://docs.treasure.ai","lang":"en-US","jsonLd":{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.treasure.ai/","name":"Treasure AI","url":"https://www.treasure.ai/","logo":"https://www.treasure.ai/hubfs/assets/images/logos/primary-logo.svg"},{"@type":"WebSite","@id":"https://docs.treasure.ai/#website","name":"Treasure AI Documentation","url":"https://docs.treasure.ai/","inLanguage":["en","ja"],"publisher":{"@id":"https://www.treasure.ai/"}}]},"llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"lookup-catalog-sync-workflow","__idx":0},"children":["Lookup Catalog Sync Workflow"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Lookup Catalog sync workflow reads from the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cdp_lookup_catalog"]}," database in Data Workbench, detects changed records using hash-based change tracking, and uploads only modified rows to RT 2.0's internal storage via the Bulk Load API."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Workflow files on GitHub"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The latest workflow files are available in the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/treasure-boxes/tree/master/realtime-box/lookup-catalog-sync"},"children":["treasure-data/treasure-boxes"]}," repository under ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["realtime-box/lookup-catalog-sync/"]},":"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["lookup_catalog_sync.dig"]}," — main workflow"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["scripts.py"]}," — generates type-aware JSON payload SQL per table"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["queries/discover_tables.sql"]}," — discovers eligible tables in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cdp_lookup_catalog"]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The workflow automatically discovers all eligible tables in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cdp_lookup_catalog"]}," on each run — there is no separate configuration step per table. An optional ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["p_table_name"]}," parameter can restrict a single run to one table, which is useful for testing."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"how-it-works","__idx":1},"children":["How It Works"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The workflow follows the same per-table sync pipeline for every discovered table:"]},{"$$mdtype":"Tag","name":"Diagram","attributes":{"data-language":"mermaid","diagramType":"mermaid","diagramSource":"flowchart TD\n A[Discover tables] --> B[For each table]\n B --> C[Extract changed records\\nvia hash comparison]\n C --> D{Changes exist?}\n D -->|Yes| E[Upload to RT 2.0\\nInternal Storage]\n D -->|No| F[Skip — no update]\n E --> G[Update digest table]\n G --> H[Cleanup temp tables]\n","diagramHtml":"

Yes

No

Discover tables

For each table

Extract changed records
via hash comparison

Changes exist?

Upload to RT 2.0
Internal Storage

Skip — no update

Update digest table

Cleanup temp tables

","diagramHtmlDark":"

Yes

No

Discover tables

For each table

Extract changed records
via hash comparison

Changes exist?

Upload to RT 2.0
Internal Storage

Skip — no update

Update digest table

Cleanup temp tables

"},"children":["flowchart TD\n A[Discover tables] --> B[For each table]\n B --> C[Extract changed records\\nvia hash comparison]\n C --> D{Changes exist?}\n D -->|Yes| E[Upload to RT 2.0\\nInternal Storage]\n D -->|No| F[Skip — no update]\n E --> G[Update digest table]\n G --> H[Cleanup temp tables]\n"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Change detection"]},": Each table has a companion digest table (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_{table_name}_digests"]},") that stores a hash of each row's JSON payload. On each run, only rows whose hash has changed (or that are new) are uploaded."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Internal table naming"]},": All workflow-internal and temporary tables use a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_"]}," prefix to distinguish them from source tables. The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["discover_tables.sql"]}," query automatically excludes ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_*"]}," tables from sync."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"configuration-parameters","__idx":2},"children":["Configuration Parameters"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"30%","data-label":"Parameter"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Parameter"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"20%","data-label":"Default"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Default"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"50%","data-label":"Description"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Description"]}," "]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_importer_endpoint"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["(required)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Region-specific Bulk Load API endpoint. See the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"#region-specific-endpoints"},"children":["region-specific endpoints"]}," table below."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_instance"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["(required)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Instance identifier provisioned by Treasure AI when RT 2.0 is set up for your account (format: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["a{accountId}n{instanceId}"]},"). Contact your Customer Success Manager or Treasure AI Support to obtain this value."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td.database"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cdp_lookup_catalog"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Source database containing the lookup tables."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["p_table_name"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["\"\""]}," (all tables)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["When set, syncs only the specified table. Leave empty to sync all discovered tables."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["batch_size"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["1000"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Number of records per upload request to the Bulk Load API."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["parallelism"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["10"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Number of concurrent upload threads per table."]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"region-specific-endpoints","__idx":3},"children":["Region-Specific Endpoints"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"20%","data-label":"Region"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Region"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"80%","data-label":"reactor_importer_endpoint"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_importer_endpoint"]}]}," "]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["US"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["https://bulk-storage-importer-api-production-aws-us-east-1.internal.treasuredata.com"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["EU (eu01)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["https://bulk-storage-importer-api-production-eu01-eu-central-1.internal.treasuredata.com"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Tokyo"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["https://bulk-storage-importer-api-production-aws-tokyo-ap-northeast-1.internal.treasuredata.com"]}]}]}]}]}]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Internal endpoint access"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":[".internal.treasuredata.com"]}," endpoints are only reachable from within the Treasure Data platform (for example, from a TD Workflow task) and cannot be accessed from the public internet."]}]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"warning","name":"No instance-level data isolation"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_instance"]}," parameter routes the upload request but does not restrict which instances can read the data. All instances within the account share the same lookup catalog storage — any Lookup Catalog Attribute in any instance can read any table uploaded under the account, regardless of which ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_instance"]}," value was used to upload it. If your account operates multiple RT 2.0 instances that must not share catalog data, do not rely on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_instance"]}," for isolation; use separate accounts or contact Treasure AI Support."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"setting-up-the-workflow","__idx":4},"children":["Setting Up the Workflow"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["In Data Workbench, create a new Workflow project (or use an existing one)."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Copy the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["lookup-catalog-sync/"]}," directory from ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/treasure-boxes/tree/master/realtime-box/lookup-catalog-sync"},"children":["realtime-box/lookup-catalog-sync/"]}," in the treasure-boxes repository and upload the files (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["lookup_catalog_sync.dig"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["scripts.py"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["queries/discover_tables.sql"]},") into the project."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Edit ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["lookup_catalog_sync.dig"]}," and set ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_importer_endpoint"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reactor_instance"]},", and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td.database"]}," in the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_export"]}," block. Optionally set ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["p_table_name"]}," to restrict a run to a single table for testing."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Store your TD API key in the project's ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Secrets"]}," tab as ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td.apikey"]},". Use a key scoped to the minimum required permission (write access for the Bulk Load API) rather than a full-access key."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Set a schedule (daily is recommended) and run the workflow manually once to verify the initial full upload succeeds."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The discovery query automatically finds all base tables in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cdp_lookup_catalog"]}," and excludes ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_*"]}," internal tables."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Parallelism guidance"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Start with the default settings (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["batch_size: 1000"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["parallelism: 10"]},") for the initial full upload. Increase gradually during incremental runs while monitoring for throttling errors. Larger tables or higher parallelism amplify the risk of throttling."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"scheduling-recommendations","__idx":5},"children":["Scheduling Recommendations"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"35%","data-label":"Data type"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data type"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"65%","data-label":"Recommended schedule"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recommended schedule"]}," "]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Flash sale / time-sensitive pricing"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Every 1–2 hours (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0 */1 * * *"]},")"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Daily promotions / coupon tables"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Every 6 hours (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0 */6 * * *"]},")"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Product catalog (stable data)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Daily (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0 2 * * *"]},")"]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"troubleshooting","__idx":6},"children":["Troubleshooting"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"35%","data-label":"Symptom"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Symptom"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"65%","data-label":"Cause and fix"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Cause and fix"]}," "]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["All records upload on every run"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The digest table does not exist or has stale hashes. Drop ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_{table_name}_digests"]}," and re-run to rebuild from scratch."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["400 error: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["\"Expected string, received...\""]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["A row has a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["NULL"]}," or empty primary key. Filter out or fix null-key rows in the source table before the next run."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["400 error: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["\"Missing required header: x-lookup-table-name\""]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["x-lookup-table-name"]}," header is missing from the upload step. Verify the workflow template headers are configured correctly."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Workflow fails with ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["varchar = bigint"]}," type error"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The source table column type was changed to an incompatible type (for example, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["string"]}," → ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["int"]},"). Drop the digest table, fix the source data, and re-run."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Schema mismatch after changing the primary key column"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Drop the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_{table_name}_digests"]}," table, then re-run. The workflow will re-upload all rows with the new key. Previously uploaded data under the old key is not automatically deleted."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Removed column still appears in responses"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["RT 2.0 stores data as uploaded — removing a column from the source table does not delete it from internal storage. A full re-upload is required to purge the residual data: drop the digest table (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_wf_{table_name}_digests"]},") and re-run the workflow. Until then, the deleted column's data remains accessible via the API."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Upload timeout"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Reduce ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["parallelism"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["batch_size"]},", especially during the initial full upload of a large table."]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"key-column-constraints","__idx":7},"children":["Key Column Constraints"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The workflow detects the primary key as the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["first non-",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["time"]}," column"]}," in the table schema (by ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ordinal_position"]}," in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["information_schema.columns"]},") — if the table's first column is named ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["time"]},", the second column is used as the primary key instead."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"35%","data-label":"Constraint"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Constraint"]}," "]},{"$$mdtype":"Tag","name":"th","attributes":{"width":"65%","data-label":"Detail"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Detail"]}," "]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Column position"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The key column must be the first non-",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["time"]}," column in the table schema."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Column name"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Must not be named ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["time"]}," (reserved by Treasure AI platform)."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Null values"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Key column must not contain ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["NULL"]}," or empty values. Upload fails with 400 if null keys are present."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Supported types"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["string"]},". Other types are not supported for the primary key column."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Uniqueness"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Data must be deduplicated. The workflow does not deduplicate rows."]}]}]}]}]}]},"headings":[{"value":"Lookup Catalog Sync Workflow","id":"lookup-catalog-sync-workflow","depth":1},{"value":"How It Works","id":"how-it-works","depth":2},{"value":"Configuration Parameters","id":"configuration-parameters","depth":2},{"value":"Region-Specific Endpoints","id":"region-specific-endpoints","depth":3},{"value":"Setting Up the Workflow","id":"setting-up-the-workflow","depth":2},{"value":"Scheduling Recommendations","id":"scheduling-recommendations","depth":2},{"value":"Troubleshooting","id":"troubleshooting","depth":2},{"value":"Key Column Constraints","id":"key-column-constraints","depth":2}],"frontmatter":{"seo":{"title":"Lookup Catalog Sync Workflow","description":"Learn how to configure and run the Treasure Workflow sync workflow that uploads cdp_lookup_catalog table data to RT 2.0 internal storage for use in RT 2.0 personalization."}},"lastModified":"2026-07-10T23:37:04.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/products/customer-data-platform/real-time/lookup-catalog-workflow","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}