{"templateId":"markdown","sharedDataIds":{},"props":{"metadata":{"markdoc":{"tagList":["blog-author","blog-share"]},"redocly_category":"Developer Blog","type":"markdown"},"seo":{"title":"Orchestrate dbt with Treasure Workflow","description":"A walkthrough of how to leverage dbt Core (command line tools) with the Treasure Data ecosystem.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]},"image":"/assets/orchestrate.164d7a459eb8a7827525e3b96a4bb6c1dbacfe8a5f10386732a635b267827ce6.64842105.png"},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"orchestrate-dbt-with-treasure-workflow","__idx":0},"children":["Orchestrate dbt with Treasure Workflow"]},{"$$mdtype":"Tag","name":"BlogAuthor","attributes":{"name":"KuoHuei (Ansel) Lin","date":"2023-11-01","image":"/assets/ansel.a7244bc7c9da9a2523451933bcab41cd9c9db627d88393970bb8e0a6e042fd1d.978384e4.jpeg"},"children":[]},{"$$mdtype":"Tag","name":"BlogShare","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In this post we are going to focus on dbt Core’s components integration with Treasure Data’s solutions."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"what-is-dbt","__idx":1},"children":["What is dbt?"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.getdbt.com/product/what-is-dbt"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["dbt"]}]}," (data build tool) is an open-source analytics engineering tool primarily focused on data transformation within data engineering systems. It allows users to define, execute, and manage data transformations in a structured and repeatable manner. By leveraging SQL-based transformations and a version-controlled workflow, dbt-Core facilitates the organization and maintenance of complex data transformation processes, enabling more efficient analytics and reporting pipelines. It's widely used to prepare data for analytics, modeling, and visualization, making it a crucial tool in modern data pipelines."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Apart from ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["dbt Cloud"]},", which is a SaaS for Data Ops, ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["dbt Core"]}," is an open source command line tool and library that enables data teams to transform data using analytics engineering best practices."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["More references:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.getdbt.com/product/what-is-dbt"},"children":["What is dbt?"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.getdbt.com/blog/what-exactly-is-dbt"},"children":["What, exactly, is dbt?"]}]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["dbt Core"]},", as a CLI tool, with a bunch of SQL files, configuration files for defining the transformation, requires some external components:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Databases / Data Warehouses"]},", where the data transformation are executed and stored."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Examples include Snowflake, BigQuery, Redshift, and others, allowing you to directly query and transform data stored in these platforms."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Here, we are going to use Treasure Data (TD)’s ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/data-workbench/queries/trino/quickstart"},"children":["Presto Query Engine"]}," as demonstration."]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Orchestration Tools"]},", where the dbt CLI is executed and scheduled."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Some common orchestration tools are Apache Airflow, Prefect, or Dagster."]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here, we are going to use Treasure Data (TD)’s ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench/workflows"},"children":["Treasure Workflow"]}," as demonstration."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Besides, there are still some optional but common external components, like:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Version Control Systems"]},", where the configuration and definition of transformation can be versioned, collaborated, and shared as package importing."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data Catalogs and Metadata Management"]},", where the data assets and transformations can be observed, searched, traversed."]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image1.21a4f6e4b472160a30b286b07d08caa767e0536fc8165eb6dacb57e9cb0d4b14.64842105.png","alt":"Minimum architecture for dbt-core & external components"},"children":[]}," ","<center>Minimum architecture for dbt-core & external components"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"prerequisites","__idx":2},"children":["Prerequisites"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"recommended-background-knowledge","__idx":3},"children":["Recommended Background Knowledge"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/data-workbench/queries/trino/writing_trino_queries"},"children":["SQL Query Syntax"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench"},"children":["Treasure Data Workbench"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench/workflows"},"children":["Treasure Workflow"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://courses.getdbt.com/courses/fundamentals"},"children":["dbt Fundamentals"]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"software-resuirements","__idx":4},"children":["Software Resuirements"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/tools/cli-and-sdks/quickstart"},"children":["TD Toolbelt installation"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/tools/cli-and-sdks/quickstart#set-api-key"},"children":["API Key setup"]}]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.digdag.io/getting_started.html#downloading-the-latest-version"},"children":["digdag installation"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["digdag (0.10.5)"]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/docs/core/pip-install"},"children":["dbt installation (with pip)"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["dbt-core (1.6.6)"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["dbt-trino (1.6.2)"]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["(The specified versions above are for reference and verified in following examples.)"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"leverage-treasure-data-presto-query-engine-for-data-warehousing","__idx":5},"children":["Leverage Treasure Data Presto Query Engine for Data Warehousing"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"setup--init","__idx":6},"children":["Setup & ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["init"]}]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Create a ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles#about-the-profilesyml-file"},"children":["connection profile"]}," and put into ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_profiles/profiles.yml"]},".",{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Filling your Treasure API Key into ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["user"]},"."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Decide a default TD database name into ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["schema"]},"."]}]}]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"dbt_project:\n  target: td\n  outputs:\n    td:\n      type: trino\n      method: none\n      user: XXXXXX # TD_API_KEY\n      database: td-presto # TD presto schema name, not changable\n      host: api-presto.treasuredata.com # TD presto endpoint\n      port: 443\n      schema: dbt_example_db # TD database name\n      threads: 2\n      http_scheme: https\n      prepared_statements_enabled: false\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Create the TD database if it has not been created yet. In this example we are calling the database ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_example_db"]},"."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ td db:create dbt_example_db\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Initiate a dbt project called ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_project"]},"."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ dbt init dbt_project --profile dbt_profiles/profiles.yml\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["The folder structure should look like the following:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sh","header":{"controls":{"copy":{}}},"source":"├── dbt_profiles\n│   └── profiles.yml\n└── dbt_project\n    ├── README.md\n    ├── analyses\n    ├── dbt_project.yml\n    ├── macros\n    ├── models\n    │   └── example\n    │       ├── my_first_dbt_model.sql\n    │       ├── my_second_dbt_model.sql\n    │       └── schema.yml\n    ├── seeds\n    ├── snapshots\n    └── tests\n\n","lang":"sh"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"tuning-tips","__idx":7},"children":["Tuning Tips"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"quoting","__idx":8},"children":["Quoting"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Add the following configuration to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_project/dbt_project.yml"]},", since TD’s schema name ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td-presto"]}," needs be quoted, otherwise, the syntax error will be reported."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"quoting:\n  database: true\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"disable-view--materialized-view","__idx":9},"children":["Disable View & Materialized view"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Treasure Data Presto Query Engine does not support view or materialized view."," ","If you try to use unsupported views the following error displays."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["TrinoUserError: Table 'system.metadata.materialized_views' does not exist"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To prevent this error, try the following tips:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Avoid using ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/reference/resource-configs/trino-configs#view"},"children":["View"]}," and ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/reference/resource-configs/trino-configs#materialized-view"},"children":["Materialized view"]}," as model’s ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["materialized"]}," configurations."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Only use following options for ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/docs/build/materializations#materializations"},"children":["materialized configurations"]},":",{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["table"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["incremental"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ephemeral"]}]}]}]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"models:\n  dbt_project:\n    # Config indicated by + and applies to all files under models/example/\n    example:\n      +materialized: table\n      # +materialized: view ## Don't use view in any cases\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Adapter patch to bypass fetching view related catalog.",{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Create a macro file ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_project/macros/td_adapter_patch/adapters.sql"]},"."]}]}]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql{%","header":{"controls":{"copy":{}}},"source":"-- Bypass `materialized_views` to be compatible with TD Presto\n-- Related to https://github.com/starburstdata/dbt-trino/issues/298\n{% macro trino__list_relations_without_caching(relation) %}\n  {% call statement('list_relations_without_caching', fetch_result=True) -%}\n    select\n      table_catalog as database,\n      table_name as name,\n      table_schema as schema,\n      'table' as table_type\n    from {{ relation.information_schema() }}.tables\n    where table_schema = '{{ relation.schema | lower }}'\n  {% endcall %}\n  {{ return(load_result('list_relations_without_caching').table) }}\n{% endmacro %}\n","lang":"sql{%"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"first-run","__idx":10},"children":["First ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["run"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Once all the configurations are complete, your first ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt run"]}," should look  like this:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ dbt run --profiles-dir dbt_profiles/ --project-dir dbt_project\n\n  19:34:57  Running with dbt=1.6.6\n  19:34:57  Registered adapter: trino=1.6.2\n  19:34:57  Unable to do partial parsing because a project config has changed\n  19:34:57  Found 2 models, 4 tests, 0 sources, 0 exposures, 0 metrics, 372 macros, 0 groups, 0 semantic models\n  19:34:57\n  19:35:01  Concurrency: 2 threads (target='td')\n  19:35:01\n  19:35:01  1 of 2 START sql table model dbt_example_db.my_first_dbt_model ........ [RUN]\n  19:35:06  1 of 2 OK created sql table model dbt_example_db.my_first_dbt_model ... [SUCCESS in 4.81s]\n  19:35:06  2 of 2 START sql table model dbt_example_db.my_second_dbt_model ....... [RUN]\n  19:35:09  2 of 2 OK created sql table model dbt_example_db.my_second_dbt_model .. [SUCCESS in 3.80s]\n  19:35:09\n  19:35:09  Finished running 2 table models in 0 hours 0 minutes and 11.92 seconds (11.92s).\n  19:35:09\n  19:35:09  Completed successfully\n  19:35:09\n  19:35:09  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The two executed models should show up in the Treasure Data workbench as tables."," ",{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image2.eab64c256cdac4c867059511be541fe159dd1b28e5da1aafb4ed1dee9583b42b.64842105.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"leverage-treasure-workflow-for-orchestration","__idx":11},"children":["Leverage Treasure Workflow for Orchestration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Treasure Workflow's are an enhanced version of the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.digdag.io/"},"children":["digdag"]}," workflow engine. In order to leverage Treasure Workflow as the orchestration tool we will need to create a digdag project."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"ordinary-digdag-project","__idx":12},"children":["Ordinary ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["digdag"]}," Project"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Create a folder for workflow project called ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project"]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Move both ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_profiles"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dbt_project"]}," into ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project"]}," folder."]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Prepare Python Wrapper for invoking ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/reference/programmatic-invocations"},"children":["dbtRunner"]}," in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/scripts/dbt_wrapper.py"]}," so that digdag engine can use `py>`` operator to invoke dbt CLI library."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"python","header":{"controls":{"copy":{}}},"source":"from dbt.cli.main import dbtRunner\n\ndef invoke(command_list):\n    for cmd in command_list:\n        dbtRunner().invoke([\n            *(cmd.split()),\n            '--project-dir', './dbt_project',\n            '--profiles-dir', './dbt_profiles',\n        ])\n","lang":"python"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Prepare workflow definition in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/dbt_exec_workflow.dig"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"\n+dbt_invoke:\n  py>: scripts.dbt_wrapper.invoke\n  _export:\n    command_list:\n      - run\n","lang":"yaml"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Now the folder structure should look like:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sh","header":{"controls":{"copy":{}}},"source":"td_wf_project\n├── dbt_exec_workflow.dig\n├── dbt_packages\n├── dbt_profiles\n│   └── profiles.yml\n├── dbt_project\n│   ├── README.md\n│   ├── analyses\n│   ├── dbt_project.yml\n│   ├── macros\n│   │   └── td_adapter_patch\n│   │       └── adapters.sql\n│   ├── models\n│   │   └── example\n│   │       ├── my_first_dbt_model.sql\n│   │       ├── my_second_dbt_model.sql\n│   │       └── schema.yml\n│   ├── seeds\n│   ├── snapshots\n│   └── tests\n└── scripts\n    └── dbt_wrapper.py\n","lang":"sh"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Run the workflow (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["digdag run dbt_exec_workflow.dig"]},") to see results."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ digdag run dbt_exec_workflow.dig\n\n...\n22:38:09  Finished running 2 table models in 0 hours 0 minutes and 12.71 seconds (12.71s).\n22:38:09\n22:38:09  Completed successfully\n22:38:09\n22:38:09  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2\nSuccess. Task state is saved at .../dbt_in_td_example/td_wf_project/.digdag/status/20231016T000000+0000 directory.\n  * Use --session <daily | hourly | \"yyyy-MM-dd[ HH:mm:ss]\"> to not reuse the last session time.\n  * Use --rerun, --start +NAME, or --goal +NAME argument to rerun skipped tasks.\n","lang":"bash"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"convert-the-digdag-project-to-treasure-workflow-project","__idx":13},"children":["Convert the DigDag project to Treasure Workflow project"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["py>"]}," operator in Treasure Workflow is containerized as part of the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench/workflows/customscript"},"children":["Custom Scripts"]}," feature. To enable this in our project we need to add a few dependencies to our project."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Add runtime pip install in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/scripts/dbt_wrapper.py"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"python","header":{"controls":{"copy":{}}},"source":"import os\nimport sys\nos.system(f\"{sys.executable} -m pip install dbt-core==1.6.6 dbt-trino==1.6.2\")\n\nfrom dbt.cli.main import dbtRunner\n\ndef invoke(command_list):\n    for cmd in command_list:\n        dbtRunner().invoke([\n            *(cmd.split()),\n            '--project-dir', './dbt_project',\n            '--profiles-dir', './dbt_profiles',\n        ])\n","lang":"python"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Add docker option in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/dbt_exec_workflow.dig"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+dbt_invoke:\n  py>: scripts.dbt_wrapper.invoke\n  _export:\n    command_list:\n      - run\n  docker:\n    image: \"digdag/digdag-python:3.10\"\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"protect-secrets","__idx":14},"children":["Protect Secrets"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Keeping secrets safe is always important. To do this we are going to follow the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench/workflows/customscript/how-to-use-secrets-in-custom-scripts"},"children":["How to use Secrets in Custom Scripts"]}," docs to protect Treasure API Key."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Adding an environment variable in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/dbt_exec_workflow.dig"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+dbt_invoke:\n  py>: scripts.dbt_wrapper.invoke\n  _export:\n    command_list:\n      - run\n  _env:\n    TD_API_KEY: ${secret:td.apikey}\n  docker:\n    image: \"digdag/digdag-python:3.10\"\n","lang":"yaml"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Replace the Treasure API Key, previously set in user  ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/dbt_profiles/profiles.yml"]}," with a referring to the environment variable."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"      ...\n      user: \"{{ env_var('TD_API_KEY') }}\"\n      ...\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"deploy-to-treasure-workflow","__idx":15},"children":["Deploy to Treasure Workflow"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To deploy the workflow to the Treasure Data platform you need to clean up and the push the project."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Clean up before encapsulating package."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Recommend to add ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["logs"]}," into the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.getdbt.com/reference/project-configs/clean-targets#remove-logs-when-running-dbt-clean"},"children":["clean targets"]},"."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sh","header":{"controls":{"copy":{}}},"source":"$ dbt clean --profiles-dir dbt_profiles/ --project-dir dbt_project\n","lang":"sh"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Push project to Treasure Workflow."]},{"$$mdtype":"Tag","name":"div","attributes":{"data-language":"bash"},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["td workflow push td_wf_project"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["2023-10-16 16:31:56 -0700: Digdag v0.10.5"," ","Creating .digdag/tmp/archive-3702033545268916293.tar.gz..."," ","Archiving dbt_profiles/profiles.yml"," ","Archiving dbt_exec_workflow.dig"," ","..."," ","Workflows:"," ","dbt_exec_workflow.dig"," ","Uploaded:"," ","id: xxxxxx"," ","name: td_wf_project"," ","..."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Use ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td workflow workflows"]}," to show all workflows."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"\n- Set secret for Treasure Workflow `td_wf_project` project.\n\n```bash\n$ td workflow secrets --project td_wf_project --set td.apikey=XXXXXX\n\n2023-10-16 16:39:58 -0700: Digdag v0.10.5\nSecret 'td.apikey' set\n"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In Treasure Data Workbench’s Workflows page, search project name ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project"]},", then you should see ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td.apikey"]}," is set."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image3.cdab1234a7a081e9e61ebd28f551658d6bece2b11cd09164c4577f4acde7159d.64842105.png","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Trigger a ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["“New Run”"]},", the execution results will show in the workflow execution logs."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image4.18b399b94ac1aef8b0a5e3d83f6eb399c9297901cff224882cdafc934e483461.64842105.png","alt":""},"children":[]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"misc-options","__idx":16},"children":["Misc Options"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Some additional things you may want to consider doing with the project include:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Add more models or referring other sources in dbt_project."]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Add ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/customer-data-platform/data-workbench/workflows/scheduling-workflows"},"children":["scheduling"]}," to periodically trigger workflow and dbt run in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td_wf_project/dbt_exec_workflow.dig"]},", like:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"timezone: UTC\n\nschedule:\n  daily>: 07:00:00\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Then simply re-push project by:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ td workflow push td_wf_project\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"next-time","__idx":17},"children":["Next Time"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["From above steps, a minimum architecture integration is created and deployed. However, there are still many optional augments and tips can greatly empower data operation’s efficiency, maintainability, and observability. Following topics will be covered a future post."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Invoke with parameters"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Version Control & Documentation"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Observability - log & store results"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Materialized - window refresh for batch processing pipelines"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Package management"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Schema auto evolution"]}]}]},"headings":[{"value":"Orchestrate dbt with Treasure Workflow","id":"orchestrate-dbt-with-treasure-workflow","depth":1},{"value":"What is dbt?","id":"what-is-dbt","depth":1},{"value":"Prerequisites","id":"prerequisites","depth":1},{"value":"Recommended Background Knowledge","id":"recommended-background-knowledge","depth":2},{"value":"Software Resuirements","id":"software-resuirements","depth":2},{"value":"Leverage Treasure Data Presto Query Engine for Data Warehousing","id":"leverage-treasure-data-presto-query-engine-for-data-warehousing","depth":1},{"value":"Setup & init","id":"setup--init","depth":2},{"value":"Tuning Tips","id":"tuning-tips","depth":2},{"value":"Quoting","id":"quoting","depth":3},{"value":"Disable View & Materialized view","id":"disable-view--materialized-view","depth":3},{"value":"First run","id":"first-run","depth":2},{"value":"Leverage Treasure Workflow for Orchestration","id":"leverage-treasure-workflow-for-orchestration","depth":1},{"value":"Ordinary digdag Project","id":"ordinary-digdag-project","depth":2},{"value":"Convert the DigDag project to Treasure Workflow project","id":"convert-the-digdag-project-to-treasure-workflow-project","depth":2},{"value":"Protect Secrets","id":"protect-secrets","depth":2},{"value":"Deploy to Treasure Workflow","id":"deploy-to-treasure-workflow","depth":2},{"value":"Misc Options","id":"misc-options","depth":2},{"value":"Next Time","id":"next-time","depth":1}],"frontmatter":{"title":"Orchestrate dbt with Treasure Workflow","author":"Ansel","date":"2023-11-01T00:00:00.000Z","categories":["dbt","workflow"],"image":"orchestrate_dbt/orchestrate.png","seo":{"title":"Orchestrate dbt with Treasure Workflow","description":"A walkthrough of how to leverage dbt Core (command line tools) with the Treasure Data ecosystem.","image":"/assets/orchestrate.164d7a459eb8a7827525e3b96a4bb6c1dbacfe8a5f10386732a635b267827ce6.64842105.png"}},"lastModified":"2026-06-04T10:17:23.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/blog/orchestrate_dbt","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}