{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Using Treasure Workflow With Import and Export connectors","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"using-treasure-workflow-with-import-and-export-connectors","__idx":0},"children":["Using Treasure Workflow With Import and Export connectors"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Most batch import and export integrations can be added to workflows as part of a more advanced data pipeline. The basic steps are:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Prepare a workflow definition, which is a YAML file, using the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td_load>"]}," operator for an import integration and the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td>"]}," operator for an export integration."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Run the workflow on the Treasure Console, or from a command-line interface"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"usea-pre-defined-source-of-an-import-integration","__idx":1},"children":["Use a pre-defined Source of an import integration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you have already created a Source for an import integration, you can run it in a workflow using its unique ID"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Identify the target Source on the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Integration Hub"]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Select the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["..."]}," menu, and select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Copy Unique ID"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/using-td-workflow-with-td-integrations-2024-08-09.f0acc37b697f4693c6d5e5f2fccc80706461e31eff5d1baabef37ed64a2c2773.18b4736f.png","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Define a workflow task using the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td_load>"]}," operator."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+my_first_worklfow_step:\n   td_load>: unique_id_of_your_source\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"usea-configuration-filefor-animport-integration","__idx":2},"children":["Use a configuration file for an import integration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Alternatively, you can provide all the parameters of the import connector in a YAML file, and refer to it in the workflow definition"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Prepare a configuration file for your import integration, ex: ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["load_config.yml"]},". Examples and the full parameter explanation can be found in the document of each integration"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Define a workflow refers to it"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+my_first_workflow_step:\n   td_load>: config/load_config.yml\n   database: my_database_name\n   table: my_table_name\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"define-a-workflow-involving-an-export-integration","__idx":3},"children":["Define a workflow involving an export integration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["There are a few ways to define a workflow task to query and upload the results to an export integration"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Use the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td>"]}," operator and the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["name"]}," of a pre-defined Authentication"," ",{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/using-td-workflow-with-td-integrations-2024-08-09-1.a52133de619be06eed4ff85425ba571cd590adc97e5e7d600dd617864ea051f6.18b4736f.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+query_and_export:\n   td>: queries/sample.sql\n   database: my_database\n   result_connection: ads_with_oauth\n   result_settings:\n      parameter1: value1\n      parameter2: value2\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The parameter explanation can be found on each integration document, as the one below from the SFCC Export Integration"," ",{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/using-td-workflow-with-td-integrations-2024-08-14.c8d92f28ae6ba0e800fa2fda32d81638c3713763f4f9e5dd43fdb074178ff49e.18b4736f.png","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Use the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td>"]}," operator and the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["ID"]}," of a pre-defined Authentication"," ",{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/google-ads-import-integration-v2-2024-08-06-9.fdc162a377e20998a8f41e69c38afef49bbd1ab0b84f6b913fd06fe82b2010f3.7c52a81f.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+query_data_and_write_result_with_existing_authentication_by_url:\n   td>:\n   query: 'select * from wf_test'\n   database: my_database\n   result_url: '{\"type\": \"google_adwords_v2\", \"td_authentication_id\": 330392, \"bucket\": \"qa-auto-test-3\", \"path\": \"viet/assume_role_from_eu01.csv\"}'\n","lang":"yaml"},"children":[]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Use the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td_run>"]}," operator and a saved query"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"+query_a_saved_sql:\n   td_run>: presto_select_all_s3v2\n   database: my_database\n   result_url: '{\"type\":\"s3_v2\", \"td_authentication_id\":260370, \"bucket\": \"qqus-east-1\", \"path\":\"foo/wf_jan16_1444.csv\"}'\n","lang":"yaml"},"children":[]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"run-a-workflow","__idx":4},"children":["Run a workflow"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["There are two options to define and run a workflow"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Use the graphical interface on ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Treasure Console"]}," > ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data Workbench"]}," > ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Workflows"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Use the command-line interface of TD Toolbelt"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["While using the Treasure Console is more straightforward than the command-line interface, the Treasure CLI tool offers options for more complex configurations."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Refer ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/data-workbench/workflows/about-workflow-syntax"},"children":["here"]}," for more details about the syntax and usages of the Treasure Workflow"]}]},"headings":[{"value":"Using Treasure Workflow With Import and Export connectors","id":"using-treasure-workflow-with-import-and-export-connectors","depth":1},{"value":"Use a pre-defined Source of an import integration","id":"usea-pre-defined-source-of-an-import-integration","depth":2},{"value":"Use a configuration file for an import integration","id":"usea-configuration-filefor-animport-integration","depth":2},{"value":"Define a workflow involving an export integration","id":"define-a-workflow-involving-an-export-integration","depth":2},{"value":"Run a workflow","id":"run-a-workflow","depth":3}],"frontmatter":{"seo":{"title":"Using Treasure Workflow With Import and Export connectors"}},"lastModified":"2026-06-01T09:09:59.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/using-td-workflow-with-td-integrations","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}