{"templateId":"markdown","sharedDataIds":{},"props":{"metadata":{"markdoc":{"tagList":["blog-author","blog-share"]},"type":"markdown"},"seo":{"title":"Continuous Deployment of Treasure Workflow with Azure DevOps","description":"How to integrate Treasure Workflow with Azure DevOps","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]},"image":"/assets/azure-td.146d1840db181592cd25388b2107812aed7f875dc2c0bb836e9a876c2456deb0.8446fe5d.jpg"},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"continuous-deployment-of-treasure-workflow-with-azure-devops","__idx":0},"children":["Continuous Deployment of Treasure Workflow with Azure DevOps"]},{"$$mdtype":"Tag","name":"BlogAuthor","attributes":{"name":"Toru Takahashi","date":"2023-06-16","image":"./images/toru.png"},"children":[]},{"$$mdtype":"Tag","name":"BlogShare","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Treasure Workflow can be used with version control tools like Git and CI/CD. However, a recent enablement now allows you to set up a continuous deployment pipeline using Azure Repos and Azure Pipelines in Azure DevOps Sevices."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"prerequisites","__idx":1},"children":["Prerequisites"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Azure Repos"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Azure Pipelines"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Treasure Data User Account"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"azure-repo-repository","__idx":2},"children":["Azure Repo repository"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you have not already done so, create a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["git"]}," repository on Azure Repo for your workflow project. For more information on how to use Azure Repos, see the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://learn.microsoft.com/en-us/azure/devops/repos/?view=azure-devops"},"children":["Azure Repos documentation"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["I recommend having the following directory structure in your Treasure Workflow repo."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"my_project\n├── README.md\n├── config\n│   ├── params.test.yml     <- Configuration file for run through test. Mirrors params.yml except for `td.database`\n│   └── params.yml          <- Configuration file for production\n├── awesome_workflow.dig    <- Main workflow to be executed\n├── ingest.dig              <- Data ingestion workflow\n├── py_scripts              <- Python scripts directory\n│   ├── __init__.py\n│   ├── data.py             <- Script to upload data to Treasure Data\n│   └── my_script.py        <- Main script to execute e.g. Data enrichment, ML training\n├── queries                 <- SQL directory\n│   └── example.sql\n├── run_test.sh             <- Test shell script for local run through test\n└── test.dig                <- Test workflow for local run through test\n└── azure-pipeline.yml           <- Deploy this repo to Treasure Workflow through Azure Pipeline (This file is automatically created when a new pipeline is created)\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/azure_1.2e1f9e453fd4d3c4f77bac43b71bee3f441310cb15d7fadc8ba826af64120c07.8446fe5d.png","alt":"Directory Layout"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For more information on custom script development, see the blog post \"",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://chezo.uno/blog/2020-03-05_py-operator-development-guide-for-python-users"},"children":["py> operator development guide for Python users"]},".\""]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"configure-azure-pipeline","__idx":3},"children":["Configure Azure Pipeline"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To createa new Azure Pipeline for your project:"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Azure Repos Git"]}," in Select section"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Python Package"]}," in Configure section"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/azure_2.7ac650b876c6b701675cdf2785f82e26ff3679113167480e7f782ce407dd8238.8446fe5d.png","alt":"image2"},"children":[]}]},{"$$mdtype":"Tag","name":"ol","attributes":{"start":3},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Set Variables on the right-hand side."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Enter ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td_apikey"]}," into the Name field"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Enter your Treasure API key in the Value field."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Keep this value secret"]},"."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Save variables"]},"."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/azure_3.1802282ec40ed412fba896a9c0211ccd7f91ccc9711e4434eaf67ead15e40e6a.8446fe5d.png","alt":"image3"},"children":[]}]},{"$$mdtype":"Tag","name":"ol","attributes":{"start":8},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Update the Azure Pipeline configuration file for the project. You must update ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["tdWorkflowEndpoint"]}," for your account region and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["tdPrjName"]}," for your workflow project in variables"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"jobs:\n- job: 'td_workflow_deployment'\n  pool:\n    vmImage: 'ubuntu-latest'\n  strategy:\n    matrix:\n      Python311:\n        python.version: '3.11'\n\n  variables:\n    tdWorkflowEndpoint: api-workflow.treasuredata.com\n    # US: api-workflow.treasuredata.com\n    # EU01: api-workflow.eu01.treasuredata.com\n    # Tokyo: api-workflow.treasuredata.co.jp\n    # AP02: api-workflow.ap02.treasuredata.com\n    # Ref. /en/overview/aboutendpoints\n    tdPrjName: azure_devops_wf # YOUR PROJECT NAME\n\n\n  steps:\n  - script: pip install tdworkflow mypy_extensions\n    displayName: 'Install tdworkflow lib'\n\n  - task: PythonScript@0\n    inputs:\n      scriptSource: inline\n      script: |\n        import os\n        import sys\n        import shutil\n        import tdworkflow\n        endpoint = \"$(tdWorkflowEndpoint)\"\n        apikey = \"$(td_apikey)\"\n        project_name = \"$(tdPrjName)\"\n        shutil.rmtree('.git/') # Remove unnessary temp files\n        client = tdworkflow.client.Client(endpoint=endpoint, apikey=apikey)\n        project = client.create_project(project_name, \".\")\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For more information on Azure Pipeline configuration, see \"",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://learn.microsoft.com/bs-latn-ba/azure/devops/pipelines/process/environments?view=azure-devops"},"children":["Azure Pipelines: Create and target an environment"]},".\""]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"configure-td-credentials","__idx":4},"children":["Configure TD Credentials"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For workflows to be pushed to Treasure Data, you must configure a TD Master API Key for the project. See ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/products/my-settings/getting-your-api-keys"},"children":["Getting Your API Keys"]}," for more information."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"start-deploying","__idx":5},"children":["Start Deploying"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Azure Pipelines pushes your workflow to Treasure Data and does this every time you push a change to Azure Repos. You can change a deployment condition depending on your needs. Once deployment is completed, Treasure Workflow on Treasure Data is updated automatically."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/azure_4.31fe652714c60776ee487e27479eb613e3df303dde715c157975f99b8da25aa4.8446fe5d.png","alt":"image4"},"children":[]}]}]},"headings":[{"value":"Continuous Deployment of Treasure Workflow with Azure DevOps","id":"continuous-deployment-of-treasure-workflow-with-azure-devops","depth":1},{"value":"Prerequisites","id":"prerequisites","depth":2},{"value":"Azure Repo repository","id":"azure-repo-repository","depth":2},{"value":"Configure Azure Pipeline","id":"configure-azure-pipeline","depth":2},{"value":"Configure TD Credentials","id":"configure-td-credentials","depth":2},{"value":"Start Deploying","id":"start-deploying","depth":2}],"frontmatter":{"title":"Continuous Deployment of Treasure Workflow with Azure DevOps","author":"Toru","date":"2023-06-16T00:00:00.000Z","image":"azure_devops/azure-td.jpg","categories":["workflow","cicd","azure"],"seo":{"title":"Continuous Deployment of Treasure Workflow with Azure DevOps","description":"How to integrate Treasure Workflow with Azure DevOps","image":"/assets/azure-td.146d1840db181592cd25388b2107812aed7f875dc2c0bb836e9a876c2456deb0.8446fe5d.jpg"}},"lastModified":"2026-06-04T10:17:23.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/ja/blog/azure_devops","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}