{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Luigi Workflow Engine","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"luigi-workflow-engine","__idx":0},"children":["Luigi Workflow Engine"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/spotify/luigi"},"children":["Luigi"]}," is a powerful, easy-to-use open-source framework for building data pipelines with Python. It handles dependency resolution, workflow management, visualization etc."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image-20190926-220811.9031c12a47c87d604e79992c1c15bed4366e1faedd68e62406804d7f61f0e654.9b6395b9.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Luigi helps to build the data pipeline, typically associated with long-running batch processes. You want to chain many tasks together, automate them, and gracefully handle any failures."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Luigi pipelines automatically handle dependency resolution, checkpointing and failure recovery, parallel execution, command line integration, and much more. Pipelines are expressed as simple, easy-to-read Python that can be reused anywhere."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://luigi-td.readthedocs.org/en/latest/"},"children":["Luigi-TD"]}," is a helper library to make it easier to issue Treasure Data queries from Luigi tasks. This will allow you to create complex pipelines on top of TD easily."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://luigi-td.readthedocs.org/en/latest/gettingstarted.html"},"children":["Luigi-TD Documentation"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://luigi.readthedocs.org/en/latest/"},"children":["Luigi Documentation"]}]}]}]},"headings":[{"value":"Luigi Workflow Engine","id":"luigi-workflow-engine","depth":1}],"frontmatter":{"seo":{"title":"Luigi Workflow Engine"}},"lastModified":"2026-01-27T10:05:25.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/luigi-workflow-engine","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}