Custom Scripts enable running Python scripts in isolated Docker containers within Treasure Workflow, providing greater flexibility of custom logic. Typical uses include:
- Extend the capabilities of data connectors and other integrations.
- Create efficient data manipulation and processing logic in Python and invoke it from workflows.
- Productionize your data science work, by enabling Python models to be run as part of regularly scheduled Treasure Workflows.
- Consolidate your data management into one environment. Use Treasure Workflow to connect multiple data environments.
Specify a Docker image in your workflow definition using the py> operator:
+py_custom_code:
py>: tasks.printMessage
docker:
image: "treasuredata/customscript-python:3.12.11-td2"This task launches the specified Docker image and runs the printMessage function from the tasks Python module.
Each Docker image includes a specific Python version and pre-installed libraries. You can also install additional libraries as needed.
- Python 3.12
- Python 3.10
- Python 3.9
Currently, Treasure Workflow supports the following Docker images:
- treasuredata/customscript-python:3.12.11-td2 [current stable]
- digdag/digdag-python:3.10.18-td0
- digdag/digdag-python:3.10.1 (3.10) [deprecated]
- digdag/digdag-python:3.9.22-td2
- digdag/digdag-python:3.9.2 [deprecated]
- digdag/digdag-python:3.9.1 (3.9) [deprecated]
Some images listed above still use the deprecated versioning scheme (such as 3.9, 3.9.1, 3.9.2) and contain security vulnerabilities. We recommend using images with the current versioning scheme (such as 3.12.11-td2) that include security fixes.
For details about our versioning policy, see Docker Image Versioning Policy.
For the full list of installed libraries, see Custom Scripts Docker Images.
See Adding a Custom Python Script to Your Workflow to get started.