Operators are the actions that workflow tasks run, such as running a query, waiting for data to arrive, or sending an email. Each task in a workflow definition uses one operator. This page lists the operators available in Treasure Workflow and the common parameters that you can use with every operator.
| Operator | Description |
|---|---|
| call | Calls another workflow |
| http_call | Call workflow fetched by HTTP |
| require | Depends on another workflow |
| loop | Repeat tasks |
| for_each | Repeat tasks for values |
| for_range | Repeat tasks for a range |
| if | Conditional execution |
| fail | Makes the workflow failed |
| echo | Shows a message |
| wait | Wait for a specific duration |
| Operator | Description |
|---|---|
| td | Treasure Data queries |
| td_run | Treasure Data saved queries |
| td_ddl | Treasure Data operations |
| td_load | Treasure Data bulk loading |
| td_for_each | Repeat using Treasure Data queries |
| td_wait | Waits for data arriving at Treasure Data table |
| td_wait_table | Waits for data arriving at Treasure Data table |
| td_table_export | Treasure Data table export to S3 |
| td_result_export | Treasure Data result exporter |
| Operator | Description |
|---|---|
| databricks | Execute SQL statements on Databricks |
| pg | PostgreSQL operations |
| snowflake | Execute SQL statements on Snowflake |
| Operator | Description |
|---|---|
| s3_wait | Wait for a file in Amazon S3 |
| s3_copy | Copy files in Amazon S3 |
| s3_delete | Delete files in Amazon S3 |
| s3_move | Move files in Amazon S3 |
| redshift | Redshift operations |
| redshift_load | Redshift load operations |
| redshift_unload | Redshift unload operations |
| Operator | Description |
|---|---|
| gcs_wait | Wait for a file in Google Cloud Storage |
| bq | Running Google BigQuery queries |
| bq_ddl | Managing Google BigQuery Datasets and Tables |
| bq_extract | Exporting Data from Google BigQuery |
| bq_load | Importing Data into Google BigQuery |
| Operator | Description |
|---|---|
| py | Python scripts |
| Operator | Description |
|---|---|
| embulk | Embulk data transfer |
| emr | Running Amazon EMR jobs |
| sh | Running shell scripts |
| rb | Running Ruby scripts |
| param_get | Getting persistent data |
| param_set | Storing persistent data |
Common parameters control how a task runs, such as how long it can take. You can use the following parameters with every operator.
_timeout: DURATION
The maximum time the task can take. If the task does not finish within this time, it fails, and the operation it started, such as a running query, is canceled.
Set a duration with a time unit, such as
30s,10m, or1h 30m.The time is measured from the first time the task starts running. If the task retries, the retries count toward the same limit.
Some operators cannot cancel their operation after it starts. For these operators, the task may run longer than the specified timeout. In that case, a warning appears in the task log.
Examples:
+load_data: td>: load_data.sql _timeout: 1h