{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Scheduling A Data Connector Job Execution From The CLI","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"scheduling-a-data-connector-job-execution-from-the-cli","__idx":0},"children":["Scheduling A Data Connector Job Execution From The CLI"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Scheduled execution supports additional configuration parameters that control the behavior of the data connector during its periodic attempts to fetch data from the integration source:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["incremental"]}," This configuration is used to control the load mode, which governs how the data connector fetches data from the integration based on one of the native timestamp or numeric field associated with each object"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["incremental_columnn"]}," This configuration is used to define a based column to imported into Treasure Data. You can define only one column for this field. Suggested value is ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["created, createdTimestamp, updated, updatedTimestamp"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Timezone abbreviations like PST, CST are not supported and may lead to unexpected schedules."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["The --timezone option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here’s an example of a load file using incremental mode"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"in:\n  type: intg_type\n  data_center: US1\n  authentication_mode: key_secret\n  application_key: your_application_user_key\n  secret_key: your_application_secret_key\n  api_key: your_api_key\n  data_source: account\n  batch_size: 1000\n  query: SELECT * FROM table_name\n  incremental: true\n  incremental_column: created\nfilters:\n- type: add_time\n  from_value:\n    mode: upload_time\n  to_column:\n    name: time\n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"create-the-schedule","__idx":1},"children":["Create the Schedule"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A new schedule can be created using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the data connector configuration file are required."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cron"]}," parameter accepts these options: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@hourly"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@daily"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@monthly"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["By default, the schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For example, you can create a scheduled job using the command td connector:create to run daily:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"td connector:create connector_name @daily \\\nconnector_database connector_table load.yml\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["It’s also recommended to specify the ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["--time-column"]}," option, because Treasure Data’s storage is partitioned by time (see also ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/smart/project-product-documentation/data-partitioning-in-treasure-data"},"children":["data partitioning"]},")."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"td connector:create daily_import \\\n\"10 0 * * *\" \\\ntd_sample_db td_sample_table load.yml \\\n--time-column created_at\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["cron"]}," parameter also accepts three special options: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@hourly"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@daily"]},", and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["@monthly"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["By default, the schedule is setup in the UTC timezone. You can set the schedule in a timezone using -t or --timezone option. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["--timezone"]}," option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"list-all-schedules","__idx":2},"children":["List All Schedules"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can see the list of currently scheduled entries by running the command ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["td connector:list"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"td connector:list\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"show-schedule-settings-and-history","__idx":3},"children":["Show Schedule Settings and History"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"em","attributes":{},"children":["td connector:show"]}," shows the execution setting of a schedule entry."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"td connector:show daily_import\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"Name     : daily_import\nCron     : 10 0 * * *\nTimezone : UTC\nDelay    : 0\nDatabase : td_sample_db\nTable    : td_sample_table\nConfig\n---\nin:\n  type: s3\n  access_key_id: access_key_id\n  secret_access_key: secret_access_key\n  endpoint: endpoint\n  bucket: bucket\n  path_prefix: path_prefix\n  parser:\n    charset: UTF-8\n    ...\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"em","attributes":{},"children":["td connector:history"]}," shows the execution history of a schedule entry. To investigate the results of each individual run, use ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["td job jobid"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"td connector:history daily_import\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+\n| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |\n+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+\n| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-18 00:10:05 +0000 | 160      |\n| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-17 00:10:07 +0000 | 161      |\n| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-16 00:10:03 +0000 | 152      |\n| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-15 00:10:04 +0000 | 163      |\n| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-14 00:10:04 +0000 | 164      |\n| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-13 00:10:04 +0000 | 155      |\n| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-12 00:10:05 +0000 | 156      |\n| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-11 00:10:04 +0000 | 157      |\n+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+\n8 rows in set\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"delete-schedule","__idx":4},"children":["Delete Schedule"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"em","attributes":{},"children":["td connector:delete"]}," removes the schedule."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"td connector:delete daily_import\n"},"children":[]}]},"headings":[{"value":"Scheduling A Data Connector Job Execution From The CLI","id":"scheduling-a-data-connector-job-execution-from-the-cli","depth":1},{"value":"Create the Schedule","id":"create-the-schedule","depth":2},{"value":"List All Schedules","id":"list-all-schedules","depth":2},{"value":"Show Schedule Settings and History","id":"show-schedule-settings-and-history","depth":2},{"value":"Delete Schedule","id":"delete-schedule","depth":2}],"frontmatter":{"seo":{"title":"Scheduling A Data Connector Job Execution From The CLI"}},"lastModified":"2025-12-23T04:19:16.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/scheduling-a-data-connector-job-execution-from-the-cli","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}