Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.

This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.

• Store data: Store an infinite amount of data in a bucket.

Review the information in the following table to understand the features that are available in v2 that are not available in v1.

Amazon S3 Export v2 is a separate integration from v1. There is no in-place upgrade: to move an existing v1 export, create a new Amazon S3 v2 authentication, point your query's Export Results at it, validate the output, and then retire the v1 export.

Most settings map directly (Bucket, Path prefix, Format, Header line, Delimiter, Quote character, Escape character, and Part Size). The items below change behavior and should be reviewed before you switch a production export.

When exporting in TSV format, special characters within string values (tab, newline, backspace, form feed, and the backslash escape character) are currently encoded differently than in v1. For example, a literal tab is written as the bare letter t with the leading backslash dropped, rather than as the escaped sequence \t. If your TSV string columns can contain these characters, export a sample and compare it with your v1 output before switching a production export.

1. Create a new Amazon S3 v2 authentication (Integrations Hub > Catalog > S3 v2).
2. Edit your query's Export Results to use the new Amazon S3 v2 authentication, and map the settings above.
3. Run a test export to a temporary path; do not overwrite your production output yet.
4. Validate the output file: format, header, delimiter, quoting, NULL representation, line endings, compression, and encryption.
5. Switch the production export. For scheduled queries, confirm the next scheduled run, not only a manual run.
6. Keep the v1 authentication until you have validated v2 in production. If the v2 output is not correct, switch Export Results back to the v1 authentication. Retire v1 only after every dependent export has been migrated.

• The following AWS permissions for the IAM User :
  • with s3:PutObject, s3:AbortMultipartUpload permissions.
  • with kms:Decrypt, kms:GenerateDataKey permissions when selecting the sse-kms setting.
• (Optional) Basic knowledge of Treasure Data, including the TD Toolbelt.

• The default query result limit for export to S3 is 100GB. You can configure the part size setting up to 5000 (MB). The file limit will be about 5TB.
• The default export format is CSV RFC 4180.
• Output in TSV and JSONL format are also supported.
• TLS v.1.2 supported

If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.

Please find the complete list of static IP addresses, organized by region, at the following document

You can encrypt upload data with AWS S3 Server-Side Encryption. You don't need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).

Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have the server-side encryption enabled, you don't have to turn on the SSE option. However, job results may fail if you have bucket policies that reject HTTP requests without encryption information.

You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3).

When you enable AWS KMS for server-side encryption in Amazon S3:

• If you don't input the KMS Key ID, it will use (or create) the default KMS key.
• If you input the KMS Key ID, you must choose symmetric CMK (not asymmetric CMKs).
• The AWS KMS CMK must be in the same region as the bucket.

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

1. Open Treasure Console.
2. Navigate to Integrations Hub > Catalog.
3. Search for S3 v2and select Amazon S3 (v2).
4. Select Create Authentication.

A new Authentication dialog opens. Depending on the Authentication method you choose, the dialog may look like one of these screens:

1. Configure the authentication fields, and then select Continue.

2. Name your new AWS S3 connection, and select Done.

1. Create a new authentication with the assume_role authentication method.
2. Make a note of the numeric portion of the value in the TD's Instance Profile field.

1. Create your AWS IAM role.

1. Select Export Results.
2. You can select an existing authentication or create a new authentication for the external service to be used for output. Choose one of the following:

• Use Existing Integration

• Create a New Integration

(Optional) Specify information for Export to Amazon S3.

1. Define any additional Export Results details and content review the integration parameters. For example, your Export Results screen might be different, or you might not have additional details to fill out.

1. Select Done.
2. Run your query
3. Validate that your data moved to the destination you specified.

SELECT * FROM www_access

You can also send segment data to the target platform by creating an activation in the Audience Studio.

1. Navigate to Audience Studio.
2. Select a parent segment.
3. Open the target segment, right-mouse click, and then select Create Activation.
4. In the Details panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters.
5. Customize the activation output in the Output Mapping panel.

• Attribute Columns
  • Select Export All Columns to export all columns without making any changes.
  • Select + Add Columns to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select + Add Columnsto add new columns for your activation output.
• String Builder
  • + Add string to create strings for export. Select from the following values:
    • String: Choose any value; use text to create a custom value.
    • Timestamp: The date and time of the export.
    • Segment Id: The segment ID number.
    • Segment Name: The segment name.
    • Audience Id: The parent segment number.

6. Set a Schedule.

• Select the values to define your schedule and optionally include email notifications.

7. Select Create.

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

Treasure Data's scheduler feature supports periodic query execution to achieve high availability.

When two specifications provide conflicting schedule specifications, the specification requesting to execute more often is followed while the other schedule specification is ignored.

For example, if the cron schedule is '0 0 1 * 1', then the 'day of month' specification and 'day of week' are discordant because the former specification requires it to run every first day of each month at midnight (00:00), while the latter specification requires it to run every Monday at midnight (00:00). The latter specification is followed.

1. Navigate to Data Workbench > Queries

2. Create a new query or select an existing query.

3. Next to Schedule, select None.

4. In the drop-down, select one of the following schedule options:

   Drop-down Value	Description
   Custom cron...	Review Custom cron... details.
   @daily (midnight)	Run once a day at midnight (00:00 am) in the specified time zone.
   @hourly (:00)	Run every hour at 00 minutes.
   None	No schedule.

 *    *    *    *    *
 -    -    -    -    -
 |    |    |    |    |
 |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 |    |    |    +---------- month (1 - 12)
 |    |    +--------------- day of month (1 - 31)
 |    +-------------------- hour (0 - 23)
 +------------------------- min (0 - 59)

The following named entries can be used:

• Day of Week: sun, mon, tue, wed, thu, fri, sat.
• Month: jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec.

A single space is required between each field. The values for each field can be composed of:

1. (Optional) You can delay the start time of a query by enabling the Delay execution.

Save the query with a name and run, or just run the query. Upon successful completion of the query, the query result is automatically exported to the specified destination.

(Optional) You can delay the start time of a query by enabling the Delay execution.

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Exporting Data with Parameters.

_export:
  td:
  database: td.database

+s3v2_test_export_task:
  td>: export_s3v2_test.sql
  database: ${td.database}
  result_connection: s3v2_conn
  result_settings:
    bucket: my-bucket
    path: /path/to/target.csv
    sse_type: sse-s3
    format: csv
    compression: gz
    header: false
    delimiter: default
    null_value:  empty
    newline: LF
    quote_policy: MINIMAL
    escape: '"'
    quote: '"'
    part_size: 20
    json_columns: col_map,col_array,col_json_string

To output the result of a single query to an S3 bucket add the "--result option" to the td query command. After the job is finished, the results are written into your S3 bucket.

You can specify detailed settings to export your S3 via the "--result parameter".

Create an Authentication with Assume Role is only supported through the console UI. Follows the instruction Reuse the existing Authentication to get the Authentication ID, and reuse it for CLI

td query \
--result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto

td query \
--result '{"type":"s3_v2","td_authentication_id": 77348,"bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto