# Batch Import

Batch import lets you load historical data in bulk or schedule incremental data collection. Treasure AI supports the following import methods:

## Upload CSV or TSV files

You can upload CSV or TSV files directly from the Treasure Console.

1. Open Treasure Console.
2. Navigate to **Integrations Hub** > **Catalog**. The upload option is in the upper-right corner.
![Upload File button in Integrations Hub Catalog](/assets/image-20191016-205214.73dbb26b5977bcdd997ec53e497bf84ec2155b07d6c085ec6f98a8ef1e96642a.b2039cd8.png)
3. Select **Upload File**.
4. Drag your file into the window or browse to select it.
![Upload data from local file dialog with drag-and-drop area](/assets/image-20191016-203336.937b1cfb4c0daa9f0a192bbbaf89fec6c04e231c3364dc2014618f9365bc784f.b2039cd8.png)
5. Preview your data.
6. Optionally, select **Advanced Settings** to configure file decoding, delimiters, column names, and data types.
7. Define the destination database and table in Treasure AI.
8. Optionally, choose to append or replace existing data in the table.
9. Optionally, specify a timestamp to use as the partition key.
10. Select **Upload**.


File size limit
CSV and TSV uploads from the Treasure Console must be less than 200 MB.

## Import using integrations

You can bulk load data into Treasure AI by configuring an import integration from the catalog. Treasure AI offers connectors for website tracking, mobile apps, social media, cloud storage, databases, and more.

To set up an integration-based import:

1. Open Treasure Console and go to **Integrations Hub** > **Catalog**.
2. Select the data source type you want to add.
3. Provide authentication credentials to access the data source.
4. Define the source configuration (what data to fetch).
5. Preview your data (optional).
6. Define the destination database and table.
7. Set a schedule for data collection.


After setup, you can manage your sources from **Integrations Hub** > **Sources**. For details, see [Creating a New Data Source](/products/customer-data-platform/integration-hub/batch/import/creating-a-new-data-source).

## Bulk data import with Embulk

For large-scale imports, use [Embulk](https://www.embulk.org/docs/), Treasure AI's supported open-source bulk data loader. Embulk transfers data between databases, storage, file formats, and cloud services.

- [Installing Bulk Data Import](/products/customer-data-platform/integration-hub/batch/import/bulk-data-import#installing-bulk-data-import) — install Embulk and the Treasure AI plugin
- [Bulk Import Integration Options](/products/customer-data-platform/integration-hub/batch/import/bulk-data-import#bulk-import-integration-options) — supported source connectors
- [Using Environment Variables with Bulk Import](/products/customer-data-platform/integration-hub/batch/import/using-environment-variables-with-bulk-import)


## Data transformation 

During import, you can apply parsers, filters, and decoders to transform data before it lands in Treasure AI.

- **Parsers** — interpret file formats such as CSV, JSON, Avro, and MessagePack. See [Parsers](/products/customer-data-platform/integration-hub/batch/import/parser/csv-parser-function).
- **Filters** — manipulate columns, expand JSON, mask data, and more. See [About Filter Functions](/products/customer-data-platform/integration-hub/batch/import/filter/about-filter-functions).
- **Decoders** — decode compressed or encoded files before parsing. See [File Decoder Function](/products/customer-data-platform/integration-hub/batch/import/decoder/file-decoder-function).


## Import using TD Toolbelt (CLI)

Advanced users can import data using the TD Toolbelt command-line interface:

- [Using Database Drivers to Import Your Data](/tools/cli-and-sdks/using-database-drivers-to-import-your-data)
- [Capturing Streaming Log Data](/products/customer-data-platform/integration-hub/streaming/td-agent/capturing-streaming-log-data)
- [Incremental Data Import using Treasure Workflow](/products/customer-data-platform/integration-hub/batch/import/about-incremental-data-import-using-td-workflow)
- [Setting the TD Import Time Format for a Windows Batch File](/tools/cli-and-sdks/setting-the-td-import-time-format-for-a-windows-batch-file)