{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Logstash Import Integration","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"logstash-import-integration","__idx":0},"children":["Logstash Import Integration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.elastic.co/products/logstash"},"children":["Logstash"]}," is an open source software for log management and widely used as a part of the ELK stack."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Many plugins exist for Logstash to collect, filter, and store data from many sources, and into many destinations. You can ingest data from Logstash into Treasure Data by creating a Treasure Data plugin."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"install-and-configure-the-treasure-data-logstash-plugin","__idx":1},"children":["Install and Configure the Treasure Data Logstash Plugin"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can install the Treasure Data plugin for Logstash. The following example assumes that you already have Logstash installed and configured."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ cd /path/of/logstash\n$ bin/plugin install logstash-output-treasure_data\nValidating logstash-output-treasure_data\nInstalling logstash-output-treasure_data\nInstallation successful\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Configure Logstash with Treasure Data services. You must provide the name of a database and table into which your Logdash data is imported. You can retrieve your API key from your profile in Treasure Console. Use your write-only key."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"conf","header":{"controls":{"copy":{}}},"source":"input {\n  # ...\n}\noutput {\n  treasure_data {\n    apikey   => \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n    database => \"dbname\"\n    table    => \"tablename\"\n    endpoint => \"api-import.treasuredata.com\"\n  }\n}\n","lang":"conf"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Launch Logstash with the configuration file."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ bin/logstash -f your.conf\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can see rows of your data in Treasure Console. Log message texts are stored in message column, and some additional columns will exist (for example: time, host and version)."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/logtash-td.3c9a41c1e020ed11569ac4a2873a1951219c2ac01bd125e726091791f45b9d62.8c2a6faf.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"parameters","__idx":2},"children":["Parameters"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Logs are stored in a table. Create 2 or more sections in your configuration file if you want to insert data into 2 or more tables."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In your Logstash configuration, you can specify the following options:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["apikey (required)"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["database (required)"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["table (required)"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["auto_create_table [true]: creates a table if table doesn’t exists"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["endpoint [",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://api.treasuredata.com"},"children":["api.treasuredata.com"]},"]"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["use_ssl [true]"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["http_proxy [none]"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["connect_timeout [60s]"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["read_timeout [600s]"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["send_timeout [600s]"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The plugin work with default values for almost all cases, but some of the parameters might help you in unstable network environments."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This plugin buffers data in memory buffer in 5 minutes at most. Buffered data is lost if the Logstash process crashes. To avoid losing your data during import, we recommend to use td-agent with your Logstash plugin."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"the-logstash-plugin-in-combination-with-td-agent","__idx":3},"children":["The Logstash Plugin in Combination with td-agent"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Logstash plugin is limited in the areas of buffering, stored table specifications, and performance."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can use Treasure Agent (",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/smart/project-product-documentation/about-treasure-data-s-server-side-agent"},"children":["td-agent"]},") for more flexible and high performance transferring of data. You use logstash-output-fluentd to do it."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"[host a]\n  -> (logstash-output-fluentd) -+\n[host b]                        |\n  -> (logstash-output-fluentd) -+- [Treasure Agent] -> [Treasure Data]\n[host c]                        |\n  -> (logstash-output-fluentd) -+\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Logstash can be configured to send the logs to a td-agent node, and that td-agent stores all the data into Treasure Data."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"conf","header":{"controls":{"copy":{}}},"source":"# Configuration for Logastash\ninput {\n  # ...\n}\noutput {\n  fluentd {\n    host => \"your.host.name.example.com\"\n    port => 24224 # default\n    tag  => \"td.database.tablename\"\n  }\n}\n# Configuration for Fluentd\n<source>\n  @type forward\n  port 24224\n</source>\n<match td.*.*>\n  type tdlog\n  endpoint api-import.treasuredata.com\n  apikey YOUR_API_KEY\n  auto_create_table\n  buffer_type file\n  buffer_path /var/log/td-agent/buffer/td\n  use_ssl true\n  num_threads 8\n</match>\n","lang":"conf"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Fluentd tdlog plugin can store data into many database-table combinations by parsing td.dbname.tablename. You can configure any database and table pairs in Logstash configuration files."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"more-information","__idx":4},"children":["More Information"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/tagomoris/logstash-output-treasure_data"},"children":["github.com/tagomoris/logstash-output-treasure_data"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/tagomoris/logstash-output-fluentd"},"children":["github.com/tagomoris/logstash-output-fluentd"]}]}]}]},"headings":[{"value":"Logstash Import Integration","id":"logstash-import-integration","depth":1},{"value":"Install and Configure the Treasure Data Logstash Plugin","id":"install-and-configure-the-treasure-data-logstash-plugin","depth":2},{"value":"Parameters","id":"parameters","depth":2},{"value":"The Logstash Plugin in Combination with td-agent","id":"the-logstash-plugin-in-combination-with-td-agent","depth":2},{"value":"More Information","id":"more-information","depth":2}],"frontmatter":{"seo":{"title":"Logstash Import Integration"}},"lastModified":"2026-06-01T09:09:59.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/logstash-import-integration","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}