{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Kafka Import Integration And Fluentd Consumer","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"kafka-import-integration-and-fluentd-consumer","__idx":0},"children":["Kafka Import Integration And Fluentd Consumer"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can send data from Kafka to Treasure Data, using the Fluentd consumer for Kafka. Other than the consumer itself, and depending on your current setup, there may be a few additional requirements."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/2020-kafka-import-integration.4fbf2e5d70d6e900644e1ca016b7bb1c74df7f0af9dbb3192095f881859d7d3f.0af5a2b9.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This document assumes Ubuntu Precise Debian; commands should otherwise be chosen according to the operating system."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This feature is in Beta. For more information, contact ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"mailto:support@treasuredata.com"},"children":["support@treasuredata.com"]},"."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"prerequisites","__idx":1},"children":["Prerequisites"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Basic knowledge of Treasure Data"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Basic knowledge of ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://kafka.apache.org/41/getting-started/quickstart/"},"children":["Apache Kafka"]}," setup, configuration, and architecture including producers and consumers"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Ability to build with Gradle"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Working knowledge of Linux"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Working knowledge of ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://www.fluentd.org/"},"children":["Fluentd"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["A working Kafka single-node or multi-broker installation"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"install-and-configure-fluentd","__idx":2},"children":["Install and configure Fluentd"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"preinstall","__idx":3},"children":["Preinstall"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Review the pre-setup and installation instructions for ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/installation"},"children":["installing Fluentd"]},". Ensure the following before setting up fluentd:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Installed ntp server and set up proper restrict values in /etc/ntp.conf"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Added the local clock as NTP backup and set also logging parameters"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Restarted the NTP service daemon"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Configured the correct number of file descriptors in /etc/security/limits.conf"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"install-fluentd","__idx":4},"children":["Install Fluentd"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For Trusty:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Refer to Treasure Agent ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/integration-hub/streaming/td-agent/about-treasure-data-s-server-side-agent"},"children":["(td-agent)"]}," installation for other distributions."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"edit-fluentdconf","__idx":5},"children":["Edit Fluentd.conf"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ sudo nano /etc/td-agent/td-agent.conf\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You will be using the Fluentd Consumer for Kafka to forward your events (fluentd receives events from default port of 24224) so your configuration file should look similar to the following:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"source\n  type forward\n</source>\n\n<match td.*.*>\n  type tdlog\n  apikey \"#{ENV['TD_API_KEY']}\"\n\n  auto_create_table\n  buffer_type file\n  buffer_path /var/log/td-agent/buffer/td\n  flush_interval 5s\n\n  secondary\n    type file\n    path /var/log/td-agent/failed_records\n  </secondary>\n</match>\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Set your Treasure Data Write-Only API as an environment variable on your machine. Then, you might want to restart Fluentd:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ /etc/init.d/td-agent restart\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"installing-and-configuring-kafka-consumer-for-fluentd","__idx":6},"children":["Installing and Configuring Kafka Consumer for Fluentd"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"download-the-jar-file","__idx":7},"children":["Download the JAR File"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Download pre-compiled JAR file from the GitHub releases page."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/kafka-fluentd-consumer/releases"},"children":["https://github.com/treasure-data/kafka-fluentd-consumer/releases"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The jar file name is ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["kafka-fluentd-consumer-x.y.z-all.jar"]},"."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"download-configuration-files","__idx":8},"children":["Download Configuration Files"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You also must download two configuration files from GitHub."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ curl -L https://raw.githubusercontent.com/treasure-data/kafka-fluentd-consumer/master/config/fluentd-consumer.properties -o fluentd-consumer.properties\n$ curl -L https://raw.githubusercontent.com/treasure-data/kafka-fluentd-consumer/master/config/log4j.properties -o log4j.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["fluentd-consumer.properties: Use to set Kafka consumer and log forwarding parameters"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["log4j.properties: Use to set logging parameters of consumer itself."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is file list example of kafka-fluentd-consumer:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ ls\nkafka-fluentd-consumer-0.2.4-all.jar    fluentd-consumer.properties    log4j.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"upgrade-java-on-host","__idx":9},"children":["Upgrade Java on Host"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Make sure you are running at least Java version 1.7:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ java -version\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If not, complete the following steps:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ sudo add-apt-repository ppa:webupd8team/java\n$ sudo apt-get update\n$ sudo apt-get install oracle-java7-installer\n$ sudo apt-get install oracle-java7-set-default\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"run-kafka-fluentd-consumer","__idx":10},"children":["Run Kafka Fluentd Consumer"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"start-zookeeper","__idx":11},"children":["Start Zookeeper"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/zookeeper-server-start.sh config/zookeeper.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"create-a-test-topic","__idx":12},"children":["Create a Test Topic"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"send-multiple-messages-in-json-format","__idx":13},"children":["Send Multiple Messages in JSON Format"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The Kafka Fluentd Consumer requires messages to be sent in JSON format."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test\n{\"a\": 1}\n{\"a\": 1, \"b\": 2}\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"confirm-the-messages-by-starting-a-consumer","__idx":14},"children":["Confirm the Messages by Starting a Consumer"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You should see:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"{\"a\": 1}\n{\"a\": 1, \"b\": 2}\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"configure-and-run-the-kafka-fluentd-consumer","__idx":15},"children":["Configure and Run the Kafka Fluentd Consumer"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Modify fluentd-consumer.properties with an appropriate configuration. Change the topic name in the following to your intended Treasure Data table name. fluentd.consumer.topics=test."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Run the command to start the consumer. You may need to adjust this command line to account for the paths to the relevant .jar and .properties files."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ java -Dlog4j.configuration=file:///path/to/log4j.properties -jar kafka-fluentd-consumer-0.2.1-all.jar fluentd-consumer.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can go back and send a few more events from your producer, as long as they are in valid JSON format."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"query-your-data-on-treasure-data","__idx":16},"children":["Query your Data on Treasure Data"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Run the following query on Treasure Console:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"select * from test\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Under Query Result, you can see if the run was successful."]}]},"headings":[{"value":"Kafka Import Integration And Fluentd Consumer","id":"kafka-import-integration-and-fluentd-consumer","depth":1},{"value":"Prerequisites","id":"prerequisites","depth":1},{"value":"Install and configure Fluentd","id":"install-and-configure-fluentd","depth":1},{"value":"Preinstall","id":"preinstall","depth":2},{"value":"Install Fluentd","id":"install-fluentd","depth":2},{"value":"Edit Fluentd.conf","id":"edit-fluentdconf","depth":2},{"value":"Installing and Configuring Kafka Consumer for Fluentd","id":"installing-and-configuring-kafka-consumer-for-fluentd","depth":1},{"value":"Download the JAR File","id":"download-the-jar-file","depth":2},{"value":"Download Configuration Files","id":"download-configuration-files","depth":2},{"value":"Upgrade Java on Host","id":"upgrade-java-on-host","depth":2},{"value":"Run Kafka Fluentd Consumer","id":"run-kafka-fluentd-consumer","depth":1},{"value":"Start Zookeeper","id":"start-zookeeper","depth":2},{"value":"Create a Test Topic","id":"create-a-test-topic","depth":2},{"value":"Send Multiple Messages in JSON Format","id":"send-multiple-messages-in-json-format","depth":2},{"value":"Confirm the Messages by Starting a Consumer","id":"confirm-the-messages-by-starting-a-consumer","depth":2},{"value":"Configure and Run the Kafka Fluentd Consumer","id":"configure-and-run-the-kafka-fluentd-consumer","depth":2},{"value":"Query your Data on Treasure Data","id":"query-your-data-on-treasure-data","depth":2}],"frontmatter":{"seo":{"title":"Kafka Import Integration And Fluentd Consumer"}},"lastModified":"2026-06-01T09:09:59.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/kafka-import-integration-and-fluentd-consumer","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}