{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-@l10n/ja/sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"type":"markdown"},"seo":{"title":"Kafka インポート連携とFluentd Consumer","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"kafka-インポート連携とfluentd-consumer","__idx":0},"children":["Kafka インポート連携とFluentd Consumer"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Kafka用のFluentd consumerを使用して、KafkaからTreasure Dataにデータを送信できます。consumer自体に加えて、現在のセットアップによっては、いくつかの追加要件がある場合があります。"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/2020-kafka-import-integration.4fbf2e5d70d6e900644e1ca016b7bb1c74df7f0af9dbb3192095f881859d7d3f.0af5a2b9.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["このドキュメントは、Ubuntu Precise Debianを想定しています。それ以外の場合は、オペレーティングシステムに応じてコマンドを選択する必要があります。"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["この機能はベータ版です。詳細については、",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"mailto:support@treasuredata.com"},"children":["support@treasuredata.com"]},"までお問い合わせください。"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"前提条件","__idx":1},"children":["前提条件"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Treasure Dataの基本知識"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://kafka.apache.org/41/getting-started/quickstart/"},"children":["Apache Kafka"]},"のセットアップ、設定、およびプロデューサーとコンシューマーを含むアーキテクチャに関する基本知識"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Gradleでビルドする能力"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Linuxの実用的な知識"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://www.fluentd.org/"},"children":["Fluentd"]},"の実用的な知識"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["動作しているKafkaシングルノードまたはマルチブローカーのインストール"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"fluentdのインストールと設定","__idx":2},"children":["Fluentdのインストールと設定"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"プレインストール","__idx":3},"children":["プレインストール"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/installation"},"children":["Fluentdのインストール"]},"の事前セットアップおよびインストール手順を確認してください。fluentdをセットアップする前に、以下を確認してください。"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["ntpサーバーをインストールし、/etc/ntp.confに適切なrestrict値を設定"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["ローカルクロックをNTPバックアップとして追加し、ログパラメータも設定"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["NTPサービスデーモンを再起動"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["/etc/security/limits.confで正しいファイルディスクリプタ数を設定"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"fluentdのインストール","__idx":4},"children":["Fluentdのインストール"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Trustyの場合:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["他のディストリビューションについては、Treasure Agent ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/integration-hub/streaming/td-agent/about-treasure-data-s-server-side-agent"},"children":["(td-agent)"]},"のインストールを参照してください。"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"fluentdconfの編集","__idx":5},"children":["Fluentd.confの編集"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ sudo nano /etc/td-agent/td-agent.conf\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Kafka用のFluentd Consumerを使用してイベントを転送します(fluentdはデフォルトポート24224からイベントを受信します)。そのため、設定ファイルは次のようになります。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"source\n  type forward\n</source>\n\n<match td.*.*>\n  type tdlog\n  apikey \"#{ENV['TD_API_KEY']}\"\n\n  auto_create_table\n  buffer_type file\n  buffer_path /var/log/td-agent/buffer/td\n  flush_interval 5s\n\n  secondary\n    type file\n    path /var/log/td-agent/failed_records\n  </secondary>\n</match>\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Treasure Data Write-Only APIキーをマシンの環境変数として設定してください。その後、Fluentdを再起動することをお勧めします。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ /etc/init.d/td-agent restart\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"fluentd用kafka-consumerのインストールと設定","__idx":6},"children":["Fluentd用Kafka Consumerのインストールと設定"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"jarファイルのダウンロード","__idx":7},"children":["JARファイルのダウンロード"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["GitHubのリリースページから、プリコンパイルされたJARファイルをダウンロードします。"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/kafka-fluentd-consumer/releases"},"children":["https://github.com/treasure-data/kafka-fluentd-consumer/releases"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["JARファイル名は",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["kafka-fluentd-consumer-x.y.z-all.jar"]},"です。"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"設定ファイルのダウンロード","__idx":8},"children":["設定ファイルのダウンロード"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["GitHubから2つの設定ファイルもダウンロードする必要があります。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ curl -L https://raw.githubusercontent.com/treasure-data/kafka-fluentd-consumer/master/config/fluentd-consumer.properties -o fluentd-consumer.properties\n$ curl -L https://raw.githubusercontent.com/treasure-data/kafka-fluentd-consumer/master/config/log4j.properties -o log4j.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["fluentd-consumer.properties: Kafkaコンシューマーとログ転送パラメータを設定するために使用"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["log4j.properties: コンシューマー自体のログパラメータを設定するために使用"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["以下は、kafka-fluentd-consumerのファイルリストの例です。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ ls\nkafka-fluentd-consumer-0.2.4-all.jar    fluentd-consumer.properties    log4j.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"ホストでのjavaのアップグレード","__idx":9},"children":["ホストでのJavaのアップグレード"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["少なくともJavaバージョン1.7を実行していることを確認してください。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ java -version\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["そうでない場合は、次の手順を実行してください。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ sudo add-apt-repository ppa:webupd8team/java\n$ sudo apt-get update\n$ sudo apt-get install oracle-java7-installer\n$ sudo apt-get install oracle-java7-set-default\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"kafka-fluentd-consumerの実行","__idx":10},"children":["Kafka Fluentd Consumerの実行"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"zookeeperの起動","__idx":11},"children":["Zookeeperの起動"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/zookeeper-server-start.sh config/zookeeper.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"テストトピックの作成","__idx":12},"children":["テストトピックの作成"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"json形式での複数メッセージの送信","__idx":13},"children":["JSON形式での複数メッセージの送信"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Kafka Fluentd Consumerでは、メッセージをJSON形式で送信する必要があります。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test\n{\"a\": 1}\n{\"a\": 1, \"b\": 2}\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"コンシューマーを起動してメッセージを確認","__idx":14},"children":["コンシューマーを起動してメッセージを確認"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["次のように表示されるはずです。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"{\"a\": 1}\n{\"a\": 1, \"b\": 2}\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"kafka-fluentd-consumerの設定と実行","__idx":15},"children":["Kafka Fluentd Consumerの設定と実行"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["適切な設定でfluentd-consumer.propertiesを変更してください。以下のトピック名を、意図したTreasure Dataのテーブル名に変更してください。fluentd.consumer.topics=test。"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["次のコマンドを実行してコンシューマーを起動します。関連する.jarファイルと.propertiesファイルへのパスを調整する必要がある場合があります。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"$ java -Dlog4j.configuration=file:///path/to/log4j.properties -jar kafka-fluentd-consumer-0.2.1-all.jar fluentd-consumer.properties\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["プロデューサーから追加のイベントを送信できます。ただし、それらは有効なJSON形式である必要があります。"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"treasure-dataでデータをクエリ","__idx":16},"children":["Treasure Dataでデータをクエリ"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Treasure コンソールで次のクエリを実行します。"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"select * from test\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["「Query Result」で実行が成功したかどうかを確認できます。"]}]},"headings":[{"value":"Kafka インポート連携とFluentd Consumer","id":"kafka-インポート連携とfluentd-consumer","depth":1},{"value":"前提条件","id":"前提条件","depth":1},{"value":"Fluentdのインストールと設定","id":"fluentdのインストールと設定","depth":1},{"value":"プレインストール","id":"プレインストール","depth":2},{"value":"Fluentdのインストール","id":"fluentdのインストール","depth":2},{"value":"Fluentd.confの編集","id":"fluentdconfの編集","depth":2},{"value":"Fluentd用Kafka Consumerのインストールと設定","id":"fluentd用kafka-consumerのインストールと設定","depth":1},{"value":"JARファイルのダウンロード","id":"jarファイルのダウンロード","depth":2},{"value":"設定ファイルのダウンロード","id":"設定ファイルのダウンロード","depth":2},{"value":"ホストでのJavaのアップグレード","id":"ホストでのjavaのアップグレード","depth":2},{"value":"Kafka Fluentd Consumerの実行","id":"kafka-fluentd-consumerの実行","depth":1},{"value":"Zookeeperの起動","id":"zookeeperの起動","depth":2},{"value":"テストトピックの作成","id":"テストトピックの作成","depth":2},{"value":"JSON形式での複数メッセージの送信","id":"json形式での複数メッセージの送信","depth":2},{"value":"コンシューマーを起動してメッセージを確認","id":"コンシューマーを起動してメッセージを確認","depth":2},{"value":"Kafka Fluentd Consumerの設定と実行","id":"kafka-fluentd-consumerの設定と実行","depth":2},{"value":"Treasure Dataでデータをクエリ","id":"treasure-dataでデータをクエリ","depth":2}],"frontmatter":{"seo":{"title":"Kafka インポート連携とFluentd Consumer"}},"lastModified":"2026-06-01T09:34:27.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/ja/int/kafka-import-integration-and-fluentd-consumer","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}