{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":["admonition"]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Logs Import Using Custom Format","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"logs-import-using-custom-format","__idx":0},"children":["Logs Import Using Custom Format"]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["td-agent was discontinued in December 2023 and has been replaced by ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["fluent-package"]},". The fluent-package is the official successor maintained by the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.cncf.io/projects/"},"children":["Cloud Native Computing Foundation"]},"."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If your logs are in a custom format, you must write a custom parser (",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/plugin-development#customizing-the-tail-input-plugin-parser"},"children":["instructions"]},"). After you write the parser, you put the file into your ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["/etc/fluent/plugin/"]}," directory (for fluent-package)."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"common-parser-examples","__idx":1},"children":["Common Parser Examples"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Treasure Data provides two example parsers: “URL-param style key-value pairs” and “ascii character delimited format”. Both formats are fairly common among our users."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"# URL-param style key-value pairs\nlast_name=smith&first_name=brian&age=22&state=CA\n\n# ASCII character delimited format. In this case, the delimiter is '|'.\n# There is usually a separate file that annotates the column names\nsmith|brian|22|CA\n"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://gist.github.com/2565478"},"children":["Custom Parser for URL-Param Style Key-Value Pairs"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://gist.github.com/2565493"},"children":["Custom Parser for Ascii-Character Delimited Logs"]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Tailing existing logs is an easy way to get started with Treasure Data. We recommend logging everything as JSON. ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://blog.treasuredata.com/blog/2012/04/26/log-everything-as-json/"},"children":["Here's why"]},"."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"filtering-the-records","__idx":2},"children":["Filtering the Records"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you need to filter logs (ex: filtering out impressions and just keeping clicks), ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/v1.0/articles/out_exec_filter"},"children":["the exec-filter plugin"]}," is useful. This plugin launches another script which takes STDIN as input and STDOUT as output, and filters logs accordingly."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here’s an example configuration."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"conf","header":{"controls":{"copy":{}}},"source":"<source>\n  @type tail\n  path /path/to/the/file1\n  tag filter\n  pos_file /var/log/fluent/file1.pos\n  <parse>\n    @type json\n  </parse>\n</source>\n\n<match filter>\n  @type exec_filter\n  command /opt/fluent/bin/ruby /etc/fluent/filter.rb\n  in_format json\n  out_format json\n  tag_key tag\n  time_key time\n</match>\n\n<match td.*.*>\n  @type tdlog\n  endpoint api.treasuredata.com\n  apikey ...\n  auto_create_table\n  use_ssl true\n  <buffer>\n    @type file\n    path /var/log/fluent/buffer/td\n  </buffer>\n</match>\n","lang":"conf"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["/etc/fluent/filter.rb"]}," is the filter script, as shown in the following example. The script filters out all the lines where the field \"field0\" is equal to \"certain_value\". Errors are recorded in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["/var/log/fluent/filter.rb.log"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"ruby","header":{"controls":{"copy":{}}},"source":"open('/var/log/fluent/filter.rb.log', 'a') { |f|\n  f.puts \"-- begin --\"\n  begin\n    require 'json'\n    STDOUT.sync = true\n    while line = STDIN.gets\n      # parse\n      begin\n        h = JSON.parse line\n      rescue => e\n        next # broken line\n      end\n      # filter\n      # next if h[\"field0\"] == \"certain_value\"\n      # emit\n      h['tag'] = 'td.testdb.test_table'\n      puts h.to_json\n    end\n  rescue LoadError => e\n    f.puts e.to_s\n  end\n}\n","lang":"ruby"},"children":[]}]},"headings":[{"value":"Logs Import Using Custom Format","id":"logs-import-using-custom-format","depth":1},{"value":"Common Parser Examples","id":"common-parser-examples","depth":2},{"value":"Filtering the Records","id":"filtering-the-records","depth":2}],"frontmatter":{"seo":{"title":"Logs Import Using Custom Format"}},"lastModified":"2026-02-12T08:44:10.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/logs-import-using-custom-format","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}