{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Nginx Access Logs","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"nginx-access-logs","__idx":0},"children":["Nginx Access Logs"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can tail Nginx access logs from Treasure Agent (td-agent), to continuously import the access logs into the cloud."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Continue to the following topics:"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"installing-td-agent","__idx":1},"children":["Installing td-agent"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["‘td-agent’ must be installed on your application servers. td-agent is a daemon program dedicated to the streaming upload of any kind of the time-series data. td-agent is developed and maintained by Treasure Data."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To set up td-agent, refer to the following articles; we provide deb/rpm packages for Linux systems."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"OS"},"children":["OS"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Link"},"children":["Link"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["MacOS X"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/installation/install-by-dmg"},"children":["Installing td-agent on MacOS X"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Ubuntu System"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.fluentd.org/installation/install-by-deb"},"children":["Installing td-agent for Debian and Ubuntu"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["RHEL / CentOS System"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"http://docs.fluentd.org/installation/install-by-rpm"},"children":["Installing td-agent for Redhat and CentOS"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["AWS Elastic Beanstalk"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/elastic-beanstalk-td-agent"},"children":["Installing td-agent on AWS Elastic Beanstalk"]}]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"modifying-td-agentconf","__idx":2},"children":["Modifying td-agent.conf"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Specify your authentication key by setting the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apikey"]}," option. You can view your API key from the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://console.treasuredata.com/app/users/current"},"children":["Treasure Console"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Access /etc/td-agent/td-agent.conf to set the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apikey"]}," option in your td-agent.conf file."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Note: ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["YOUR_API_KEY"]}," should be your API key string."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"conf","header":{"controls":{"copy":{}}},"source":"# Tailing the Apache Log\nsource\n  type tail\n  path /var/log/nginx/access.log\n  pos_file /var/log/td-agent/nginx-access.pos\n  tag td.production.access\n  format nginx\n</source>\n\n# Treasure Data Input and Output\n<match td.*.*>\n  type tdlog\n  endpoint api.treasuredata.com\n  apikey YOUR_API_KEY\n  auto_create_table\n  buffer_type file\n  buffer_path /var/log/td-agent/buffer/td\n  use_ssl true\n</match>\n","lang":"conf"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Restart your agent when the following lines are in place."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ sudo /etc/init.d/td-agent restart\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["td-agent keeps tailing the log, buffers the log (",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["var/log/td-agent/buffer/td"]},"), and automatically uploads the log into the cloud."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"confirm-data-import","__idx":3},"children":["Confirm Data Import"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Sending a SIGUSR1 signal flushes td-agent’s buffer; upload starts immediately."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"# generate access logs\n$ curl http://host:port/\n\n# flush the buffer\n$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To confirm that your data has been uploaded successfully, issue the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td tables"]}," command as follows."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"$ td tables\n+------------+------------+------+-----------+\n| Database   | Table      | Type | Count     |\n+------------+------------+------+-----------+\n| production | access     | log  | 1         |\n+------------+------------+------+-----------+\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Check ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["/var/log/td-agent.log"]}," if it’s not working correctly. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["td-agent:td-agent"]}," must have permission to access the logs."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["td-agent handles log-rotation"]},". td-agent keeps a record of the last position of the log, ensuring that each line is read exactly once even if the td-agent process goes down. However, because the information is kept in a file, the \"exactly once\" guarantee breaks down if the file becomes corrupted."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"next-steps","__idx":4},"children":["Next Steps"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language."]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/data-workbench/databases/schema-management"},"children":["Schema Management"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/customer-data-platform/data-workbench/queries/hive/quickstart"},"children":["Hive Query Language"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/apis/td-api"},"children":["Programmatic Access with REST API and its Bindings"]}]}]}]},"headings":[{"value":"Nginx Access Logs","id":"nginx-access-logs","depth":1},{"value":"Installing td-agent","id":"installing-td-agent","depth":2},{"value":"Modifying td-agent.conf","id":"modifying-td-agentconf","depth":2},{"value":"Confirm Data Import","id":"confirm-data-import","depth":2},{"value":"Next Steps","id":"next-steps","depth":2}],"frontmatter":{"seo":{"title":"Nginx Access Logs"}},"lastModified":"2026-06-01T09:09:59.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/nginx-access-logs","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}