{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"redocly_category":"Integrations","type":"markdown"},"seo":{"title":"Amazon Kinesis Import Integration","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"amazon-kinesis-import-integration","__idx":0},"children":["Amazon Kinesis Import Integration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Amazon Kinesis is a platform for streaming data on AWS, offering load and analysis of streaming data, and also providing the ability for you to build custom streaming data applications."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This article explains how to ingest data from Amazon Kinesis Stream to Treasure Data, by using AWS Lambda."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"prerequisites","__idx":1},"children":["Prerequisites"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Basic knowledge of Treasure Data."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Basic knowledge of Amazon Kinesis"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"retrieve-master-api-key","__idx":2},"children":["Retrieve Master API key"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/products/my-settings/getting-your-api-keys"},"children":["get the master key"]}," from your Treasure Console."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"set-up-aws-lambda-function","__idx":3},"children":["Set up AWS Lambda function"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://aws.amazon.com/lambda/"},"children":["AWS Lambda"]}," is a part of the data ingestion pipeline. By using AWS Lambda, you can execute code in response to triggers from Amazon Kinesis."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"select-blueprint","__idx":4},"children":["Select Blueprint"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select kinesis-process-record-python."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image-20191206-001703.7b5e0fc793f7bd5d0f760ff7b3ec3924e96307c1b40c0dd1e07ffca4a86b3374.9035ab13.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"configure-event-sources","__idx":5},"children":["Configure Event Sources"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select Kinesis as Event source type."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Specify the name of stream as Kinesis Stream."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image-20191206-001744.a7daf1169c2f8f31d97b45af8bef93679604ce0164d23e64c27cf569fc3e8b93.9035ab13.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"configure-function","__idx":6},"children":["Configure Function"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Specify Name, Description."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Select Python 3.x as Runtime."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image-20191206-001809.6eb77e0b0d5bc0f498a7e3d67ecfa15c352d2d0fc9ad464cf1f5e8b42ac3aea8.9035ab13.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"configure-lambda_handler-python-script","__idx":7},"children":["Configure Lambda_handler Python Script"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Streaming events in Kinesis are processed by python function lambda_handler.py and TreasureData provides an example script that imports Kinesis Firehose data stream events as one of the solutions from Treasure Boxes."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Copy and paste the Python script from ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/treasure-data/treasure-boxes/tree/master/integration-box/Kinesis"},"children":["Treasure Boxes link"]},". README file should give more details about steps to run."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"review-the-function","__idx":8},"children":["Review the Function"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Review your configuration."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Create function"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/image-20191206-002721.06c7862bdb944c11f25917d1b86bb126dd9e6ac71cac2d8397217021546a539a.9035ab13.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"test-the-function","__idx":9},"children":["Test the Function"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["After configuration, as mentioned in the Treasure Boxes, you can test the function with the following event sample."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You can use the following records for the one-time testing from Lambda UI (Actions > Configure test event)."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"{\n  \"invocationId\": \"invocationIdExample\",\n  \"deliveryStreamArn\": \"arn:aws:kinesis:EXAMPLE\",\n  \"region\": \"us-west-2\",\n  \"records\": [\n    {\n      \"recordId\": \"49546986683135544286507457936321625675700192471156785154\",\n      \"approximateArrivalTimestamp\": 1495072949453,\n      \"data\": \"eyJmb28iOiAiYmFyIn0=\"\n    }\n  ]\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The data part of the records resolves to {\"foo\": bar}, with base64 encoding."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"confirm-data-upload","__idx":10},"children":["Confirm Data Upload"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To confirm that the data imported successfully:"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Navigate to Treasure Console > Databases."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Data digestion takes 1-3 minutes to complete."]}]},"headings":[{"value":"Amazon Kinesis Import Integration","id":"amazon-kinesis-import-integration","depth":1},{"value":"Prerequisites","id":"prerequisites","depth":2},{"value":"Retrieve Master API key","id":"retrieve-master-api-key","depth":2},{"value":"Set up AWS Lambda function","id":"set-up-aws-lambda-function","depth":2},{"value":"Select Blueprint","id":"select-blueprint","depth":3},{"value":"Configure Event Sources","id":"configure-event-sources","depth":3},{"value":"Configure Function","id":"configure-function","depth":3},{"value":"Configure Lambda_handler Python Script","id":"configure-lambda_handler-python-script","depth":3},{"value":"Review the Function","id":"review-the-function","depth":3},{"value":"Test the Function","id":"test-the-function","depth":3},{"value":"Confirm Data Upload","id":"confirm-data-upload","depth":2}],"frontmatter":{"seo":{"title":"Amazon Kinesis Import Integration"}},"lastModified":"2026-06-01T09:09:59.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/int/amazon-kinesis-import-integration","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}