{"templateId":"markdown","sharedDataIds":{},"props":{"metadata":{"markdoc":{"tagList":["blog-author","blog-share"]},"redocly_category":"Developer Blog","type":"markdown"},"seo":{"title":"How to prepare simple test data for Hive and Presto","description":"This article show how to prepare simple data for Hive/Presto testing without using the actual table","siteUrl":"https://docs.treasure.ai","lang":"en-US","jsonLd":{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.treasure.ai/","name":"Treasure AI","url":"https://www.treasure.ai/","logo":"https://www.treasure.ai/hubfs/assets/images/logos/primary-logo.svg"},{"@type":"WebSite","@id":"https://docs.treasure.ai/#website","name":"Treasure AI Documentation","url":"https://docs.treasure.ai/","inLanguage":["en","ja"],"publisher":{"@id":"https://www.treasure.ai/"}}]},"llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]},"image":"/assets/getty2.4d4474eea4571914f82e9c292ddb038dc04c449a020afa63f4b358647be0f3be.978384e4.jpg"},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"how-to-prepare-simple-test-data-for-hive-and-presto","__idx":0},"children":["How to prepare simple test data for Hive and Presto"]},{"$$mdtype":"Tag","name":"BlogAuthor","attributes":{"name":"Kazuki Ito","date":"2023-06-01","image":"/assets/default.416e2ccda21c12d40a43f43bc53adffb2ae15dab198bd54da325c0d1f43da13d.978384e4.png"},"children":[]},{"$$mdtype":"Tag","name":"BlogShare","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["As you might know, Treasure Data provides two types of query engines: Hive and Presto. Since the engines are different, there are natural differences regarding the syntax of query statements and supported functions. Of course, the basic syntax of typical functions like ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["COUNT"]}," are the same. Although you can learn about the behavior of each function and syntax by looking at the documentation, I imagine that most of you will probably want to actually run the Hive or Presto job before implementation to production."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["How, then, do you prepare your test data? Even in cases where a few rows of records are sufficient for testing, it is very troublesome to create an empty table and insert rows using the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INSERT"]}," statement or to import a table via a bulk load job (e.g. using a Data Connector). Also, from an operational standpoint, it is not good to have an increasing number of tables that are used for testing even if the tables are deleted after the test is completed."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Therefore, let me show you in this article how to easily check the behavior of Hive or Presto jobs without creating a test table."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"case-1-when-one-row-is-enough-for-testing","__idx":1},"children":["Case 1: When one row is enough for testing"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["First, let me show this small test case. Suppose you wanted to use the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://trinodb.github.io/docs.trino.io/350/functions/string.html#split"},"children":["SPLIT"]}," function in Presto. The documentation says the following:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/prestosplitfunction.e34a4227b13ea709e36b274c84c088a4d4c2e066c2f3921b3f81fd1df2c9e6f3.978384e4.png","alt":"Presto Split Function","title":"#width=500px;"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you don't know this function, it might be better to confirm the behavior before you use it in production query statements. Usually, you might create a new test table and store records in it for testing. After that you would run the following query statement."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT SPLIT(col1, ',') as split_col\n  FROM table_a\n WHERE col2=100;\n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Of course, this will work for you. However, I think that it is excessive to create a new table in order to check the behavior of a function or a simple test case. Instead, consider passing a string directly to the argument of the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["SPLIT"]}," function instead of specifying a column name? It is easy to assume that a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["FROM"]}," clause is required, but the query statement works without it. Here is an example:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT SPLIT('aaaa,bbbb,cccc,dddd', ',') as split_col;\n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To see an example of the output of this query from the Treasure Console, go to ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data Workbench > Queries > New Query"]},". Select a database, and then enter your SQL query. Then select ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Run"]},". Here is an example of the output using Treasure Console:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/onelinesqlconsole.f47412f5c86e3a00561fa0459d767309bbb6e08bac4a3351fbe7ce304d35657f.978384e4.png","alt":"Presto One Line SQL Test","title":"#width=800px;"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is an example of the output using the Treasure CLI or cURL:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Curl Example","header":{"title":"Curl Example","controls":{"copy":{}}},"source":"curl --location --request POST 'https://api.treasuredata.com/v3/job/issue/presto/sample_test_db' --header 'Accept: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789' -d \"query=SELECT SPLIT('aaaa,bbbb,cccc,dddd', ',') as split_col;\"\n\n> {\"job\":\"1838627512\",\"job_id\":\"1838627512\",\"database\":\"sample_test_db\"}\n\n\n\n\ncurl --location 'https://console.treasuredata.com/v3/job/result/1838627512' --header 'Content-Type: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789'\n\n> \"[\"\"aaaa\"\", \"\"bbbb\"\", \"\"cccc\"\", \"\"dddd\"\"]\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Treasure CLI Example ","header":{"title":"Treasure CLI Example ","controls":{"copy":{}}},"source":"td query -d  sample_db_test -T presto -w \"SELECT SPLIT('aaaa,bbbb,cccc,dddd', ',') as split_col;\"\n\n>\nJob 1838376808 is queued.\nUse 'td job:show 1838376808' to show the status.\n  started at 2023-06-06T19:43:51Z\n  presto version: 350\n  executing query: SELECT SPLIT('aaaa,bbbb,cccc,dddd', ',') as split_col\n  .\n  .\n  .\n  .\nStatus      : success\nResult      :\nWARNING: the job result is being downloaded...\n\n+-------------------------------+\n| split_col                     |\n+-------------------------------+\n| [\"aaaa\",\"bbbb\",\"cccc\",\"dddd\"] |\n+-------------------------------+\n1 row in set\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"case2-when-multiple-rows-are-required","__idx":2},"children":["Case2: When multiple rows are required"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you would like to check the behavior of an aggregation function via a query, a single row is not enough for testing. Therefore, you need to prepare multiple rows as test data. Let me show you how to achieve this without creating a test table using Presto and Hive."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"presto-case","__idx":3},"children":["Presto Case"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In the case of Presto, you can prepare multiple rows by using ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["VALUES"]},". You can define the multiple rows after the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["VALUES"]}," in the form ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["(<first_row>), (second_row), ..., (nth_row)"]},". Regarding column names, you can define them by aliasing them with ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AS"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT *\n  FROM (\n    VALUES\n      (1, 100, 'test1'),  -- first row\n      (1, 200, 'test2'),  -- second row\n      (2, 200, 'test3'),  -- third row\n      (2, 400, 'test4'),  -- fourth row\n      (2, 600, 'test5')   -- fifth row\n       ) AS t(col1, col2, col3)  -- define column names\n;\n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is an example of the output using Treasure Console:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/multlinesqlconsolepresto.39f92d01c41f21ae90739b5c5b26ac03a436df19a07f8b60803a8a9f2fcb122d.978384e4.png","alt":"Presto Multiple Line SQL Test","title":"#width=800px;"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is an example of the output using the Treasure CLI or cURL:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Curl Example","header":{"title":"Curl Example","controls":{"copy":{}}},"source":"curl --location --request POST 'https://api.treasuredata.com/v3/job/issue/presto/sample_test_db' --header 'Accept: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789' -d \"query=SELECT * FROM (VALUES (1, 100, 'test1'),(1, 200, 'test2'),(2, 200, 'test3'),(2, 400, 'test4'),(2, 600, 'test5') ) AS t(col1, col2, col3);\"\n\n> {\"job\":\"1838631936\",\"job_id\":\"1838631936\",\"database\":\"sample_test_db\"}\n\ncurl --location 'https://console.treasuredata.com/v3/job/result/1838631936' --header 'Content-Type: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789'\n\n>\n1,100,test1\n1,200,test2\n2,200,test3\n2,400,test4\n2,600,test5\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Treasure CLI Example","header":{"title":"Treasure CLI Example","controls":{"copy":{}}},"source":"td query -d  sample_db_test -T presto -w \"SELECT * FROM (VALUES (1, 100, 'test1'),(1, 200, 'test2'),(2, 200, 'test3'),(2, 400, 'test4'),(2, 600, 'test5')) AS t(col1, col2, col3)\"\n\n>\nJob 1838434823 is queued.\nUse 'td job:show 1838434823' to show the status.\n  started at 2023-06-06T21:00:40Z\n  presto version: 350\n  executing query: SELECT * FROM (VALUES (1, 100, 'test1'),(1, 200, 'test2'),(2, 200, 'test3'),(2, 400, 'test4'),(2, 600, 'test5')) AS t(col1, col2, col3)\n  .\n  .\n  .\n  .\nStatus      : success\nResult      :\nWARNING: the job result is being downloaded...\n+------+------+-------+\n| col1 | col2 | col3  |\n+------+------+-------+\n| 1    | 100  | test1 |\n| 1    | 200  | test2 |\n| 2    | 200  | test3 |\n| 2    | 400  | test4 |\n| 2    | 600  | test5 |\n+------+------+-------+\n5 rows in set\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The advantage of this approach is that you can share the example with your colleague without the existing table."," ","Of course, if you want a few dozen rows of test data, it may be easier to prepare a table instead of this example."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"hive-case","__idx":4},"children":["Hive Case"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In the case of Hive, you can prepare multiple rows by using ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["STACK"]},". Unlike ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["VALUES"]}," in Presto, you need to specify the number of rows as the first argument and not enclose each row in parenthesis."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT *\n  FROM (SELECT STACK(5,  -- the number of rows\n                     1, 100, 'test1',  -- first row\n                     1, 200, 'test2',  -- second row\n                     2, 200, 'test3',  -- third row\n                     2, 400, 'test4',  -- fourth row\n                     2, 600, 'test5'   -- fifth row\n                    ) AS (col1, col2, col3)  -- define column names\n       ) AS t\n;\n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is an example of the output using Treasure Console:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/multlinesqlconsolehive.379460897cb34459926b4eb85afb9fbdc1cbf38da42bc7ec2ccf593d13ec486d.978384e4.png","alt":"Hive Multiple Line SQL Test","title":"#width=800px;"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here is an example of the output using the Treasure CLI or cURL:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Curl Example","header":{"title":"Curl Example","controls":{"copy":{}}},"source":"curl --location --request POST 'https://api.treasuredata.com/v3/job/issue/hive/sample_test_db' --header 'Accept: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789' -d \"query=SELECT * FROM (SELECT STACK (5,1, 100, 'test1',1, 200, 'test2',2, 200, 'test3',2, 400, 'test4',2, 600, 'test5') AS (col1, col2, col3) ) AS t;\"\n\n> {\"job\":\"1838639145\",\"job_id\":\"1838639145\",\"database\":\"sample_test_db\"}\n\ncurl --location 'https://console.treasuredata.com/v3/job/result/1838639145' --header 'Content-Type: application/json' --header 'Authorization: TD1 123/abcdef•••••••••••••••••••••••0123456789'\n\n>\n1,100,test1\n1,200,test2\n2,200,test3\n2,400,test4\n2,600,test5\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","data-title":"Treasure CLI Example","header":{"title":"Treasure CLI Example","controls":{"copy":{}}},"source":"td query -d  sample_db_test -T hive -w \"SELECT * FROM (SELECT STACK(5,1, 100,'test1',1, 200,'test2',2, 200,'test3',2, 400,'test4',2, 600,'test5')AS(col1, col2, col3)) AS t;\"\n\n>\nJob 1838369237 is queued.\nUse 'td job:show 1838369237' to show the status.\n  Hive history file=/mnt/hive/tmp/3867/hive_job_log_5c427fc1-ee26-4e52-9d8f-fa0c308e206e_123537841.txt\n  Job is running in resource pool: hadoop2 with priority: default\n  Execution engine version: 2020.1\n  TD Hive worker version: release-0.2.4-11028\n  .\n  .\n  .\n  .\nStatus      : success\nResult      :\n+------+------+-------+\n| col1 | col2 | col3  |\n+------+------+-------+\n| 1    | 100  | test1 |\n| 1    | 200  | test2 |\n| 2    | 200  | test3 |\n| 2    | 400  | test4 |\n| 2    | 600  | test5 |\n+------+------+-------+\n5 rows in set\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"summary","__idx":5},"children":["Summary"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The techniques described in this article will help you when you attempt to confirm the behavior of a function or query statement using either Hive or Presto. When you encounter unexpected behavior, use these methods to reproduce the behavior, and then you can share the query statement with your team or with support."]}]},"headings":[{"value":"How to prepare simple test data for Hive and Presto","id":"how-to-prepare-simple-test-data-for-hive-and-presto","depth":1},{"value":"Case 1: When one row is enough for testing","id":"case-1-when-one-row-is-enough-for-testing","depth":1},{"value":"Case2: When multiple rows are required","id":"case2-when-multiple-rows-are-required","depth":1},{"value":"Presto Case","id":"presto-case","depth":2},{"value":"Hive Case","id":"hive-case","depth":2},{"value":"Summary","id":"summary","depth":1}],"frontmatter":{"title":"How to prepare simple test data for Hive and Presto","author":"Kazuki Ito","date":"2023-06-01T00:00:00.000Z","categories":["best-practice","hive","presto"],"image":"getty2.jpg","seo":{"title":"How to prepare simple test data for Hive and Presto","description":"This article show how to prepare simple data for Hive/Presto testing without using the actual table","image":"/assets/getty2.4d4474eea4571914f82e9c292ddb038dc04c449a020afa63f4b358647be0f3be.978384e4.jpg"}},"lastModified":"2026-06-04T10:17:23.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/blog/prepare_test_data_for_query","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}