The msgpack parser plugin for Treasure Data's Data Connector parses msgpack data. It accepts the following options:
| Option | Description |
|---|---|
row_encoding | type of a row. “array” or “map” (enum, default: map) |
file_encoding | if a file includes a big array, set “array”. Otherwise, if a file includes a sequence of rows, set “sequence” (enum, default: sequence) |
columns | Columns (schema, required) |
The columns option declares the list of columns. This msgpack parser plugin ignores the header line.
| Option | Description |
|---|---|
index | Index number of the column (optional) |
name | Name of the column |
type | Type of the column (see below) |
boolean : true or false
long : 64-bit signed integers
double : 64-bit floating point numbers
string : Strings
timestamp : Date and time with nano-seconds precision
json : JSON dataformat| Format of the timestamp if type is timestamp
You can set the parser section. For example:
in:
...
parser:
type: msgpack
row_encoding: map
file_encoding: sequence
columns:
- {index: 0, name: id, type: long}
- {index: 1, name: name, type: string}
- {index: 2, name: price, type: double}
- {index: 3, name: tag, type: string}
- {index: 4, name: timestamp, type: timestamp, format: '%Y-%m-%d'}
out:
...id,name,price,tag,timestamp
1,"A green door",11.50,"green","2016-01-02"
2,"A blue door",12.50,"blue","2016-01-03"
3,"A red door",13.50,"red","2016-01-04"
4,"A pink door",14.50,"pink","2016-01-05"
5,"A white door",15.50,"white","2016-01-06"
6,"A black door",16.50,"black","2016-01-07"
7,"A yellow door",17.50,"yellow","2016-01-08"
8,"A purple door",18.50,"purple","2016-01-09" in:
...
parser:
type: msgpack
row_encoding: map
file_encoding: sequence
columns:
- {index: 0, name: id, type: long}
- {index: 1, name: name, type: string}
- {index: 2, name: price, type: double}
- {index: 3, name: tag, type: string}
- {index: 4, name: timestamp, type: timestamp, format: '%Y-%m-%d'}
filters: []
out: {mode: append}
exec: {} You can show parsed msgpack data by preview command.
$ td connector:preview load.yml+---------+-----------------+--------------+------------+---------------------------+
| id:long | name:string | price:double | tag:string | timestamp:timestamp |
+---------+-----------------+--------------+------------+---------------------------+
| 1 | "A green door" | 11.5 | "green" | "2016-01-02 00:00:00 UTC" |
| 2 | "A blue door" | 12.5 | "blue" | "2016-01-03 00:00:00 UTC" |
| 3 | "A red door" | 13.5 | "red" | "2016-01-04 00:00:00 UTC" |
| 4 | "A pink door" | 14.5 | "pink" | "2016-01-05 00:00:00 UTC" |
| 5 | "A white door" | 15.5 | "white" | "2016-01-06 00:00:00 UTC" |
| 6 | "A black door" | 16.5 | "black" | "2016-01-07 00:00:00 UTC" |
| 7 | "A yellow door" | 17.5 | "yellow" | "2016-01-08 00:00:00 UTC" |
| 8 | "A purple door" | 18.5 | "purple" | "2016-01-09 00:00:00 UTC" |
+---------+-----------------+--------------+------------+---------------------------+If your data doesn’t have a time column you may add it using add_time filter option. More details at add_time filter plugin.
$ td connector:issue load.yml --database <database name> --table <table name> --time-column timestamp --auto-create-table