Calcite Filter Function

This filter function allows translating rows by SQL queries in Pages received from input plugin and sending the query results to next filter or output plugin as modified Pages. It uses Apache Calcite, which is the foundation for your next high-performance database and enables executing SQL queries to customized storage by the custom adaptor. The plugin applies Page storage adaptor to Apache Calcite and then enables executing SQL queries to Pages via JDBC Driver provided.

This filter uses memory to store data before processing. The memory is managed by Embulk on the host machine.

• Basic Knowledge of Treasure Data.

• Basic knowledge of TD Toolbelt

• The filter plugin aims to support a flexible data filter to remove unnecessary records during data ingestion. Thus, only WHERE clause is well supported.
• Other use cases such joining data and inserting data into external services are not supported.

Here is an Embulk configuration example for this plugin:

filters:
  - type: calcite
    query: SELECT * FROM $PAGES

The Calcite Filter cannot be applied via Treasure Console. It must be used via the Treasure CLI.

To use the Calcite Filter with Treasure CLI, do the following.

1. Create a config.yml file that includes a filters sections. Here is an example using an S3 Input connector:

in:
  type: s3
  access_key_id: sample_access_key_id
  secret_access_key: sample_secret_access_key
  bucket: viet-us-east-1
  path_prefix: s3_basic.csv
  parser:
    type: csv
    stop_on_invalid_record: false
    allow_optional_columns: true
    allow_extra_columns: true
    columns:
      - {name: path, type: string}
      - {name: code, type: string}
      - {name: method, type: string}
out:
  mode: replace
exec: {}
filters:
  - type: calcite
    query: SELECT * FROM $PAGES WHERE "method" LIKE 'GET' and "path" like '%item%'
  - type: add_time
    to_column:
      name: time
      type: timestamp
    from_value:
      mode: upload_time

td connector:issue config.yml \
--database vietdb \
--table test_cal --auto-create-table