# Columns Filter Function

The columns filter plugin for Treasure Data's integrations allows you to add or filter out columns.

* [Options](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#options)
* [add_columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#add_columns)
* [columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#columns)
* [drop_columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#drop_columns)
* [Configuration](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#configuration)
* [Use Case Example: add_columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#use-case-example-add_columns)
* [Use Case Example: columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#use-case-example-columns)
* [Use Case Example: drop_columns](/products/customer-data-platform/integration-hub/batch/import/filter/columns-filter-function#use-case-example-drop_columns)


# Options

This plugin contains the following option:

| **Option** | **Description** |
|  --- | --- |
| add_columns | Columns to add (array of hash) |
| columns | Columns to retain (array of hash) |
| drop_columns | Columns to drop (array of hash) |
| default_timestamp_format | The default timestamp format for timestamp columns (string, default is `%Y-%m-%d %H:%M:%S.%N %z`) |
| default timezone | The default timezone for timestamp columns (string, default is `UTC`) |


## add_columns

| **Child Elements** | **Description** |
|  --- | --- |
| name | The name of the column (required) |
| src | The source column name to be copied (either of `src` or `default` is required) |
| default | The value of column (either of `src` or `default` is required) |
| type | The type of the default value (required for `default`) |
| format | A special option for the timestamp column. Specify the format of the default timestamp (string, default is `default_timestamp_format`) |
| timezone | A special option for the timestamp column. Specify the timezone of the default timestamp (string, default is `default_timezone`) |


## columns

| **Child Elements** | **Description** |
|  --- | --- |
| name | The name of the column (required) |
| src | The source column name to be copied (optional, default is `name`) |
| default | The default value used, if input is null (optional) |
| type | The type of the default value (required for `default`) |
| format | A special option for timestamp column. Specify the format of the default timestamp (string, default is `default_timestamp_format`) |
| timezone | A special option for timestamp column. Specify the timezone of the default timestamp (string, default is `default_timezone`) |


## drop_columns

| **Child Elements** | **Description** |
|  --- | --- |
| name | The name of the column (required) |


# Configuration

Add the `filter` section, including the type, to your `load.yml`. In the following example, the type is `column`:

```yaml
in:
...
filters:
- type: column
  add_columns:
    - {name: copy_id, src: id}
    - {name: email_address_md5, src: email_address}
out:
...
```

# Use Case Example: add_columns

Example of Source data

```csv
time,id,key,score
2015-07-13,0,Vqjht6YE,1370
2015-07-13,1,VmjbjAA0,3962
2015-07-13,2,C40P5H1W,7323
```

Example of `load.yml`

```yaml
in:
...
filters:
- type: column
  add_columns:
    - {name: d, type: timestamp, default: "2017-07-13", format: "%Y-%m-%d"}
    - {name: copy_id, src: id}
out: ...
```

The example adds `d` column, and `copy_id` column which is a copy of `id` column as follows:

```csv
time,id,key,score,d,copy_id
2015-07-13,0,Vqjht6YE,1370,2015-07-13,0
2015-07-13,1,VmjbjAA0,3962,2015-07-13,1
2015-07-13,2,C40P5H1W,7323,2015-07,13,2
```

# Use Case Example: columns

Example of Source data

```csv
time,id,key,score
2015-07-13,0,Vqjht6YE,1370
2015-07-13,1,VmjbjAA0,3962
2015-07-13,2,C40P5H1W,7323
```

Example of `load.yml`

```yaml
in:
...
filters:
- type: column
  columns:
    - {name: time, default: "2017-07-13", format: "%Y-%m-%d"}
    - {name: id}  - {name: key, default: "foo"}
out: ...
```

The example reduces columns to only `time`, `id`, and `key` columns as follows:

```csv
time,id,key
2015-07-13,0,Vqjht6YE
2015-07-13,1,VmjbjAA0
2015-07-13,2,C40P5H1W
```

# Use Case Example: drop_columns

Example of Source data

```csv
time,id,key,score
2015-07-13,0,Vqjht6YE,1370
2015-07-13,1,VmjbjAA0,3962
2015-07-13,2,C40P5H1W,7323
```

Example of `load.yml`

```yaml
in:
...
filters:
- type: column
  drop_columns:
    - {name: time}
    - {name: id}
out: ...
```

Based on the example, the drop `time` and `id` columns are as follows:

```csv
key,score
Vqjht6YE,1370
VmjbjAA0,3962
C40P5H1W,7323
```