field.set
Set the value of a certain field.
Description
Set the value of a certain field to any value. It is not allowed to set the .Position
field.
The new value can be a Go template expression, the processor will evaluate the output and assign the value to the target field.
If the provided field
doesn't exist, the processor will create that field and assign its value.
This processor can be used for multiple purposes, like extracting fields, hoisting data, inserting fields, copying fields, masking fields, etc.
Note that this processor only runs on structured data, if the record contains raw JSON data, then use the processor
json.decode
to parse it into structured data first.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "field.set"
settings:
# Field is the target field that will be set. Note that it is not
# allowed to set the `.Position` field.
# For more information about the format, see [Referencing
# fields](https://conduit.io/docs/using/processors/referencing-fields).
# Type: string
field: ""
# Whether to decode the record key using its corresponding schema from
# the schema registry.
# Type: bool
sdk.schema.decode.key.enabled: "true"
# Whether to decode the record payload using its corresponding schema
# from the schema registry.
# Type: bool
sdk.schema.decode.payload.enabled: "true"
# Whether to encode the record key using its corresponding schema from
# the schema registry.
# Type: bool
sdk.schema.encode.key.enabled: "true"
# Whether to encode the record payload using its corresponding schema
# from the schema registry.
# Type: bool
sdk.schema.encode.payload.enabled: "true"
# Value is a Go template expression which will be evaluated and stored
# in `field` (e.g. `{{ .Payload.After }}`).
# Type: string
value: ""
Name | Type | Default | Description |
---|---|---|---|
field | string | null | Field is the target field that will be set. Note that it is not allowed
to set the For more information about the format, see Referencing fields. |
sdk.schema.decode.key.enabled | bool | true | Whether to decode the record key using its corresponding schema from the schema registry. |
sdk.schema.decode.payload.enabled | bool | true | Whether to decode the record payload using its corresponding schema from the schema registry. |
sdk.schema.encode.key.enabled | bool | true | Whether to encode the record key using its corresponding schema from the schema registry. |
sdk.schema.encode.payload.enabled | bool | true | Whether to encode the record payload using its corresponding schema from the schema registry. |
value | string | null | Value is a Go template expression which will be evaluated and stored in |
Examples
Add field
This example adds a new field to the record. The field is
added to .Payload.After
and is set to bar
.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "field.set"
settings:
field: ".Payload.After.foo"
value: "bar"
Name | Value |
---|---|
field | .Payload.After.foo |
value | bar |
Record difference
Before | After | ||||
1 | { | 1 | { | ||
2 | "position": null, | 2 | "position": null, | ||
3 | "operation": "snapshot", | 3 | "operation": "snapshot", | ||
4 | "metadata": null, | 4 | "metadata": null, | ||
5 | "key": { | 5 | "key": { | ||
6 | "my-key": "id" | 6 | "my-key": "id" | ||
7 | }, | 7 | }, | ||
8 | "payload": { | 8 | "payload": { | ||
9 | "before": null, | 9 | "before": null, | ||
10 | - | "after": null | 10 | + | "after": { |
11 | + | "foo": "bar" | |||
12 | + | } | |||
11 | } | 13 | } | ||
12 | } | 14 | } |
Sets the record operation to update
This example sets the .Operation
field to update
for all records.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "field.set"
settings:
field: ".Operation"
value: "update"
Name | Value |
---|---|
field | .Operation |
value | update |
Record difference
Before | After | ||||
1 | { | 1 | { | ||
2 | "position": null, | 2 | "position": null, | ||
3 | - | "operation": "create", | 3 | + | "operation": "update", |
4 | "metadata": null, | 4 | "metadata": null, | ||
5 | "key": null, | 5 | "key": null, | ||
6 | "payload": { | 6 | "payload": { | ||
7 | "before": null, | 7 | "before": null, | ||
8 | "after": null | 8 | "after": null | ||
9 | } | 9 | } | ||
10 | } | 10 | } |
Set field using Go template
This example sets the .Payload.After.postgres
field to true
if the .Metadata.table
field contains postgres
.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "field.set"
settings:
field: ".Payload.After.postgres"
value: "{{ eq .Metadata.table "postgres" }}"
Name | Value |
---|---|
field | .Payload.After.postgres |
value | {{ eq .Metadata.table "postgres" }} |
Record difference
Before | After | ||||
1 | { | 1 | { | ||
2 | "position": null, | 2 | "position": null, | ||
3 | "operation": "snapshot", | 3 | "operation": "snapshot", | ||
4 | "metadata": { | 4 | "metadata": { | ||
5 | "table": "postgres" | 5 | "table": "postgres" | ||
6 | }, | 6 | }, | ||
7 | "key": null, | 7 | "key": null, | ||
8 | "payload": { | 8 | "payload": { | ||
9 | "before": null, | 9 | "before": null, | ||
10 | "after": { | 10 | "after": { | ||
11 | - | "postgres": "false" | 11 | + | "postgres": "true" |
12 | } | 12 | } | ||
13 | } | 13 | } | ||
14 | } | 14 | } |