Skip to main content

field.set

Set the value of a certain field.

Description​

Set the value of a certain field to any value. It is not allowed to set the .Position field. The new value can be a Go template expression, the processor will evaluate the output and assign the value to the target field. If the provided field doesn't exist, the processor will create that field and assign its value. This processor can be used for multiple purposes, like extracting fields, hoisting data, inserting fields, copying fields, masking fields, etc. Note that this processor only runs on structured data, if the record contains raw JSON data, then use the processor json.decode to parse it into structured data first.

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.set"
        settings:
          # Field is the target field that will be set. Note that it is not
          # allowed to set the `.Position` field.
          # For more information about the format, see [Referencing
          # fields](https://conduit.io/docs/using/processors/referencing-fields).
          # Type: string
          field: ""
          # Whether to decode the record key using its corresponding schema from
          # the schema registry.
          # Type: bool
          sdk.schema.decode.key.enabled: "true"
          # Whether to decode the record payload using its corresponding schema
          # from the schema registry.
          # Type: bool
          sdk.schema.decode.payload.enabled: "true"
          # Whether to encode the record key using its corresponding schema from
          # the schema registry.
          # Type: bool
          sdk.schema.encode.key.enabled: "true"
          # Whether to encode the record payload using its corresponding schema
          # from the schema registry.
          # Type: bool
          sdk.schema.encode.payload.enabled: "true"
          # Value is a Go template expression which will be evaluated and stored
          # in `field` (e.g. `{{ .Payload.After }}`).
          # Type: string
          value: ""

Examples​

Add field​

This example adds a new field to the record. The field is added to .Payload.After and is set to bar.

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.set"
        settings:
          field: ".Payload.After.foo"
          value: "bar"

Record difference​

Before
After
1
{
1
{
2
  "position": null,
2
  "position": null,
3
  "operation": "snapshot",
3
  "operation": "snapshot",
4
  "metadata": null,
4
  "metadata": null,
5
  "key": {
5
  "key": {
6
    "my-key": "id"
6
    "my-key": "id"
7
  },
7
  },
8
  "payload": {
8
  "payload": {
9
    "before": null,
9
    "before": null,
10
-
    "after": null
10
+
    "after": {
11
+
      "foo": "bar"
12
+
    }
11
  }
13
  }
12
}
14
}




Sets the record operation to update​


This example sets the .Operation field to update for all records.


Configuration parameters​


version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.set"
        settings:
          field: ".Operation"
          value: "update"


Record difference​


Before
After
1
{
1
{
2
  "position": null,
2
  "position": null,
3
-
  "operation": "create",
3
+
  "operation": "update",
4
  "metadata": null,
4
  "metadata": null,
5
  "key": null,
5
  "key": null,
6
  "payload": {
6
  "payload": {
7
    "before": null,
7
    "before": null,
8
    "after": null
8
    "after": null
9
  }
9
  }
10
}
10
}




Set field using Go template​


This example sets the .Payload.After.postgres field to true if the .Metadata.table field contains postgres.


Configuration parameters​


version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.set"
        settings:
          field: ".Payload.After.postgres"
          value: "{{ eq .Metadata.table "postgres" }}"


Record difference​


Before
After
1
{
1
{
2
  "position": null,
2
  "position": null,
3
  "operation": "snapshot",
3
  "operation": "snapshot",
4
  "metadata": {
4
  "metadata": {
5
    "table": "postgres"
5
    "table": "postgres"
6
  },
6
  },
7
  "key": null,
7
  "key": null,
8
  "payload": {
8
  "payload": {
9
    "before": null,
9
    "before": null,
10
    "after": {
10
    "after": {
11
-
      "postgres": "false"
11
+
      "postgres": "true"
12
    }
12
    }
13
  }
13
  }
14
}
14
}


scarf pixel conduit-site-docs-using-processors