Skip to main content

field.exclude

Remove a subset of fields from the record.

Description​

Remove a subset of fields from the record, all the other fields are left untouched. If a field is excluded that contains nested data, the whole tree will be removed. It is not allowed to exclude .Position or .Operation fields.

Note that this processor only runs on structured data, if the record contains raw JSON data, then use the processor json.decode to parse it into structured data first.

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.exclude"
        settings:
          # Fields is a comma separated list of target fields which should be
          # excluded.
          # For more information about the format, see [Referencing
          # fields](https://conduit.io/docs/using/processors/referencing-fields).
          # Type: string
          fields: ""
          # Whether to decode the record key using its corresponding schema from
          # the schema registry.
          # Type: bool
          sdk.schema.decode.key.enabled: "true"
          # Whether to decode the record payload using its corresponding schema
          # from the schema registry.
          # Type: bool
          sdk.schema.decode.payload.enabled: "true"
          # Whether to encode the record key using its corresponding schema from
          # the schema registry.
          # Type: bool
          sdk.schema.encode.key.enabled: "true"
          # Whether to encode the record payload using its corresponding schema
          # from the schema registry.
          # Type: bool
          sdk.schema.encode.payload.enabled: "true"

Examples​

Exclude all fields in payload​

Excluding all fields in .Payload results in an empty payload.

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.exclude"
        settings:
          fields: ".Payload"

Record difference​

Before
After
1
{
1
{
2
  "position": null,
2
  "position": null,
3
  "operation": "create",
3
  "operation": "create",
4
  "metadata": {
4
  "metadata": {
5
    "key1": "val1"
5
    "key1": "val1"
6
  },
6
  },
7
  "key": null,
7
  "key": null,
8
  "payload": {
8
  "payload": {
9
-
    "before": {
9
+
    "before": null,
10
-
      "bar": "baz"
10
+
    "after": null
11
-
    },
12
-
    "after": {
13
-
      "foo": "bar"
14
-
    }
15
  }
11
  }
16
}
12
}




Exclude multiple fields​


It's possible to exclude multiple fields by providing a
comma-separated list of fields. In this example, we exclude .Metadata,
.Payload.After.foo and .Key.key1.


Configuration parameters​


version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "field.exclude"
        settings:
          fields: ".Metadata,.Payload.After.foo,.Key.key1"


Record difference​


Before
After
1
{
1
{
2
  "position": null,
2
  "position": null,
3
  "operation": "create",
3
  "operation": "create",
4
-
  "metadata": {
4
+
  "metadata": {},
5
-
    "source": "s3"
6
-
  },
7
  "key": {
5
  "key": {
8
-
    "key1": "val1",
9
    "key2": "val2"
6
    "key2": "val2"
10
  },
7
  },
11
  "payload": {
8
  "payload": {
12
    "before": {
9
    "before": {
13
      "bar": "baz"
10
      "bar": "baz"
14
    },
11
    },
15
    "after": {
12
    "after": {
16
-
      "foo": "bar",
17
      "foobar": "baz"
13
      "foobar": "baz"
18
    }
14
    }
19
  }
15
  }
20
}
16
}


scarf pixel conduit-site-docs-using-processors