webhook.http
Trigger an HTTP request for every record.
Description
A processor that sends an HTTP request to the specified URL, retries on error and saves the response body and, optionally, the response status.
A status code over 500 is regarded as an error and will cause the processor to retry the request. The processor will retry the request according to the backoff configuration.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "webhook.http"
settings:
# Maximum number of retries for an individual record when backing off
# following an error.
# Type: float
backoffRetry.count: "0"
# The multiplying factor for each increment step.
# Type: float
backoffRetry.factor: "2"
# The maximum waiting time before retrying.
# Type: duration
backoffRetry.max: "5s"
# The minimum waiting time before retrying.
# Type: duration
backoffRetry.min: "100ms"
# Headers to add to the request, use `headers.*` to specify the header
# and its value (e.g. `headers.Authorization: "Bearer key"`).
# Type: string
headers.*: ""
# Specifies the body that will be sent in the HTTP request. The field
# accepts a Go [templates](https://pkg.go.dev/text/template) that's
# evaluated using the
# [opencdc.Record](https://pkg.go.dev/github.com/conduitio/conduit-commons/opencdc#Record)
# as input. By default, the body is empty.
# To send the whole record as JSON you can use `{{ toJson . }}`.
# Type: string
request.body: ""
# Deprecated: use `headers.Content-Type` instead.
# Type: string
request.contentType: ""
# Method is the HTTP request method to be used.
# Type: string
request.method: "GET"
# URL is a Go template expression for the URL used in the HTTP
# request, using Go [templates](https://pkg.go.dev/text/template). The
# value provided to the template is
# [opencdc.Record](https://pkg.go.dev/github.com/conduitio/conduit-commons/opencdc#Record),
# so the template has access to all its fields (e.g. `.Position`,
# `.Key`, `.Metadata`, and so on). We also inject all template
# functions provided by [sprig](https://masterminds.github.io/sprig/)
# to make it easier to write templates.
# Type: string
request.url: ""
# Specifies in which field should the response body be saved.
# For more information about the format, see [Referencing
# fields](https://conduit.io/docs/using/processors/referencing-fields).
# Type: string
response.body: ".Payload.After"
# Specifies in which field should the response status be saved. If no
# value is set, then the response status will NOT be saved.
# For more information about the format, see [Referencing
# fields](https://conduit.io/docs/using/processors/referencing-fields).
# Type: string
response.status: ""
# Whether to decode the record key using its corresponding schema from
# the schema registry.
# Type: bool
sdk.schema.decode.key.enabled: "true"
# Whether to decode the record payload using its corresponding schema
# from the schema registry.
# Type: bool
sdk.schema.decode.payload.enabled: "true"
# Whether to encode the record key using its corresponding schema from
# the schema registry.
# Type: bool
sdk.schema.encode.key.enabled: "true"
# Whether to encode the record payload using its corresponding schema
# from the schema registry.
# Type: bool
sdk.schema.encode.payload.enabled: "true"
Name | Type | Default | Description |
---|---|---|---|
backoffRetry.count | float | 0 | Maximum number of retries for an individual record when backing off following an error. |
backoffRetry.factor | float | 2 | The multiplying factor for each increment step. |
backoffRetry.max | duration | 5s | The maximum waiting time before retrying. |
backoffRetry.min | duration | 100ms | The minimum waiting time before retrying. |
headers.* | string | null | Headers to add to the request, use |
request.body | string | null | Specifies the body that will be sent in the HTTP request. The field accepts a Go templates that's evaluated using the opencdc.Record as input. By default, the body is empty. To send the whole record as JSON you can use |
request.contentType | string | null | Deprecated: use |
request.method | string | GET | Method is the HTTP request method to be used. |
request.url | string | null | URL is a Go template expression for the URL used in the HTTP request, using Go templates.
The value provided to the template is opencdc.Record,
so the template has access to all its fields (e.g. |
response.body | string | .Payload.After | Specifies in which field should the response body be saved. For more information about the format, see Referencing fields. |
response.status | string | null | Specifies in which field should the response status be saved. If no value is set, then the response status will NOT be saved. For more information about the format, see Referencing fields. |
sdk.schema.decode.key.enabled | bool | true | Whether to decode the record key using its corresponding schema from the schema registry. |
sdk.schema.decode.payload.enabled | bool | true | Whether to decode the record payload using its corresponding schema from the schema registry. |
sdk.schema.encode.key.enabled | bool | true | Whether to encode the record key using its corresponding schema from the schema registry. |
sdk.schema.encode.payload.enabled | bool | true | Whether to encode the record payload using its corresponding schema from the schema registry. |
Examples
Send a request to an HTTP server
This example shows how to use the HTTP processor to send a record's .Payload.After
field as a string to a dummy
HTTP server that replies back with a greeting.
The record's .Payload.After
is overwritten with the response. Additionally, the example shows how to set a request
header and how to store the value of the HTTP response's code in the metadata field http_status
.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "webhook.http"
settings:
backoffRetry.count: "0"
backoffRetry.factor: "2"
backoffRetry.max: "5s"
backoffRetry.min: "100ms"
headers.content-type: "application/json"
request.body: "{{ printf "%s" .Payload.After }}"
request.method: "GET"
request.url: "http://127.0.0.1:54321"
response.body: ".Payload.After"
response.status: ".Metadata["http_status"]"
Name | Value |
---|---|
backoffRetry.count | 0 |
backoffRetry.factor | 2 |
backoffRetry.max | 5s |
backoffRetry.min | 100ms |
headers.content-type | application/json |
request.body | {{ printf "%s" .Payload.After }} |
request.method | GET |
request.url | http://127.0.0.1:54321 |
response.body | .Payload.After |
response.status | .Metadata["http_status"] |
Record difference
Before | After | ||||
1 | { | 1 | { | ||
2 | "position": "cG9zLTE=", | 2 | "position": "cG9zLTE=", | ||
3 | "operation": "update", | 3 | "operation": "update", | ||
4 | - | "metadata": null, | 4 | + | "metadata": { |
5 | + | "http_status": "200" | |||
6 | + | }, | |||
5 | "key": null, | 7 | "key": null, | ||
6 | "payload": { | 8 | "payload": { | ||
7 | "before": null, | 9 | "before": null, | ||
8 | - | "after": "world" | 10 | + | "after": "hello, world" |
9 | } | 11 | } | ||
10 | } | 12 | } |
Send a request to an HTTP server with a dynamic URL
This example shows how to use the HTTP processor to use a record's .Payload.After.name
field in the URL path,
send it to a dummy HTTP server, and get a greeting with the name back.
The response will be written under the record's .Payload.After.response
.
Configuration parameters
- YAML
- Table
version: 2.2
pipelines:
- id: example
status: running
connectors:
# define source and destination ...
processors:
- id: example
plugin: "webhook.http"
settings:
backoffRetry.count: "0"
backoffRetry.factor: "2"
backoffRetry.max: "5s"
backoffRetry.min: "100ms"
request.method: "GET"
request.url: "http://127.0.0.1:54321/{{.Payload.After.name}}"
response.body: ".Payload.After.response"
Name | Value |
---|---|
backoffRetry.count | 0 |
backoffRetry.factor | 2 |
backoffRetry.max | 5s |
backoffRetry.min | 100ms |
request.method | GET |
request.url | http://127.0.0.1:54321/{{.Payload.After.name}} |
response.body | .Payload.After.response |
Record difference
Before | After | ||||
1 | { | 1 | { | ||
2 | "position": "cG9zLTE=", | 2 | "position": "cG9zLTE=", | ||
3 | "operation": "create", | 3 | "operation": "create", | ||
4 | "metadata": null, | 4 | "metadata": null, | ||
5 | "key": null, | 5 | "key": null, | ||
6 | "payload": { | 6 | "payload": { | ||
7 | "before": null, | 7 | "before": null, | ||
8 | "after": { | 8 | "after": { | ||
9 | - | "name": "foo" | 9 | + | "name": "foo", |
10 | + | "response": "aGVsbG8sIGZvbyE=" | |||
10 | } | 11 | } | ||
11 | } | 12 | } | ||
12 | } | 13 | } |