Skip to main content

Conduit Operator

The Conduit Operator is a Kubernetes operator designed to simplify the management and orchestration of Conduit instances. The operator extends the Kubernetes API with a custom resource for Conduit.

This custom resource allows users to define and manage Conduit instances declaratively, enabling seamless integration with Kubernetes' native features such as scaling, monitoring, and logging.

Overview​

Conduit pipelines are represented as Conduit custom resources, where each pipeline will be provisioned as a distinct Conduit instance with its own lifecycle.

The Conduit custom resource definition format is very similar to that of pipeline configurations:

apiVersion: operator.conduit.io/v1alpha
kind: Conduit
metadata:
  name: conduit-generator
spec:
  running: true
  name: generator.log
  description: generator pipeline
  connectors:
    - name: source-connector
      type: source
      plugin: builtin:generator
      settings:
        - name: format.type
          value: structured
        - name: format.options.id
          value: "int"
        - name: format.options.name
          value: "string"
        - name: format.options.company
          value: "string"
        - name: format.options.trial
          value: "bool"
        - name: recordCount
          value: "3"
    - name: destination-connector
      type: destination
      plugin: builtin:log

Quickstart​

info

To get started with the Conduit Operator, follow the steps described on the Conduit Operator GitHub repository.

Installing standalone connectors​

plugin and pluginVersion​

A standalone connector can be installed by providing the connector name via plugin, and version via pluginVersion.

The operator will then install the connector and create the necessary resources to run it. These connectors need to be referred to by their GitHub organization/repository names and are currently limited to the following organizations:

tip

Plugin version can optionally be specified or the latest will be used.

Example​

Here's a full example using conduitio/conduit-connector-generator as plugin and v0.8.0 as pluginVersion:

apiVersion: operator.conduit.io/v1alpha
kind: Conduit
metadata:
  name: conduit-generator
spec:
  running: true
  name: generator.log
  description: generator pipeline
  connectors:
    - name: source-connector
      type: source
      plugin: conduitio/conduit-connector-generator
      pluginVersion: v0.8.0
      settings:
        - name: format.type
          value: structured
        - name: format.options.id
          value: "int"
        - name: format.options.name
          value: "string"
        - name: format.options.company
          value: "string"
        - name: format.options.trial
          value: "bool"
        - name: recordCount
          value: "3"
    - name: destination-connector
      type: destination
      plugin: conduitio/conduit-connector-log

Schema Support​

As of v0.11.0 Conduit supports the use of a schema registry.

This allows connectors to automatically extract or use the schema referred to by the OpenCDC record to encode/decode data.

By default Conduit uses a builtin schema registry, however in certain use cases a schema registry needs to be shared between multiple instances. The Conduit resource allows for schema registry to be defined as of v0.2.0.

Details are provided directly via the Conduit resource. basicAuthUser and basicAuthPassword can be provided as secrets but will be copied over into controller-owned secrets for management and ease of use.

Example​

apiVersion: operator.conduit.io/v1alpha
kind: Conduit
metadata:
  name: conduit-generator-schema-registry
spec:
  running: true
  name: generator.standalone.log
  description: generator pipeline
  schemaRegistry:
    url: http://apicurio:8080/apis/ccompat/v7
    # basicAuthUser:
    #   - value: <schemaUser>
    # basicAuthPassword:
    #   - secretRef:
    #     key: schema-registry-password
    #     name: schema-registry-secret
  connectors:
    - name: source-connector
      type: source
      plugin: conduitio/conduit-connector-generator
      settings:
        - name: format.type
          value: structured
        - name: format.options.id
          value: "int"
        - name: format.options.name
          value: "string"
        - name: format.options.company
          value: "string"
        - name: format.options.trial
          value: "bool"
        - name: recordCount
          value: "3"
    - name: destination-connector
      type: destination
      plugin: conduitio/conduit-connector-log
      pluginVersion: v0.4.0

Deploying​

The operator provides a helm chart that can be used for deployment on any cluster. Additional metadata can be injected into each provisioned conduit instance via the controller.conduitMetadata configuration.

For example, to instruct a prometheus instance to scrape each Conduit instance metrics:

controller:
  conduitMetadata:
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/path: /metrics
      prometheus.io/port: 8080

For more configuration options see charts/conduit-operator/values.yaml.

Additional metadata can be injected in each provisioned conduit instance via controller.conduitMetadata configuration.

For example, to instruct a prometheus instance to scrape each Conduit instance metrics:

controller:
  conduitMetadata:
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/path: /metrics
      prometheus.io/port: 8080

For more configuration options see charts/conduit-operator/values.yaml.

Using the helm chart repository​

Alternatively the operator can be deployed via the Helm repository. To add the repository to your helm repos:

helm repo add conduit https://conduitio.github.io/conduit-operator

Install the operator in the conduit-operator namespace of your cluster:

helm install conduit-operator \
    conduit/conduit-operator --create-namespace -n conduit-operator
tip

For a low-code experience, and many other Enterprise features, we recommend you use the Conduit Platform.