Core concepts

Pipeline

A pipeline receives records from one or multiple source connectors, pushes them through zero or multiple processors until they reach one or multiple destination connectors.

Connector

A connector is the internal entity that communicates with a connector plugin and either pushes records from the plugin into the pipeline (source connector) or the other way around (destination connector).

Conduit ships with connectors that are directly built into the service (built-in connectors), but it can also be expanded with additional standalone connectors.

One of the main differences between Conduit connectors and those that you might find from other services is that all Conduit connectors are Change Data Capture-first (CDC).

CDC allows your pipeline to only get the changes that have happened over time instead of pulling down an entire upstream data store and then tracking diffs between some period of time. This is critical for building real-time, event-driven pipelines and applications. But, we'll note where connectors do or do not have CDC capabilities.

info

Here's a full list of our Conduit Connectors.

Built-in Connector

To help developers bootstrap pipelines much more quickly, Conduit ships with several built-in connectors by default.

This includes the Postgres, File, Random Data Generator, Kafka and Amazon S3 connectors.

If you are creating and distributing customized, pre-built Conduit binaries yourself, you may want to modify the list of built-in connectors, too.

By modifying the list of built-in connectors, your compiled binary will include a set of pre-installed connectors specific to your end users' needs, and it can be run by others, as is, without them having to follow any additional installation instructions.

Here's how to do it.

Standalone Connector

In addition to built-in connectors, Conduit can be used together with standalone connectors to enable even more data streaming use cases. The Conduit team and other community developers create and maintain standalone connectors.

Learn more about how you can install standalone connectors to Conduit here.

Source and Destination

• Source means the connector has the ability to get data from an upstream data store.
• Destination means the connector can to write to a downstream data store.

Connector plugin

Sometimes also referred to as "plugin", is an external process which communicates with Conduit and knows how to read/write records from/to a data source/destination (e.g. a database).

Processor

A component that executes an operation on a single record that flows through the pipeline. It can either change the record or filter it out based on some criteria.

OpenCDC Record

A record represents a single piece of data that flows through a pipeline (e.g. one database row). More info here.

Collection

A generic term used in Conduit to describe an entity in a 3rd party system from which records are read from or to which records they are written to. Examples are: topics (in Kafka), tables (in a database), indexes (in a search engine), collections (in NoSQL databases), etc.