Getting Started

Our goal

In this guide, our goal will be to stream information about flights from an imaginary airport to a file. The flight records contain just the airline name and the scheduled departure time.

Install Conduit

If you're using a macOS or Linux system, you can install Conduit with the following command:

$ curl https://conduit.io/install.sh | bash

If you're not using macOS or Linux system, you can still install Conduit following one of the different options provided in our installation page.

note

The Conduit binary contains both, the Conduit service and the Conduit CLI, with which you can interact with Conduit.

Initialize Conduit

Firs, let's initialize the working environment:

$ conduit init

Created directory: processors
Created directory: connectors
Created directory: pipelines
Configuration file written to conduit.yaml

Conduit has been initialized!

To quickly create an example pipeline, run 'conduit pipelines init'.
To see how you can customize your first pipeline, run 'conduit pipelines init --help'.

conduit init creates the directories where you can put your pipeline configuration files, connector binaries, and processor binaries. There's also a conduit.yaml that contains all the configuration parameters that Conduit supports.

In this guide, we'll only use the pipelines directory, since we won't need to install any additional connector nor to change Conduit's default configuration.

Build a pipeline

Next, we can use the Conduit CLI to build a demo pipeline:

$ conduit pipelines init

conduit pipelines init builds a demo pipeline that generates flight information from an imaginary airport every second.

If the pipelines directory, you'll notice a new file, demo-pipeline.yaml that contains our pipeline's configuration:

version: "2.2"
pipelines:
  - id: "demo-pipeline"
    description: "This pipeline was initialized using the `conduit pipelines init` command.
It is a demo pipeline that connects a source connector (generator) to a destination connector (log).
The next step is to simply run `conduit run` in your terminal and you should see a new record being logged every second.
Check out https://conduit.io/docs/using/pipelines/configuration-file to learn about how this file is structured."
    status: running
    name: "demo-pipeline"
    connectors:
      - id: "generator-source"
        type: source
        plugin: "generator"
        settings:
          # Generate field 'airline' of type string
          # Type: string
          # Optional
          format.options.airline: 'string'
          # Generate field 'scheduledDeparture' of type 'time'
          # Type: string
          # Optional
          format.options.scheduledDeparture: 'time'
          # The format of the generated payload data (raw, structured, file).
          # Type: string
          # Optional
          format.type: 'structured'
          # The maximum rate in records per second, at which records are
          # generated (0 means no rate limit).
          # Type: float
          # Optional
          rate: '1'
      - id: "log-destination"
        type: destination
        plugin: "log"

The configuration above tells us some basic information about the pipeline (ID and name) and that we want Conduit to start the pipeline automatically ( status: running).

Then we see a source connector, that uses the generator plugin, which is a built-in plugin that can generate random data. The source connector's settings translate into: generate structured data, 1 record per second. Each generated record should contain an airline field (type: string) and a scheduledDeparture field (type: duration).

What follows is a destination connector where the data will be written to. It uses the log plugin, which is a built-in plugin that writes all the incoming data to a log you can see when you run Conduit.

Run Conduit

With the pipeline configuration being ready, we can run Conduit:

$ conduit run

Once Conduit starts running the pipeline, every second, you should see a log entry like this:

2025-06-11T12:37:28+00:00 INF component=plugin.standalone connector_id=example-pipeline:example-destination plugin_name=conduit-connector-log record={"key":"aHlkYXRvcG5ldW1hdG9seXRpYw==","metadata":{"conduit.source.connector.id":"example-pipeline:example-source","opencdc.createdAt":"1749638247540840000","opencdc.payload.schema.subject":"example-pipeline:example-source:payload","opencdc.payload.schema.version":"1","opencdc.readAt":"1749638248541754000"},"operation":"create","payload":{"after":{"airline":"elytrorrhagia","scheduledDeparture":"2025-06-11T10:37:27.540841Z"},"before":null},"position":"Mg=="}

This log entry is Conduit's way of showing you the data that was generated by the generator connector. If we look into the record field, we can see the following JSON object:

{
  "key": "aHlkYXRvcG5ldW1hdG9seXRpYw==",
  "metadata": {
    "conduit.source.connector.id": "example-pipeline:example-source",
    "opencdc.createdAt": "1749638247540840000",
    "opencdc.payload.schema.subject": "example-pipeline:example-source:payload",
    "opencdc.payload.schema.version": "1",
    "opencdc.readAt": "1749638248541754000"
  },
  "operation": "create",
  "payload": {
    "after": {
      "airline": "elytrorrhagia",
      "scheduledDeparture": "2025-06-11T10:37:27.540841Z"
    },
    "before": null
  },
  "position": "Mg=="
}

The JSON object you see is the OpenCDC record that holds the data being streamed as well as other data and metadata. In the .payload.after field you will see the user data that was generated by the generator connector:

{
  "airline": "elytrorrhagia",
  "scheduledDeparture": "2025-06-11T10:37:27.540841Z"
}

The pipeline will keep streaming the data from the generator source connector to the log destination connector as long as Conduit is running. To stop Conduit, press Ctrl + C (on a Linux OS, or the equivalent on other operating systems). This will trigger a graceful shutdown that stops reads from source connectors and waits for records that are still in the pipeline to be acknowledged. The next time Conduit starts, it will start reading data from where it stopped.

tip

Run conduit pipelines init --help to see how you can customize the pipeline configuration file that's being generated.

What's next?

Now that you've got the basics of running Conduit and creating a pipeline covered, here are a few places to dive in deeper:

• Connectors
• Pipelines
• Processors

Or, if you want to experiment a bit more, check out the examples in our GitHub repository.