Skip to main content

Using a Custom Connector in Conduit

Adding the Connector to Conduit​

The Conduit Connector template includes a Makefile that can be used to build a binary version of your connector.

make build

After the command runs you should see a binary file with the name of the connector in the local directory.

To run the Connector, you will need to move it to a directory that Conduit can access. By default, this directory is the connectors directory located alongside the Conduit binary. Move your newly created Connector binary into the connectors directory and run Conduit using the following command:

./conduit run

You can use Conduit's HTTP API to list the available connector plugins:

curl localhost:8080/v1/connectors/plugins

The response will list the found connector plugins and will display their specifications. If you're looking just for the names, you can execute this command:

curl localhost:8080/v1/connectors/plugins | jq '.[].name'

Alternatively, navigate to the Conduit's Swagger UI page ( while Conduit is running) to see your Connector in the list of available Connectors. You can filter plugins by regex to find your connector easier (e.g. .*standalone.*, will give you all standalone connectors, because the standalone prefix is added to all custom connectors).

Custom Connector in Conduit Swagger UI

Using the Connector in a Conduit Pipeline​

The recommended way to use the Connector is to define a pipeline configuration file. Alternatively, the HTTP API can be used.

When using custom Connectors in a pipeline configuration, they are prefixed with phrase standalone unlike built in Connectors which are prefixed with builtin. For example, to use a custom Connector named file-sync it would look as follows: plugin: standalone:file-sync@[replace-with-connector-version].

More information about referencing connectors can be found here.

Below is an example of using a custom Connector in a pipeline configuration file:

pipelines/use-custom-connector.yml
version: 2.2
pipelines:
  - id: use-custom-connector
    # run pipeline on startup
    status: running
    description: >
      Example pipeline reading to use a custom file sync connector.

    connectors:
      - id: source-sync
        type: source
        # use the custom file-sync plugin as the source
        plugin: standalone:file-[email protected]
        settings:
          # use ./data as the directory input to the file-sync connector; the connnector will watch for new files in this directory
          directory: ./data

      - id: destination-sync
        type: destination
        # use the custom file-sync plugin as the destination
        plugin: standalone:file-[email protected]
        settings:
        # use ./new_destination as the directory input to the file-sync connector; the connnector will write files from the pipeline to this directory
          directory: ./new_destination

scarf pixel conduit-site-docs-developing-connectors