Connector Specification

A connector's specification is used to present information about the connector and validate connector parameters. The specification can be found in the connectors.yaml file, that's found in the root of a connector's repository. The following is an example of the file connector's connector.yaml:

version: "1.0"
specification:
  name: file
  summary: A file source and destination plugin for Conduit.
  description: |-
    The file source allows you to listen to a local file and
    detect any changes happening to it. Each change will create a new record. The
    destination allows you to write record payloads to a destination file, each new
    record payload is appended to the file in a new line.
  version: v0.10.0
  author: Meroxa, Inc.
  source:
    parameters:
      - name: path
        description: Path is the file path used by the connector to read/write records.
        type: string
        default: ""
        validations:
          - type: required
            value: ""
    # other parameters

The contents of this file are embedded into a connector's binary and read by the connector binary itself. connector.yaml is also being used by other tools that enrich Conduit's functionality, such as the CLI, the documentation website, etc.

Contents of connector.yaml

A part of connector.yaml is written by a developer (manually) and the other part is generated by a tool that Conduit provides, conn-sdk-cli.

A developer is writing the general information about the connector, which can be seen in this part from the example above:

specification:
  name: file
  summary: A file source and destination plugin for Conduit.
  description: |-
    The file source allows you to listen to a local file and
    detect any changes happening to it. Each change will create a new record. The
    destination allows you to write record payloads to a destination file, each new
    record payload is appended to the file in a new line.
  version: v0.10.0
  author: Meroxa, Inc.

Additional information may be added outside the specification key, for example:

version: "1.0"
specification:
  # specification section
# developer-info is just an example, any key can be used
developer-info:
  email: [email protected]
  website: https://example.com

The information about a connector's source and destination parameters is generated using a tool called [ conn-sdk-cli. conn-sdk-cli uses the configuration structs in a connector's code to generate the list of parameters, the validations for those, etc. conn-sdk-cli is described in more details in the following section.

conn-sdk-cli

conn-sdk-cli works by inspecting the configuration structs that the source and destination connectors return in the Config() method. Those configuration structs will be transformed into a map of parameter descriptions that's written into connector.yaml. More information about source and destination configuration can be found in Developing a Source Connector and Developing a Destination Connector.

If you're using the Conduit Connector Template, then the conn-sdk-cli tool is already listed in tools.go and can be installed with make install-tools.

conn-sdk-cli is run as part of go generate. It also needs access to the sdk.Connector variable that holds references to the constructor functions for the source and the destination, so it's best to place it in the connector.go file. The following is an example from the Kafka connector:

//go:generate conn-sdk-cli specgen

// Package kafka contains implementations for Kafka source and destination
// connectors for Conduit.
package kafka

import (
	_ "embed"

	sdk "github.com/conduitio/conduit-connector-sdk"
)

//go:embed connector.yaml
var specs string

var Connector = sdk.Connector{
	NewSpecification: sdk.YAMLSpecification(specs),
	NewSource:        NewSource,
	NewDestination:   NewDestination,
}

If you run go generate ./... or just use make generate (that's provided by the connector template), you'll see that the connector.yaml is updated with the source and/or destination parameters.

What you also see in the example above is how to embed the connector.yaml file (with //go:embed connector.yaml) and then extract the specification from it (NewSpecification: sdk.YAMLSpecification(specs)).

Default values

Default values are taken from the following sources (higher to lower precedence):

1. A source or destination connector's configuration (returned by a connector's Config() method).
2. The default tag on fields in a configuration struct.

For example:

type SourceConfig struct {
	URL string `default:"http://localhost:8080"`
}

func NewSource() sdk.Source {
    return sdk.SourceWithMiddleware(&Source{
        config: source.Config{
            URL: "https://example.com"
        },
    })
}

In this example, the default value in the generated specification will be "https://example.com", i.e. if a user doesn't specify a URL, its value will be "https://example.com".