How to configure Conduit

Conduit accepts CLI flags, environment variables and a configuration file to configure its behavior. Each CLI flag has a corresponding environment variable and a corresponding field in the configuration file. Conduit uses the value for each configuration option based on the following priorities:

CLI flags

CLI flags (highest priority) - if a CLI flag is provided it will always be respected, regardless of the environment variable or configuration file. To see a full list of available flags run conduit --help:

$ conduit --help
Usage of conduit:
  -api.enabled
    	enable HTTP and gRPC API (default true)
  -config string
    	global config file (default "conduit.yaml")
  -connectors.path string
    	path to standalone connectors' directory (default "./connectors")
  -db.badger.path string
    	path to badger DB (default "conduit.db")
  -db.postgres.connection-string string
    	postgres connection string, may be a database URL or in PostgreSQL keyword/value format
  -db.postgres.table string
    	postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
  -db.sqlite.path string
    	path to sqlite3 DB (default "conduit.db")
  -db.sqlite.table string
    	sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
  -db.type string
    	database type; accepts badger,postgres,inmemory,sqlite (default "badger")
  -grpc.address string
    	address for serving the gRPC API (default ":8084")
  -http.address string
    	address for serving the HTTP API (default ":8080")
  -log.format string
    	sets the format of the logging; accepts json, cli (default "cli")
  -log.level string
    	sets logging level; accepts debug, info, warn, error, trace (default "info")
  -pipelines.error-recovery.backoff-factor int
    	backoff factor applied to the last delay (default 2)
  -pipelines.error-recovery.max-delay duration
    	maximum delay before restart (default 10m0s)
  -pipelines.error-recovery.max-retries int
    	maximum number of retries (default -1)
  -pipelines.error-recovery.max-retries-window duration
    	amount of time running without any errors after which a pipeline is considered healthy (default 5m0s)
  -pipelines.error-recovery.min-delay duration
    	minimum delay before restart (default 1s)
  -pipelines.exit-on-degraded
    	exit Conduit if a pipeline enters a degraded state
  -pipelines.path string
    	path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines")
  -processors.path string
    	path to standalone processors' directory (default "./processors")
  -schema-registry.confluent.connection-string string
    	confluent schema registry connection string
  -schema-registry.type string
    	schema registry type; accepts builtin,confluent (default "builtin")
  -version
    	prints current Conduit version

Environment variables

Environment variables (lower priority) - an environment variable is only used if no CLI flag is provided for the same option. Environment variables have the prefix CONDUIT and contain underscores instead of dots and hyphens (e.g. the flag -db.postgres.connection-string corresponds to CONDUIT_DB_POSTGRES_CONNECTION_STRING).

Configuration file

Configuration file (lowest priority) - Conduit by default loads the file conduit.yaml placed in the same folder as Conduit. The path to the file can be customized using the CLI flag -config. It is not required to provide a configuration file and any value in the configuration file can be overridden by an environment variable or a flag. The file content should be a YAML document where keys can be hierarchically split on .. For example:

db:
  type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
  postgres:
    connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING