This is not urgent, but something to consider for long-term: Do we have a way to reuse a chunk of spec.json across a bunch of connectors? My use case is ssh tunnel configuration, which is quite verbose and would be identical across many, many connectors. I’d rather DRY that section of json than have it in a ton of places. That would also facilitate internationalization/localization someday if necessary, rather than spreading explanatory text around in a ton of places. The connection configuration feels like the right place to expose it to user configuration, since it includes host and port information per database instance. But the json itself could use an import/template/abstraction instead of existing in tons of connectors independently.

TL;DR: create a common superclass and override the

spec

method The way we’ve done it in python connectors is to programmatically enrich the specific connector’s specification with the common specification. For example let’s say we have an abstract class representing a file-based connector. Given a stream of bytes, this file-based connector can read any file format: CSV, JSON, Parquet, Avro, etc.. compressed or otherwise. But the stream of bytes can come from anywhere: S3 connector, SFTP, GCS, etc.. the abstract class expects a configuration on how to interpret the stream of bytes e.g

{
  "format": "csv",
  "delimiter": ",",
  "encoding": "utf-8",
  "compression": "gzip"
}

and that would be the same no matter where the stream of bytes is coming from. And then depending on the destination we want a separate “chunk” of configuration that is storage-layer-dependent e.g: role ID & bucket name for S3 or Service account credentials for GCS etc.. then we could create a common superclass

FilebasedConnector

whose

spec

method returns the “abstract” specification, and override that in each of the child classes to enrich it with the storage-specific parameters

there might be is probably a better way to do this using JSON templating but the one above can probably take us a long distance