Hi team, I’m trying to debug connectors. I’m following documentation and trying to stream data from a source to destination using docker. I’m stuck and can’t understand why the destination can’t consume messages from pipe. https://docs.airbyte.io/connector-development/tutorials/building-a-python-destination https://docs.airbyte.io/connector-development/tutorials/building-a-java-destination I run:

docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json > messages.json

cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json

or just

docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json

But I don’t see that

destination-csv

is consuming AirbyteMessages

cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:dev write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.csv.CsvDestination
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=/sample_files/configured_catalog.json, write=null, config=/secrets/config_d_csv.json}
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='/secrets/config_d_csv.json', catalogPath='/sample_files/configured_catalog.json', statePath='null'}
2021-09-21 13:29:01 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(<init>):157 - {} - initializing consumer.
2021-09-21 13:29:02 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):198 - {} - finalizing consumer.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):214 - {} - File output: /local/tmp/out/_airbyte_raw_dmesh.csv
2021-09-21 13:29:02 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.csv.CsvDestination

Could you please give me a hint of what am I doing wrong? Is there some problem with config.json?