https://linen.dev logo
Join Slack
Powered by
Hello, I am new to Airbyte and trying to build a c...
# ask-community-for-troubleshooting
k
Hello, I am new to Airbyte and trying to build a custom Source in Airbyte for an AWS service. I have followed the Python Source method to create it. While syncing it to a Local CSV or to any other destinations, only a CSV file with _airbyte___data, _airbyte_id, _airbyte_emitted_at columns are written but the data is not, at the end the sync is successful without any errors... The source is able to get the data from the API, but it is unable to read it to the destination it seems. Here is the log:
Copy code
2023-05-18 14:51:07 INFO i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/6/0/logs.log
2023-05-18 14:51:07 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.17
2023-05-18 14:51:08 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):50 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false'
2023-05-18 14:51:08 INFO i.a.c.EnvConfigs(getEnvOrDefault):1079 - Using default value for environment variable METRIC_CLIENT: ''
2023-05-18 14:51:08 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to
2023-05-18 14:51:08 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):50 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false'
2023-05-18 14:51:08 INFO i.a.w.g.DefaultReplicationWorker(run):125 - start sync worker. job id: 6 attempt id: 0
2023-05-18 14:51:08 INFO i.a.w.g.DefaultReplicationWorker(run):141 - configured sync modes: {null.EVA and Equity  EVA by Industry=full_refresh - overwrite}
2023-05-18 14:51:08 INFO i.a.w.i.DefaultAirbyteDestination(start):65 - Running destination...
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION -----
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-csv:0.2.10 exists...
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-csv:0.2.10 was found locally.
2023-05-18 14:51:08 INFO i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = destination-csv-write-6-0-cneip with resources io.airbyte.config.ResourceRequirements@6f08b143[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-05-18 14:51:08 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/6/0 --log-driver none --name destination-csv-write-6-0-cneip --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_CONNECTOR_IMAGE=airbyte/destination-csv:0.2.10 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.17 -e WORKER_JOB_ID=6 airbyte/destination-csv:0.2.10 write --config destination_config.json --catalog destination_catalog.json
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-aws-dataexchange-api:dev exists...
2023-05-18 14:51:08 INFO i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-aws-dataexchange-api-read-6-0-znwgl with resources io.airbyte.config.ResourceRequirements@7ee6c8e6[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-05-18 14:51:08 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-aws-dataexchange-api:dev was found locally.
2023-05-18 14:51:08 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/6/0 --log-driver none --name source-aws-dataexchange-api-read-6-0-znwgl --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_CONNECTOR_IMAGE=airbyte/source-aws-dataexchange-api:dev -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.17 -e WORKER_JOB_ID=6 airbyte/source-aws-dataexchange-api:dev read --config source_config.json --catalog source_catalog.json
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):372 - Source has no more messages, closing connection.
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):381 - Total records read: 0 (0 bytes)
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(run):190 - One of source or destination thread complete. Waiting on the other.
2023-05-18 14:51:11 destination > 2023-05-18 14:51:11 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):65 - Airbyte message consumer: succeeded.
2023-05-18 14:51:11 destination > 2023-05-18 14:51:11 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):179 - finalizing consumer.
2023-05-18 14:51:11 destination > 2023-05-18 14:51:11 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):195 - File output: /local/test/_airbyte_raw_EVA_and_Equity_EVA_by_Industry.csv
2023-05-18 14:51:11 destination > 2023-05-18 14:51:11 INFO i.a.i.b.IntegrationRunner(runInternal):171 - Completed integration: io.airbyte.integrations.destination.csv.CsvDestination
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(run):192 - Source and destination threads complete.
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(run):297 - Source did not output any state messages
2023-05-18 14:51:11 WARN i.a.w.g.DefaultReplicationWorker(run):308 - State capture: No state retained.
2023-05-18 14:51:08 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):454 - Destination output thread started.
"maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "replicationStartTime" : 1684421468080,
    "replicationEndTime" : 1684421471222,
    "sourceReadStartTime" : 1684421468130,
    "sourceReadEndTime" : 1684421471088,
    "destinationWriteStartTime" : 1684421468224,
    "destinationWriteEndTime" : 1684421471222
  },
  "streamStats" : [ ]
}
2023-05-18 14:51:11 INFO i.a.w.g.DefaultReplicationWorker(run):317 - failures: [ ]
2023-05-18 14:51:11 INFO i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...
2023-05-18 14:51:11 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):176 - sync summary:
Any help would be appreciated
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, I recommend reviewing the [Airbyte documentation](https://docs.airbyte.com) for more information on building custom connectors and troubleshooting issues.
k
Mainly the error is, I am getting these outputs: • Source has no more messages, closing connection. • Total records read: 0 (0 bytes) But the Sync is successful
Open in Slack
PreviousNext
Powered by