https://linen.dev logo
Join Slack
Powered by
How would I go about logging during execution? I s...
# replication-troubleshooting
b
How would I go about logging during execution? I sometimes get stuck for a looong time on 
preparing tables at destination completed
and have no way of knowing wat happened
u
Hey Berzan, here's a doc on browsing output logs, let me know if this helps in your case: https://airbytehq.github.io/operator-guides/browsing-output-logs
b
I meant more as in where in the CDK would i insert a log.info to see for example when the connection is made, so I can check if it's just my http request taking long
The same connection worked before (no code changes). And the API works in postman, so I'm not sure what's happening in airbyte
Copy code
{"type": "LOG", "log": {"level": "INFO", "message": "SourceDataguardPicCrm runtimes:\nSyncing stream companies 0:05:33.914985"}}
{"type": "LOG", "log": {"level": "FATAL", "message": "1 validation error for AirbyteRecordMessage\ndata\n  value is not a valid dict (type=type_error.dict)\nTraceback (most recent call last):\n  File \"/Users/berzanmikailiyildiz/data-warehouse/airbyte/airbyte-integrations/connectors/source-dataguard-pic-crm/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 129, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 120, in run\n    for message in generator:\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 129, in read\n    raise e\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 115, in read\n    yield from self._read_stream(\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 165, in _read_stream\n    for record in record_iterator:\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 255, in _read_full_refresh\n    yield self._as_airbyte_record(configured_stream.stream.name, record)\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 288, in _as_airbyte_record\n    message = AirbyteRecordMessage(stream=stream_name, data=data, emitted_at=now_millis)\n  File \"pydantic/main.py\", line 342, in pydantic.main.BaseModel.__init__\npydantic.error_wrappers.ValidationError: 1 validation error for AirbyteRecordMessage\ndata\n  value is not a valid dict (type=type_error.dict)"}}
{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1668789087790.557, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "1 validation error for AirbyteRecordMessage\ndata\n  value is not a valid dict (type=type_error.dict)", "stack_trace": "Traceback (most recent call last):\n  File \"/Users/berzanmikailiyildiz/data-warehouse/airbyte/airbyte-integrations/connectors/source-dataguard-pic-crm/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 129, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 120, in run\n    for message in generator:\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 129, in read\n    raise e\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 115, in read\n    yield from self._read_stream(\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 165, in _read_stream\n    for record in record_iterator:\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 255, in _read_full_refresh\n    yield self._as_airbyte_record(configured_stream.stream.name, record)\n  File \"/opt/homebrew/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 288, in _as_airbyte_record\n    message = AirbyteRecordMessage(stream=stream_name, data=data, emitted_at=now_millis)\n  File \"pydantic/main.py\", line 342, in pydantic.main.BaseModel.__init__\npydantic.error_wrappers.ValidationError: 1 validation error for AirbyteRecordMessage\ndata\n  value is not a valid dict (type=type_error.dict)\n", "failure_type": "system_error"}}}
This is the error that I get
Ok so thanks for running locally I found the error. In my opinion this should be thrown in the logs as well but it is not
u
Glad you found it! Could you please open a GitHub feature request for that?
b
I found and fixed it, but it seems like for some reason when running in the workspace airbyte fails still without error
It works fine locally with my own catalog
I run this locally:
Copy code
docker run --rm --init -i -w /data/132/0 --name source-dataguard-pic-crm-read-132-0-ptvdk --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_CONNECTOR_IMAGE=airbyte/source-dataguard-pic-crm:0.0.16 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.18 -e WORKER_JOB_ID=132 airbyte/source-dataguard-pic-crm:0.0.16 read --config source_config.json --catalog source_catalog.json --state input_state.json
It works. In airbyte it doesn't. Not sure why? Is it the postgres destination?
n
Could you please clarify what you're trying to accomplish - 1. When you say 'in the workspace' - do you mean in the Airbyte web app? 2. Could you elaborate what you mean by 'running locally with your own catalog'? It looks like you are hosting Airbyte locally in Docker, is that correct?
8 Views
Open in Slack
PreviousNext
Powered by