To test if the issue was related to drastic changes in the datasets I did the following test: 1. Disabled stateful ingestion and run the ingestion 2. Turn the stateful ingestion on after a successful full ingestion and run the ingestion again After disabling the stateful ingestion no errors occur, as expected. Afterwards enabling the stateful ingestion again make the ingestion job fail again. 😞 I would expect the next run after a full ingestion to succeed. Since the previous ingestion was a full one, doesn’t it mean no drastic changes occurred in between? Or is my assumption wrong?

Indeed we are using an older version. 😮

Cli report:
{'cli_version': '0.9.0',
 'cli_entry_location': '/tmp/datahub/ingest/venv-glue-0.9.0/lib/python3.10/site-packages/datahub/__init__.py',
 'py_version': '3.10.9 (main, Jan 23 2023, 22:32:48) [GCC 10.2.1 20210110]',
 'py_exec_path': '/tmp/datahub/ingest/venv-glue-0.9.0/bin/python3',
 'os_details': 'Linux-5.10.167-147.601.amzn2.x86_64-x86_64-with-glibc2.31',
 'mem_info': '105.59 MB'}

We recently upgraded DataHub to 0.10.0. It seems that in the ingestion pipeline setup, in the advanced tab, the version of the CLI still 0.9.0 😮 Looks like we need to change it and enforce the correct version we want in the CLI after upgrading DataHub