Hi guys, My MongoDB’s ingestion job keeps failing with the error below. Is there anyway to deal with this case? Any help is appreciated 🤗

'File "/tmp/datahub/ingest/venv-d4f145d7-4f51-4053-990f-4c541a5eaa57/lib/python3.9/site-packages/bson/__init__.py", line 1164, in '
'_decode_all_selective\n'
'    return decode_all(data, codec_options)\n'
'File "/tmp/datahub/ingest/venv-d4f145d7-4f51-4053-990f-4c541a5eaa57/lib/python3.9/site-packages/bson/__init__.py", line 1108, in '
'decode_all\n'
'    return _decode_all(data, opts)  # type: ignore[arg-type]\n'
'\n'
'InvalidBSON: year 50577 is out of range\n'

This appears to be an issue in the BSON library we are using - unclear why a year would be considered out of range..

@big-carpet-38439 Thanks for your reply. Excluding these datasets is the only way we can do until now. 😕

Is there any chance this is bad data?

Yeah, this is the bad data. Our backend team should be clean it up before dump to mongodb. In the meantime, I’ve excluded these tables in ingestion config. Or, we need to make something with pymongo to handle these exceptions.