While trying to capture collections metadata from Mongodb, I faced this error:

OperationFailure: BSONObj size: 17365481 (0x108F9E9) is invalid. Size must be between 0 and 16793600(16MB)

Is there any configuration which let's the Datahub skip the document with larger field values ?

@witty-keyboard-20400, I will look at this and reply back.

@miniature-tiger-96062 Could you please let me know if there is any flag/config based on which I could get unblocked and ingest metadata from MongoDB? If you could point me to the place in Python/Java code where this decision of size check is done, that would also help.

Hi @witty-keyboard-20400, currently we do not have a flag for skipping documents over a certain size but that is what I am planning to introduce. But for your reference here is where we get the samples - https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/mongodb.py#L307