the schema is different across the documents over time (dont even get me started) , and scanning 10k isnt returning the "full" schema

Hi @Jeff Crooks this is an "optimization" we made to make schema discovery faster, and this limit is hardcoded. The change was made in version

0.1.9

(PR). The ideal solution would be to allow this limit to be customized. I opened an issue for this. Feel free to upvote it. A workaround could be to downgrade your Mongo connector to

0.1.8

thanks - root cause is the insanity of the people who are writing to mongo, but upvoted!

Changing your source database system could be a workaround indeed 😛 😄

been in progress for multiple years 🙂

so i have high hopes....

Good luck then!

the "actual" schema is in a separate table

would it be useful to have an “infer schema” flag which when turned off just outputs untyped JSONs?

instead of doing the whole inference thing