Hey, I am creating a custom connector and there is an issue I am hitting. This specific source has very large number of source objects (100k), so AirbyteCatalog gets quite large. And please mind that these are standard tables, not something like various API stock tickers. The Catalog JSON is ~28MB (and this is actually smaller than final size will be, since I just added mock schemas for now, not the actual ones) If I send the complete Catalog via discover it seems to wreck the frontend. First the browser fires DiscoverSchema twice (I suppose this is because of React StrictMode? But I am not running airbyte in dev mode...). It takes few minutes for the worker to get the result, but it receives it and browser seems to get result for both requests. Finaly browser sends destination_definition_specification, and receives a response, but nothing happens in the frontend. It is stuck on "we are fetching the schema of your data source." The memory footprint of the firefox tab keeps rising (6GB), until I suppose OS kills docker to get more mem. When tried on chrome the screen just gets blank. No errors in the browser console. Running latest version of airbyte in docker on Ubuntu WSL. Do you have any advice for me what to try? Thanks

I would try running via Octavia. Had a problem with that on a large MSSQL database with more than 1000 tables, and the only way I was able to make that work was with octavia-cli

Yeah, was considering that, but doubt users will appreciate that. Another thing I am considering is building a simplified FE, because in this scenario users usually know specifically what objects they want from the source. So it would be viable for them to get simplified list, or some search help via typesense, or only a prompt to insert names of objects, and then handle the registration on the backend or something.

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2685 to ensure timely resolution!

Airbyte team is working to deal with large catalogs, but today is a limitation 😞