Hello there ! was wondering about submitting prs for some connectors I made, but struggling with the integration tests timing out because I have a lot of data in these sources, even if it works fine on airbyte platform. Was wonderiong how they were made when submitting connectors, since i won’t publish my own creds ? Do you create accounts on each source to run integration tests ?

Hi. you can limit the data if the source accepts this option or in the code it self and then just remove it. I dont see a reason to test full data ever but rather a very small portion of it. If you want to test timings - this is actually better to test with a bit more data though still not in the TBs 100GB -200GB is more than enough to test the speed and memory

I’m talking about an API that has < 1GB data but it takes something like 6h to fully ingest it because of restrictions on calls 🙂 But the main topic here is : if I create a PR with which creds will acceptance tests be ran in order for the connector to be accepted ?

not yours 🙂 as you can understand the privacy here (unless I am mistaken here then lets wait for some one to correct me) 1GB of data in 6h - what is that ? I am interested to know 😄 sounds like GEO data or something like that?

It is a Saas called aircall we use to make the calls in my company, and each call is limited to 50 max with time windows that must contain less than 10000 records so... Very slow to ingest since lots of HTTP calls that each lasts for 2-3 sec 🙂 But anyways, can I still submit the pr and see if when not using our filled account it still works ?

wow sounds very slow yea i think you can try why not

Thanks @Eli Sigal, I maybe will post 2-3 Prs for new sources then, and eventually fix these issues if they arise when the checks will be ran. (Indeed it’s slow, but some APIs just have weird requirements...)