Here wanted to get some understanding on these points - related to k8s deployment 1. Is there any restriction on max syncs that can happen at any point of time. a. We were facing issues like

address is already in use

b. The sync becomes stale and no movement in logs after waiting for pod with ip 2. I understand that we have some ports defined which the jobs can use is there a way increasing the ports will help in increasing total number of syncs. 3. When pod moves to completed state can we release the ports? 4. How many ports does 1 sync job takes ?

1. With the default port setup ~10 simultaneous. There’s a transient bug resulting in “stuck” syncs that we should have a fix for on Monday.

2. Right now increasing the ports will increase the number of parallel syncs. There’s an open PR that will also allow you to set a maximum number of simultaneous syncs as long as it’s less than half of the port count.

3. This should happen automatically (but which is failing a few % of the time right now). This is the same bug mentioned above.

4. Two

Keep in mind that the load on the scheduler will increase if you increase the ports / parallelism, so if you’re using resource limits you may need to bump those up.

In a week or two that piece will be horizontally scalable just by setting the number of replicas for a certain deployment.

We also are planning on getting rid of this multiple port thing (maybe in 1-2 mo?) and using a different interface for passing records that allows a shared port. Think GCP bidirectional streaming or similar queueing systems.

@Jared Rhizor (Airbyte) thanks a lot for sharing these. 🙏

@Jared Rhizor (Airbyte) we are facing a lot of issues not sure what is reason. Adding logs here @Davin Chia (Airbyte) if you can help here

This looks like logs from temporal and jobs. Is that right?

I’m seeing multiple errors. Is this one job?

What’s the scale of jobs you are running now?

Scale as in ?

Those are logs from scheduler and most of the errors we see is that the sync stops all of a sudden or either the syncs don’t start

Scale - how many concurrent jobs running?

Syncs not starting is something I’m working on. Should be out in the next few days

I’m not sure about them stopping all of a sudden. Does it stop mid-sync?

yeah we are moving back to ec2 for now. There are about 15-16 jobs running parallely

yea sync stops in mid

Like this it stopped and we had to cancel it manually