https://linen.dev logo
h

Harshith (Airbyte)

07/17/2021, 6:17 AM
Here wanted to get some understanding on these points - related to k8s deployment 1. Is there any restriction on max syncs that can happen at any point of time. a. We were facing issues like
address is already in use
b. The sync becomes stale and no movement in logs after waiting for pod with ip 2. I understand that we have some ports defined which the jobs can use is there a way increasing the ports will help in increasing total number of syncs. 3. When pod moves to completed state can we release the ports? 4. How many ports does 1 sync job takes ?
u

user

07/17/2021, 6:34 AM
1. With the default port setup ~10 simultaneous. There’s a transient bug resulting in “stuck” syncs that we should have a fix for on Monday.
u

user

07/17/2021, 6:35 AM
2. Right now increasing the ports will increase the number of parallel syncs. There’s an open PR that will also allow you to set a maximum number of simultaneous syncs as long as it’s less than half of the port count.
h

Harshith (Airbyte)

07/17/2021, 6:36 AM
3. This should happen automatically (but which is failing a few % of the time right now). This is the same bug mentioned above.
u

user

07/17/2021, 6:36 AM
4. Two
u

user

07/17/2021, 6:38 AM
Keep in mind that the load on the scheduler will increase if you increase the ports / parallelism, so if you’re using resource limits you may need to bump those up.
u

user

07/17/2021, 6:38 AM
In a week or two that piece will be horizontally scalable just by setting the number of replicas for a certain deployment.
u

user

07/17/2021, 6:40 AM
We also are planning on getting rid of this multiple port thing (maybe in 1-2 mo?) and using a different interface for passing records that allows a shared port. Think GCP bidirectional streaming or similar queueing systems.
u

user

07/17/2021, 6:41 AM
@Jared Rhizor (Airbyte) thanks a lot for sharing these. 🙏
u

user

07/17/2021, 9:33 AM
@Jared Rhizor (Airbyte) we are facing a lot of issues not sure what is reason. Adding logs here @Davin Chia (Airbyte) if you can help here
u

user

07/17/2021, 3:38 PM
This looks like logs from temporal and jobs. Is that right?
u

user

07/17/2021, 3:39 PM
I’m seeing multiple errors. Is this one job?
u

user

07/17/2021, 3:39 PM
What’s the scale of jobs you are running now?
u

user

07/17/2021, 5:53 PM
Scale as in ?
u

user

07/17/2021, 5:55 PM
Those are logs from scheduler and most of the errors we see is that the sync stops all of a sudden or either the syncs don’t start
u

user

07/18/2021, 4:52 AM
Scale - how many concurrent jobs running?
u

user

07/18/2021, 4:52 AM
Syncs not starting is something I’m working on. Should be out in the next few days
u

user

07/18/2021, 4:53 AM
I’m not sure about them stopping all of a sudden. Does it stop mid-sync?
h

Harshith (Airbyte)

07/18/2021, 4:58 AM
yeah we are moving back to ec2 for now. There are about 15-16 jobs running parallely
u

user

07/18/2021, 5:02 AM
yea sync stops in mid
u

user

07/18/2021, 5:03 AM
Like this it stopped and we had to cancel it manually
3 Views