https://linen.dev logo
j

Jared Rhizor (Airbyte)

08/17/2021, 6:40 PM
@[DEPRECATED] Marcos Marx re: temporal issues. Do you have any examples of what you’ve been debugging or what you need to look at? From what I’ve seen, 99% of temporal errors we’ve seen have involved the underlying instance running out of space or other VM-level issues. Are you seeing something different where temporal itself is running into issues?
u

user

08/17/2021, 9:03 PM
https://airbytehq.slack.com/archives/C01MFR03D5W/p1626892883126300?thread_ts=1626892009.125000&cid=C01MFR03D5W I think @gunu is having some issues with temporal and having the instance unavailable
j

Jared Rhizor (Airbyte)

08/17/2021, 9:38 PM
I don’t know of a single “deadline exceeded” problem where the machine wasn’t out of disk space/memory/or the postgres instance was under a lot of load.
u

user

08/17/2021, 10:30 PM
the postgres instance of temporal or airbyte db?
u

user

08/17/2021, 10:33 PM
the postgres db that temporal is using (which is typically the airbyte db)
u

user

08/17/2021, 10:34 PM
if that’s overloaded we’ve seen problems with temporal, but usually it’s on the same machine so we’re still just running into resource constraints
u

user

08/17/2021, 10:34 PM
yeah I imagine (and more often than not) it’s a resourcing issue. but isn’t the point of temporal/scheduling to schedule and avoid that?
u

user

08/17/2021, 10:36 PM
keeping everything the same, would the k8s deployment not run into this specific issue?
m

Manav

08/17/2021, 10:44 PM
this is on a single node though
u

user

08/17/2021, 10:44 PM
(in most cases at least)
u

user

08/17/2021, 10:45 PM
I think if disk space / memory are underallocated for the db pod or temporal pods on kubernetes we could run into a similar issue
u

user

08/17/2021, 10:46 PM
in the near future temporal will help schedule across pods / nodes more easily, but this class of problems is more about the minimum resource requirements for temporal and its persistence (which is the airbyte postgres db)
u

user

08/18/2021, 1:27 AM
Jared, maybe the takeaway here is we can do better with minimum resource specification for temporal on our Docker deploys
u

user

08/18/2021, 2:56 AM
@Davin Chia (Airbyte) agreed both for Docker and Kube
l

Liren Tu

08/18/2021, 2:57 AM
Mostly in this thread I was curious if there were temporal-specific problems or debugging that we need to do outside of this class of issues.
u

user

08/18/2021, 2:57 AM
Temporal is pretty robust so my sense is same as yours in that it's probably going to be our fault ha ha
5 Views