https://linen.dev logo
Join Slack
Powered by
Hey, <@U04S8PLJWMT>, which of services are good to...
# ask-ai
d
Hey, @kapa.ai, which of services are good to be executed on spot instnaces, my replication jobs running on spot instances received 143 SIGTERM and died.
k
For high availability and to avoid unexpected SIGTERM (143) from spot node pre‑emptions, run the core platform services on stable/on‑demand nodes, and isolate pre‑emptible capacity to the ephemeral job pods: 1. Keep on on‑demand / non‑spot nodes: - 
server
- 
worker
(the long‑running worker deployment, not the per‑job replication pods) - Control‑plane components (scheduler, Temporal, database, etc.) These are explicitly called out as important to keep redundant and highly available (multiple replicas) rather than being frequently killed. [[Scaling HA](https://docs.airbyte.com/platform/1.6/enterprise-setup/scaling-airbyte#high-availability)] 2. Can run on spot / pre‑emptible nodes (with caveats): - Replication job pods (source/destination/orchestrator per sync), as long as you accept: - Jobs may be killed by the cloud provider (SIGTERM → exit code 143). [[Sync cancelled 143](https://discuss.airbyte.io/t/3171); [Sync cancelled 143-2](https://discuss.airbyte.io/t/2921); [Non-zero 143](https://discuss.airbyte.io/t/589)] - On current versions there are still open issues where a killed pod can cause the sync attempt to “hang” instead of cleanly failing/retrying. [[Killed pod hangs](https://github.com/airbytehq/airbyte/issues/48879)] Because of that last point, if you see many 143 exits from spot interruptions and they’re not being retried cleanly, you may want to: - Prefer on‑demand nodes for critical / long‑running syncs. - Or at least pin the control plane + workers to on‑demand, and only send non‑critical / retry‑tolerant job pods to spot nodes. The knowledge sources do not provide a formal, official “supported” spot‑vs‑on‑demand matrix, so the above is inferred from HA guidance and the observed behavior of jobs when pods are killed.
d
How I can configure node_selector per connection?
hey, @kapa.ai can I somehow set node_selector on connection?
k
A message from kapa.ai
No, there is no way in Airbyte today to set a 
nodeSelector
per connection. Node selectors for job pods are controlled globally via environment variables and/or Helm values, and (optionally) by job type, not by connection: - Global job pod selector: 
JOB_KUBE_NODE_SELECTORS
(applies to sync jobs and as default for others). [[Config jobs](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs)] - Per‑job‑type selectors: - 
SPEC_JOB_KUBE_NODE_SELECTORS
- 
CHECK_JOB_KUBE_NODE_SELECTORS
- 
DISCOVER_JOB_KUBE_NODE_SELECTORS
[[Config jobs-specific](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs-specific)] For multi‑node‑group setups, you configure this at the Helm 
values.yaml
level (e.g. using 
global.jobs.kube.nodeSelector
or 
JOB_KUBE_NODE_SELECTORS
and the job‑specific variants), but always cluster‑wide, not per connection. [[Multiple node groups](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#multiple-node-groups)] The knowledge sources do not mention any mechanism to override node selectors on a per‑connection basis.
d
@kapa.ai So there only the way all on jobs on spots or all jobs on demand instances?
k
A message from kapa.ai
You’re not limited to “all jobs on spot” vs “all jobs on on‑demand,” but the control is by job type and globally, not per connection. From the docs: - You can set global defaults for all job pods via: - 
JOB_KUBE_NODE_SELECTORS
(for sync jobs and as default for others). [[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs)] - You can override per job type: - 
SPEC_JOB_KUBE_NODE_SELECTORS
- 
CHECK_JOB_KUBE_NODE_SELECTORS
- 
DISCOVER_JOB_KUBE_NODE_SELECTORS
[[Jobs-specific vars](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs-specific)] And in a multi‑node‑group setup you typically: - Pin static/control‑plane components (server, webapp, keycloak, etc.) to one node group. - Pin job workloads (connector pods) to another node group via 
JOB_KUBE_NODE_SELECTORS
and the job‑specific variants (Helm 
values.yaml
examples show 
type=jobs
for job pods, 
type=static
for control plane). [[Multiple node groups](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#multiple-node-groups)] What you cannot do based on the docs: - No way to set 
nodeSelector
per connection. - No documented way to mix spot/on‑demand selection within the same job type (e.g., some syncs on spot, some on on‑demand) using connection‑level config. So you can: - Keep control‑plane on on‑demand and send all sync jobs to a “jobs” node group (which you can back with spot nodes). - Or keep all jobs on on‑demand. But the knowledge sources do not describe a finer‑grained, per‑connection routing mechanism.
Open in Slack
PreviousNext
Powered by