https://linen.dev logo
Join Slack
Powered by
Hi <@U04S8PLJWMT> what kind of instance type are r...
# ask-ai
i
Hi @kapa.ai what kind of instance type are reocmmended in aws eks for airbyte deployment with medium to hihgh workloads ?
k
For medium to high workloads on AWS EKS, the docs recommend: - Use an Amazon EKS cluster on EC2, not Fargate. - Prefer memory‑optimized instance families, e.g. M7i / M7g. - Use nodes with at least 2 vCPUs and 8 GiB RAM per node as a minimum; scale up cores/RAM and node count based on your workload. [[Enterprise prereqs](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites); [1.8 prereqs](https://docs.airbyte.com/platform/1.8/enterprise-setup/implementation-guide#infrastructure-prerequisites)] More generally, Airbyte suggests starting with a mid‑sized instance (4–8 cores) and then tuning instance size and cluster node count as you observe actual sync concurrency, CPU, and memory usage. [[Scaling what to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)]
i
@kapa.ai can i pass some metadaata for creating a job in airbyte ?
k
A message from kapa.ai
You can’t attach arbitrary custom metadata when creating a job via the public concepts/APIs described in the docs, based on the material available in these sources. What exists today: - Per-record metadata fields that destinations automatically add to data records, such as 
_airbyte_raw_id
, 
_airbyte_generation_id
, 
_airbyte_extracted_at
, 
_airbyte_loaded_at
, and 
_airbyte_meta
(which itself contains 
sync_id
and a list of 
changes
). These are managed by Airbyte, not user-supplied at job creation time.[[Metadata fields](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-metadata-fields)] - Jobs database 
metadata
column: internal Airbyte jobs table has a 
metadata jsonb
column for each job, but the docs only describe that it “contains metadata for the job” and do not expose a supported way for users to populate it themselves when starting jobs.[[Jobs DB](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database)] Because the documentation does not describe any supported mechanism to pass your own metadata payload when creating/scheduling a job, I’m sorry, but there’s no information indicating that this is currently possible via documented, stable interfaces.
i
@kapa.ai is there a way to do it via an api ? attach metadata to a job ?>
k
A message from kapa.ai
Based on the available docs, there is no documented API support for attaching custom metadata to a job at creation time: - The jobs table in Airbyte’s internal Jobs Database has a 
metadata jsonb
column (“JSON blob containing metadata for the job”), but the docs only describe its structure and indexes; they do not describe any public API for you to set this field yourself when starting a sync or reset.[[Jobs DB](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database); [Jobs DB 1.8](https://docs.airbyte.com/platform/1.8/understanding-airbyte/database-data-catalog#jobs-database)] - The public docs about metadata focus on record-level metadata fields (
_airbyte_raw_id
, 
_airbyte_meta
, etc.), all of which are managed by Airbyte and not user-injected per job.[[Metadata fields](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-metadata-fields)] Since none of the referenced API or database docs describe a supported way to pass arbitrary metadata when creating a job, the knowledge sources do not indicate that this is currently possible via a documented API.
Open in Slack
PreviousNext
Powered by