is there recommended way to get data from amplitude to big query? seems my implementation that runs on k8s to straggle to keep up

Syncing Amplitude to BigQuery with Airbyte Yes! Airbyte has a native Amplitude source connector that can sync data directly to BigQuery, which is the recommended approach. Amplitude Source Connector The connector supports these streams: • Active Users Counts (Incremental) • Average Session Length (Incremental) • Events (Incremental) • Annotations, Cohorts, Events List It supports both Full Refresh and Incremental sync modes. [Amplitude Source Docs] Tips for Keeping Up with Large Data Volumes If your current implementation is struggling, here are the key things to tune: 1. Reduce the

Request Time Range

— The Events stream (Export API) can time out or hit a 4GB export limit for large date ranges. Lowering this value (default: 24 hours) breaks requests into smaller chunks and avoids timeouts. [Amplitude Source Docs] 2. Use Incremental Sync — Make sure your streams are configured for incremental sync where supported, so you're not re-syncing all historical data on every run. 3. Known issue with rate limiting (v0.7.29) — If you're on connector version

0.7.29

, there was a known bug where added rate limiting caused HeartbeatTimeout failures on the events stream (emitting state messages but almost no records, stalling for 24+ hours). This was fixed in

v0.7.30

by reverting the overly restrictive rate limiting. Make sure you're on the latest connector version. [GitHub PR #75123] 4. EU Data Region — If your Amplitude project is in the EU, make sure to select the EU Residency Server data region in the connector config to avoid unnecessary latency. [Amplitude Source Docs] BigQuery Destination Performance For large syncs to BigQuery, consider: • Aligning your cursor fields with BigQuery table partitioning/clustering for optimal query performance • Using per-stream state (Airbyte tracks this automatically) so each stream resumes independently if a sync fails Running on Kubernetes The Amplitude connector does support Kubernetes (AIRBYTE_ENTRYPOINT support was added in v0.1.0). If you're self-hosting on K8s and struggling, it may be worth checking your pod resource limits, as large event exports can be memory/CPU intensive. Would you like more detail on any specific part of the setup? 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai does it make difference to use gcs staging on big query ?

GCS Staging for BigQuery Destination Yes, using GCS staging can make a significant difference. Based on the knowledge sources: Performance & Reliability • Large syncs — GCS staging is the recommended approach for large data volumes. Users syncing millions of rows have found that standard inserts fail while GCS staging handles the load better. [Forum Post] • Slow syncs — There is a known performance consideration with GCS staging: the

BigQueryBulkLoader

uses only 2 compression workers (

numPartWorkers = 2

) by default, which can bottleneck throughput on systems with many CPU cores since GZIP compression is CPU-intensive. [GitHub Issue] When Standard Inserts May Be Preferable • For smaller syncs, standard inserts work fine and avoid the added complexity of GCS bucket setup and permissions. • GCS staging requires the service account to have list, read, and write permissions on the bucket — misconfigured permissions are a common source of failures. [Forum Post] Known Issues with GCS Staging • If your GCS staging bucket is missing or inaccessible, versions prior to

3.0.20

would return a generic system error rather than a clear config error. This was fixed in destination-bigquery v3.0.20. [GitHub PR] • CSV files containing ASCII null characters could fail to load via GCS staging — fixed in destination-bigquery v3.0.5. [GitHub Issue] Summary For your Kubernetes setup with Amplitude → BigQuery, GCS staging is likely the better choice given the potentially large event volumes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai i am doing a backfill on amplitude and has to process 600M records and currently my pod dies i have 8GB memory

Handling a 600M Record Amplitude Backfill With 8GB memory and pods dying, here are the key things to address: Root Cause The Amplitude Export API enforces a 4GB export size limit and large data volumes can trigger HTTP 504 timeouts. At 600M records, you're almost certainly hitting both. [Amplitude Source Docs] Most Important Fix: Reduce Request Time Range The single most impactful change is to reduce the Request Time Range in your Amplitude connector config. This breaks the export into smaller chunks per API request, preventing both the 4GB limit and timeout errors. > Default is 24 hours — try reducing to 1–6 hours for a backfill of this size. [Amplitude Source Docs] Memory / Resource Tuning on Kubernetes The source worker reads up to 10,000 records in memory at a time. For large row sizes this adds up fast. [Scaling Airbyte] You can increase memory limits for connector pods in your `values.yaml`:

global:
  jobs:
    resources:
      limits:
        cpu: 250m
        memory: 2Gi   # increase as needed
      requests:
        cpu: 75m
        memory: 500m

[Enterprise Scaling Guide] Additional Recommendations • Use Incremental sync after the backfill completes — don't re-process 600M records on every run • Use GCS Staging on the BigQuery destination for large volumes (as discussed previously) • Monitor pod memory with

kubectl describe pod

to confirm whether it's an OOM kill (exit code 137) vs. a timeout Important Note The knowledge sources do not contain specific benchmarks for Amplitude at 600M records, so the exact memory requirement is uncertain. You may need to iteratively increase memory and decrease the time range until the sync stabilizes. Starting with a very short time range (e.g., 1–2 hours) to validate the pipeline works before expanding is a safe approach. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai this is what i have defined as memory and cpu # Connector pod resources — override connector-registry defaults to prevent OOM kills # on large Amplitude events syncs. Applies globally to all source/destination containers. # CPU: 1 core request / 2 core limit gives each connector room to burst without starving # neighbours. With 10 concurrent syncs (~30 pods) this reserves ~30 cores cluster-wide. workloads: resources: useConnectorResourceDefaults: false mainContainer: memory: limit: 8Gi request: 6Gi cpu: limit: "2" request: "1"

Reviewing Your Resource Configuration Your configuration looks reasonable, but there are a few important caveats based on the knowledge sources. Known Issue: Resource Settings May Be Ignored There is a known bug where the container orchestrator ignores memory limits and always defaults to 1Gi, regardless of what you set. The root cause is that persisted actor definitions in the Airbyte database can override your Helm values, even with

useConnectorResourceDefaults: false

. [GitHub Issue] How to Verify Your Settings Are Actually Applied Check the job logs in the Airbyte UI — when a sync starts, it logs the actual resource requirements being used:

platform > [source] image: airbyte/source-amplitude:x.x.x resources: 
ResourceRequirements(limits={memory=8Gi, cpu=2}, requests={memory=6Gi, cpu=1} ...)

If you see different values than what you configured, your settings are being overridden. [Debugging Connector Resources] Workaround If Settings Are Ignored Set resources directly at the connection level via SQL:

UPDATE connection
SET resource_requirements = '{
  "cpu_request": "1",
  "cpu_limit": "2",
  "memory_request": "6Gi",
  "memory_limit": "8Gi"
}'
WHERE id = '<your-connection-id>';

[GitHub Issue] Also Set Job-Type Specific Variables A separate known issue is that

JOB_MAIN_*

variables only apply to sync jobs — check, discover, and spec jobs fall back to cluster

LimitRange

defaults. Make sure you also set these in `global.env_vars`:

global:
  env_vars:
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: 6Gi
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: 8Gi
    CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi
    CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT: 4Gi
    DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi
    DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT: 4Gi

[GitHub Issue] Bottom Line 1. Verify the settings are actually being applied via job logs 2. If not, use the SQL workaround at the connection level 3. Don't forget to also reduce the Amplitude

Request Time Range

— this is likely as important as memory tuning for a 600M record backfill 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai it seems they are not ignored. I have also set the time range to 4. Is there more i can do to make it work?

Additional Steps to Stabilize the Backfill Good news that the resource settings are being applied. With 8Gi memory and a 4-hour time range still causing pod deaths, here are the remaining options: 1. Reduce Time Range Further 4 hours may still be too large for 600M records. Amplitude's own documentation recommends using their Amazon S3 destination for large exports instead of the REST API when volumes are this high. Try reducing to 1–2 hours to further shrink each API request. [Amplitude Source Docs] 2. Confirm It's OOM vs. Timeout Run

kubectl describe pod <pod-name>

on the dying pod and check: • Exit code 137 = OOM kill → memory issue • Exit code 1 or connection reset → likely API timeout This tells you whether to focus on memory or time range tuning. [GitHub Issue] 3. GCS Staging on BigQuery Destination If not already enabled, switch to GCS staging. It offloads data more efficiently for large volumes rather than holding more in memory on the destination side. [Forum Post] 4. Orchestrator Memory May Still Be a Bottleneck Even if your main container has 8Gi, the orchestrator container is a separate container in the same pod and has a known bug where it always defaults to 1Gi regardless of configuration. This can cause OOM on the orchestrator side even when your source/destination containers are fine. [GitHub Issue] Unfortunately, the knowledge sources indicate there is currently no reliable way to override orchestrator memory — it is an open bug being tracked by the Airbyte engineering team. Summary of What to Try I'm sorry I can't give a more definitive answer — the knowledge sources don't contain specific guidance beyond these options for Amplitude backfills at this scale. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai failure seems to be on the source pod that consumes all the memory

Source Pod Consuming All Memory Since the failure is confirmed on the source pod (exit code 137, OOM kill), the issue is specifically with how the Amplitude connector reads and buffers data. Most Likely Cause The knowledge sources do not contain Amplitude-specific memory profiling details, but based on what's documented for large syncs: • The source worker reads up to 10,000 records in memory at a time • For large row sizes (e.g. Amplitude event objects with many properties), this can easily exhaust 8Gi [Scaling Airbyte] What You Can Still Try 1. Reduce time range to 1 hour or less — this is the most direct lever you have to reduce how much data the source pod processes per chunk 2. Increase memory further — update the connection-level resource requirement directly via SQL to go beyond 8Gi:

UPDATE connection
SET resource_requirements = '{
  "cpu_request": "1",
  "cpu_limit": "2",
  "memory_request": "8Gi",
  "memory_limit": "16Gi"
}'
WHERE id = '<your-connection-id>';

[GitHub Issue] 3. Check your Amplitude connector version — make sure you're on the latest version, as older versions (e.g.

0.7.29

) had bugs causing stalled syncs and excessive resource usage [GitHub PR] Honest Limitation The knowledge sources do not contain specific guidance on tuning the Amplitude source connector's internal memory behavior beyond these options. If reducing the time range to 1 hour and increasing memory still doesn't help, this may require opening a GitHub issue with Airbyte directly, as it could indicate a memory leak or inefficiency in the connector itself — similar to what was reported for other connectors. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai should i still add jobs config and gcs

Should You Add Jobs Config and GCS Staging? Jobs Config (JOB_MAIN_* variables) Since you've confirmed your resource settings are being applied to the source pod, adding the

JOB_MAIN_*

env vars is less critical for your current issue. However, they are still useful to set for check, discover, and spec jobs which fall back to cluster

LimitRange

defaults if not set. [GitHub Issue] For your immediate backfill problem though, this won't make a difference. GCS Staging Since your failure is on the source pod (not the destination), switching to GCS staging will not help with the current OOM issue. GCS staging only affects how data is loaded into BigQuery on the destination side. [Forum Post] What Will Actually Help Right Now Focus on these two things: 1. Reduce the Amplitude time range to 1 hour or less — this is your most direct lever 2. Increase source pod memory beyond 8Gi via the connection-level SQL workaround if your cluster can accommodate it GCS staging is still a good practice for large BigQuery loads in general, but it won't solve your current source pod OOM problem. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai sometimes though i am getting this error Warning from destination: 500 Internal Server Error PUT https://bigquery.googleapis.com/upload/bigquery/v2/projects/gowish-prod/jobs?uploadType=resumable&upload_id=AAVLpEgQO7sMITAsyBHFv8UAcQiwk1aMJar8mDvtVXeDD7J5n4EFIXtVGNbzUpffHwoXxmZWg3VxEIG3IPtlUFwK7wR-L2fD2Uo8MNxFbqRN8A { "error": { "code": 500, "message": "An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: https://cloud.google.com/bigquery/sla. If the error continues to occur please contact support at https://cloud.google.com/support. Error: 7285646", "errors": [ { "message": "An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: https://cloud.google.com/bigquery/sla. If the error continues to occur please contact support at https://cloud.google.com/support. Error: 7285646", "domain": "global", "reason": "internalError" } ], "status": "INTERNAL" } } com.google.cloud.bigquery.BigQueryException: 500 Internal Server Error PUT https://bigquery.googleapis.com/upload/bigquery/v2/projects/gowish-prod/jobs?uploadType=resumable&upload_id=AAVLpEgQO7sMITAsyBHFv8UAcQiwk1aMJar8mDvtVXeDD7J5n4EFIXtVGNbzUpffHwoXxmZWg3VxEIG3IPtlUFwK7wR-L2fD2Uo8MNxFbqRN8A { "error": { "code": 500, "message": "An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: https://cloud.google.com/bigquery/sla. If the error continues to occur please contact support at https://cloud.google.com/support. Error: 7285646", "errors": [ { "message": "An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: https://cloud.google.com/bigquery/sla. If the error continues to occur please contact support at https://cloud.google.com/support. Error: 7285646", "domain": "global", "reason": "internalError" } ], "status": "INTERNAL" } }

BigQuery 500 Internal Server Error This is a transient BigQuery-side error, not an Airbyte bug. The error message itself confirms this: > "This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem" This exact error (code

500

, reason

internalError

, error

7285646

) has been seen by other Airbyte users doing large syncs to BigQuery. [GitHub Issue] What It Means • BigQuery's resumable upload API is failing intermittently on Google's side • It is not related to your Amplitude source pod OOM issue — this is a separate destination-side problem • It typically resolves on retry What You Can Do 1. Let Airbyte retry — the BigQuery destination connector has built-in retry logic with backoff for transient errors. If it's truly transient, the sync should recover automatically 2. Switch to GCS Staging — this is actually a good reason to reconsider GCS staging. By staging data in GCS first before loading into BigQuery, you reduce the risk of long-running resumable uploads failing mid-way. [Forum Post] 3. Monitor frequency — if this error occurs on every sync rather than occasionally, it may indicate a deeper issue with your BigQuery project quota or Google Cloud infrastructure, in which case contacting Google Cloud support with the error code

7285646

is the right path Revised Recommendation Given you're now seeing both source OOM issues and destination 500 errors, enabling GCS staging is now more justified — it won't fix the source OOM, but it will make the destination side more resilient for your large backfill. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Happened to see this question in my digest email. In case helpful: Amplitude has a direct-to-BigQuery solution - I've used it before and it works very well. Means you don't have to use your own compute or maintain the pipeline. The only downside I saw is that it creates a very large number of data transfer jobs (cost of these was low, though).

Thanks @Padraig Alton This is very helpful. I had seen the direct-to-Big Query solution but i wasnt sure how it works. Do you know if it works well(eg fast etc) during backfills?

Yes, it's performant for backfills because it's done via data transfers. You just configure a service account with appropriate permissions for Amplitude to use. It isn't quite 'live' in normal operation (the transfers are frequent-but-batched), but backfills are good and done with a few clicks in the UI.

Thanks again :) Sounds very good, if i have to backfill again i will follow that approach. Currently Airbyte is quite slow to backfill.