@kapa.ai I am having an issue with a Google Ads sync that is very large. The sync is taking 19+hours to run and seems like it is only using one core of the container it is running in. I have the concurrent workers set to 10 and this did not improve the source read times. I have found a env variable named CONCURRENT_SOURCE_STREAM_READ that is set to false, would setting this to true help to speed this up? Are there any other recommendations?

The knowledge sources don’t mention the

CONCURRENT_SOURCE_STREAM_READ

env var at all, so I can’t say with confidence that toggling it to

true

will help or is even supported in your version. I’d be cautious about relying on it without explicit documentation. From the available docs and issues, here’s what is documented and recommended for speeding up large syncs: 1. Use connector-level concurrency (where supported) Some connectors (including Google Ads) expose a

num_workers

/ concurrency parameter in their config. For Google Ads specifically, the Terraform schema shows a

num_workers

field: “The number of concurrent workers to use for syncing. Increasing this value may speed up syncs for accounts with many customers or streams. Adjust based on your API usage and rate limits.” [Google Ads schema] You mentioned setting concurrent workers to 10; make sure this is the connector’s own

num_workers

setting (in the source configuration), not just a platform-level worker count. 2. Understand concurrency in custom / concurrent sources The “concurrent” CDK pattern (e.g.,

ConcurrentSourceAdapter

) is what enables reading multiple stream slices in parallel inside a connector. It’s configured via a

concurrency_level

derived from config (like

num_workers

) and internal constructs such as

ConcurrentSource

. [Concurrent tutorial] The docs do not mention

CONCURRENT_SOURCE_STREAM_READ

as a user-facing toggle; concurrency is controlled in code and via connector config, not via that env var in the docs we have. 3. Scale Airbyte workers vs. per‑sync speed Increasing the number of Airbyte worker instances / replicas mainly affects how many syncs can run in parallel, not how fast a single sync runs. [Scaling Airbyte] So raising

MAX_SYNC_WORKERS

or

worker.replicaCount

will help if you run many connections at once, but won’t necessarily speed up one long Google Ads sync. 4. Resource limits and CPU usage Platform docs recommend monitoring requested vs. used CPU and memory and adjusting connector pod limits/requests in

values.yaml

. Under‑provisioned jobs can be slow or get stuck; increasing CPU/memory limits for the job can improve throughput. [Scaling Airbyte] If your container shows only one core busy, check whether the connector itself is single‑threaded for that workload; giving it more CPU may still help (fewer context switches, faster single thread), but won’t create parallelism if the connector doesn’t use it. 5. “Faster sync speed” initiative Airbyte has a “faster sync speed” initiative that can make supported connectors 4–10x faster, but it only applies if: ◦ You’re on Airbyte 1.8+ (or Cloud), and ◦ Both source and destination connectors are on versions that support faster syncs. [Faster sync] If your Google Ads source and destination are not on the latest versions, upgrading them (and the platform, if self‑managed) is a concrete, supported way to improve speed. 6. Google Ads–specific considerations The open GitHub issue for Google Ads mentions timeouts and very large datasets, and the user experimented with changing query parameters (e.g., conversion window) to reduce load. [Google Ads issue] While that’s more about avoiding timeouts/OOM than pure speed, it suggests: ◦ Narrowing date ranges (shorter windows, more frequent syncs) ◦ Possibly splitting very heavy views/streams into smaller time slices if feasible Given the docs we have, concrete recommendations: • Ensure you’re using the Google Ads source’s

num_workers

setting (in the source config) and that it’s set to 10 as you intend. [Google Ads schema] • Upgrade to Airbyte 1.8+ (or later) and latest Google Ads + destination connectors so you benefit from the “faster sync speed” improvements, if you’re not already. [Faster sync] • Verify and, if needed, increase CPU/memory limits for the job in

values.yaml

so the connector isn’t constrained by low resource requests/limits. [Scaling Airbyte] • Consider reducing the time range per sync (e.g., shorter windows, more frequent runs) for the heaviest Google Ads streams to keep each job smaller and less likely to run 19+ hours. Because the documentation does not describe

CONCURRENT_SOURCE_STREAM_READ

, I cannot recommend enabling it as a supported or guaranteed way to speed up your sync. If you share your Airbyte version and how you set

num_workers

(UI vs. config JSON), I can help you double‑check that part against the documented behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I have found more in my investigation. The Google Ads connector supports the Number of Concurrent workers, but when I change the number of jobs that use it, it still only spawns one worker container.

The knowledge sources don’t describe the internal behavior of the Google Ads connector beyond exposing a

num_workers

field in its configuration, and they do not document how many Kubernetes/Docker containers a single sync will spawn. Because of that, I can’t definitively explain why you only see one worker container per job. What is documented: • The Google Ads source has a

num_workers

config option: “The number of concurrent workers to use for syncing. Increasing this value may speed up syncs for accounts with many customers or streams. Adjust based on your API usage and rate limits.” [Google Ads config] • Platform‑level concurrency (e.g.,

MAX_SYNC_WORKERS

, worker

replicaCount

) controls how many sync jobs can run in parallel across the instance, not how many containers a single sync uses. [Scaling Airbyte] From this, two important implications: 1.

num_workers

is connector‑internal concurrency, not pod count The docs only say it controls “concurrent workers to use for syncing” inside the Google Ads connector; they do not say it will create multiple job containers. It’s likely implemented as internal parallelism (threads/tasks) within a single container, so seeing only one worker pod for that sync is consistent with the available documentation. [Google Ads config] 2. More pods vs. faster single sync Scaling Airbyte workers (

MAX_SYNC_WORKERS

,

worker.replicaCount

) is about running more connections at once, not speeding up one long‑running connection. [Scaling Airbyte] So even if

num_workers

is set to 10, you should still expect one job container per sync; the extra concurrency is inside that container. Given the limits of the docs, I can’t confirm: • Whether the current Google Ads connector version actually uses

num_workers

effectively in all cases. • Whether any env var (like

CONCURRENT_SOURCE_STREAM_READ

) would change its behavior. Actionable checks based on what is documented: • Confirm that

num_workers

is indeed set in the source configuration (not just Terraform defaults) and that the running job’s config JSON includes the expected value. [Google Ads config] • Ensure the job container has enough CPU/memory limits so internal concurrency isn’t throttled by low resource limits. [Scaling Airbyte] • If you need more throughput and the single job remains slow even with higher

num_workers

, consider splitting the workload (e.g., by date ranges into multiple connections) so multiple sync jobs can run in parallel, which is a supported scaling pattern. [Scaling Airbyte] Beyond this, the knowledge sources don’t provide more detail on why your specific Google Ads sync isn’t speeding up, so I can’t give a more precise root cause from the docs alone. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai ok, how can I check if this is being honored in the pod?

You can only use what’s documented: inspect the effective source configuration and the job logs from inside the running pod. The docs don’t show connector‑internal logging for

num_workers

, so you won’t see a dedicated “num_workers=10” line, but you can at least verify that the value is present in the config the pod is using. Here’s how, based on the Kubernetes/logs docs: 1. Exec into the job pod and inspect config files Identify the replication pod for your job, then exec into it and list `/config`:

kubectl exec -it <pod-name> -n <namespace> -c main bash
   # inside the pod
   ls /config

You should see files like

source_config.json

or similar (names vary by job type, but the pattern is like the destination example in the docs). [Browsing logs] Copy the relevant config file locally and inspect it:

kubectl cp <namespace>/<pod-name>:/config/source_config.json ./source_config.json
   cat ./source_config.json

Confirm that

num_workers

(or the “Number of Concurrent workers” field) is present and set to the value you expect. [Browsing logs; Google Ads config] 2. Check job logs for resource usage / configuration Airbyte logs the resource requirements and container setup when launching the replication pod. While the example in the docs focuses on CPU/memory, you can use the same logs to confirm that the correct source configuration is being picked up (look for the printed config or any mention of Google Ads settings). [Configuring resources] In the Airbyte UI, open the job, go to logs, and verify that: ◦ The Google Ads source is the expected version. ◦ The job is using the source you edited (matching source ID / name). If

source_config.json

(or equivalent) does not contain your

num_workers

value, then the setting isn’t reaching the connector. If it does contain the correct value but performance doesn’t change, the docs don’t provide further visibility into how the connector uses it internally, so we can’t confirm more from the available information. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.