https://docs.airbyte.com/integrations/sources/gong?_gl=1*13zxfo1*_gcl_au*OTQ5MTk3MjQ0LjE3Nzk4NDQwOTEuMTY5MDkzMDIwNS4xNzgwMDEyMjE5LjE3ODAwMTIyMTk.*_ga*MTQwMTcxMTYyNi4xNzQ3NjA1MzQ5*_ga_HDBMVFQGBH*czE3ODEwNjQ3NjkkbzE0OSRnMSR0MTc4MTA2NTA3NSRqNDQkbDAkaDE4MzYwMjgxMDY.#performance-considerations why does the call transcripts stream in the gong connector fetching one call at a time vs just bulk fetch?

The attached image and the knowledge sources explain this well. The

call transcripts

stream fetches one call at a time because it was redesigned as a substream of the

calls

stream using the

SubstreamPartitionRouter

pattern. Here's why this design was chosen: The core reason: enabling incremental sync Previously, the stream used a bulk-fetch approach with a

fromDateTime

filter, but this had a critical bug — the

fromDateTime

was hardcoded to

config['start_date']

, meaning every sync re-fetched all transcripts from the original start date rather than only new ones. [GitHub Issue] To properly fix this and enable true incremental sync, the stream was converted to a substream of

calls

(which is already incremental). This means: [PR #75132] • The parent

calls

stream provides only the new call IDs since the last sync (via

incremental_dependency: true

) • The

call transcripts

stream then fetches transcripts one call ID at a time for only those new calls • This allows subsequent incremental syncs to fetch only transcripts for new calls, significantly reducing data transfer and API usage The trade-off As the docs note (and your attached image confirms): > "The call transcripts stream fetches transcripts one call at a time as a substream of the calls stream. On the initial sync for accounts with a large number of calls, this may take longer than a bulk-fetch approach. Subsequent incremental syncs only fetch transcripts for new calls." So the initial sync is slower than a bulk approach, but subsequent incremental syncs are much faster and more efficient since they only process new calls. [Gong Source Docs] In short: the one-at-a-time approach is the deliberate cost of enabling proper incremental sync, which is more valuable for ongoing syncs than optimizing the one-time initial load. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

but i hit rate-limiting so easily now

Yes, this is a known issue. The one-call-at-a-time substream pattern significantly increases the number of API requests, which makes hitting Gong's rate limits (3 requests/second, 10,000 requests/day) much easier — especially on the initial sync for large accounts. What's been done to address this: 1. Better 429 handling (v1.2.2): A fix was merged that improves how the connector handles HTTP 429 responses. Instead of exhausting retries with generic backoff, it now honors Gong's

Retry-After

header and backs off accordingly, so syncs should recover from rate limits rather than fail outright. [PR #78482] 2. Concurrency tuning (v1.2.0+): The connector added a configurable

num_workers

field (default:

4

, max:

10

). If you're hitting rate limits, you can lower

num_workers

to reduce concurrent requests. [PR #77858] Practical workarounds right now: • Set a

start_date

— leaving it blank triggers a 90-day lookback which generates a large volume of requests on the initial sync. Setting a recent start date reduces the number of calls to process. [GitHub Issue] • Lower

num_workers

in your connector config to reduce concurrency and stay within Gong's 3 req/s limit. If you're on Airbyte Cloud, you can also open a support ticket referencing issue #78480 for further assistance. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.