Zach Schmid
12/04/2025, 7:24 PMkapa.ai
12/04/2025, 7:25 PMHttpSubStream / nested streams, where the child stream’s stream_slices read records from the parent stream and use fields (like IDs) to build the child requests. [[How does sync mode work](https://discuss.airbyte.io/t/587); [Data extraction based on multiple id](https://discuss.airbyte.io/t/1752)] For config-based connectors, the same pattern is expressed via a SubstreamPartitionRouter for nested streams. [[Nested streams](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/partition-router#nested-streams)] ### 2. Avoid re-reading the entire parent on every child sync By default, HttpSubStream re-reads all parent records in stream_slices, which means a full walk of the parent each time. To reduce this: - Enable caching on the parent stream (use_cache = True in Python CDK), so the parent’s records are read once and reused for the substream during that sync. [[How does sync mode work](https://discuss.airbyte.io/t/587)] - If you want true incremental behavior for both parent and child, you must: - Make both parent and child incremental and share the same cursor field. - Implement a custom stream_slices on the child that reads the parent using SyncMode.incremental (instead of the default full refresh behavior). [[How does sync mode work](https://discuss.airbyte.io/t/587)] The docs and forum explicitly note that the default HttpSubStream implementation calls the parent with SyncMode.full_refresh, so you need to override that if you only want to process updated parent records. [[How does sync mode work](https://discuss.airbyte.io/t/587)] ### 3. Why not a single stream? Putting all three calls into one “mega-stream” via transformations is not the recommended pattern in the provided material. The guidance for “multiple-step” or dependent APIs (Yandex Metrica, Snapchat ads, CRM SDK) is consistently to use nested/substreams and stream slices, not to collapse everything into one stream. [[CDK Yandex Metrica](https://discuss.airbyte.io/t/1867); [Iterate across multiple endpoints](https://discuss.airbyte.io/t/1069); [SubStreams for Non HTTP Source](https://discuss.airbyte.io/t/1706)] --- So, best practice with the information available: - Implement three streams with substream / nested-stream relationships. - Use caching on parent streams to avoid re-fetching within a sync. - For “only updated parent records trigger child calls” across syncs, make both parent and child incremental and override the child’s stream_slices to read the parent in incremental mode instead of full refresh.Tarush Khatri
12/04/2025, 11:59 PMkapa.ai
12/04/2025, 11:59 PMkapa.ai
12/04/2025, 11:59 PMError, Exception, or Fail to locate error details. [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs)] You can also: - Download logs (three dots → Download logs) to inspect them locally. [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs)] These logs show connector behavior and errors, but not every raw HTTP request/response. 2. Understand current limitations around HTTP call logging There is an open GitHub issue requesting exactly what you’re asking for: logging the actual HTTP calls (URL, headers, query params, cursor, etc.) for debugging custom connectors. [[Log HTTP calls issue](https://github.com/airbytehq/airbyte/issues/52077)] A maintainer notes that: - The Python CDK already has infrastructure for debug‑level logging (via DEBUG=true), and - There’s been discussion about adding a log‑level option to connector CLIs to emit more detailed HTTP logs, but this is not yet implemented as a general feature. [[Log HTTP calls comments](https://github.com/airbytehq/airbyte/issues/52077)] So, based on the sources, there is no documented, supported way in Airbyte Cloud today to see every HTTP request/response for a connector run. 3. Workarounds (from community feedback) In that same GitHub issue, a user describes a workaround: routing traffic through an HTTP proxy (e.g., nginx) with logging enabled to capture the actual requests while debugging a connector. [[Log HTTP calls comments](https://github.com/airbytehq/airbyte/issues/52077)] This is not an official feature, but it’s one way people inspect calls when they control the network path (more applicable to self‑managed than Cloud). --- Summary - In current Airbyte Cloud, you can see sync logs and error messages, but not a full trace of each HTTP request/response. - Detailed HTTP call logging is a requested but not yet delivered feature. - For now, use the Job history → View logs / Download logs for error investigation, and consider a proxy‑based approach only if you’re on a deployment where you control networking.