i have 3 different API endpoints i need to query f...
# ask-ai
z
i have 3 different API endpoints i need to query for a particular record. any incremental updates on api endpoint 1, should cause a query to api endpoint 2, and any updates to api endpoint 2, would then query api endpoint 3. what is the best way to handle this in airbyte? should this be a transformation inside of a single stream? or should these be handled as individual streams? if we go the substream/dependent stream route is there a way to prevent the substreams from resyncing all data every time, and only make requests based on the parent streams updated record?
k
You’ll want to model these as separate streams with substream / nested-stream logic, not as a single-stream transformation. ### 1. Model the 3 endpoints as parent + substreams Conceptually: - Stream A → API endpoint 1 (base / parent stream, incremental) - Stream B → API endpoint 2 (substream of A) - Stream C → API endpoint 3 (substream of B) In the Python CDK this is typically done with
HttpSubStream
/ nested streams, where the child stream’s
stream_slices
read records from the parent stream and use fields (like IDs) to build the child requests. [[How does sync mode work](https://discuss.airbyte.io/t/587); [Data extraction based on multiple id](https://discuss.airbyte.io/t/1752)] For config-based connectors, the same pattern is expressed via a
SubstreamPartitionRouter
for nested streams. [[Nested streams](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/partition-router#nested-streams)] ### 2. Avoid re-reading the entire parent on every child sync By default,
HttpSubStream
re-reads all parent records in
stream_slices
, which means a full walk of the parent each time. To reduce this: - Enable caching on the parent stream (
use_cache = True
in Python CDK), so the parent’s records are read once and reused for the substream during that sync. [[How does sync mode work](https://discuss.airbyte.io/t/587)] - If you want true incremental behavior for both parent and child, you must: - Make both parent and child incremental and share the same cursor field. - Implement a custom
stream_slices
on the child that reads the parent using
SyncMode.incremental
(instead of the default full refresh behavior). [[How does sync mode work](https://discuss.airbyte.io/t/587)] The docs and forum explicitly note that the default
HttpSubStream
implementation calls the parent with
SyncMode.full_refresh
, so you need to override that if you only want to process updated parent records. [[How does sync mode work](https://discuss.airbyte.io/t/587)] ### 3. Why not a single stream? Putting all three calls into one “mega-stream” via transformations is not the recommended pattern in the provided material. The guidance for “multiple-step” or dependent APIs (Yandex Metrica, Snapchat ads, CRM SDK) is consistently to use nested/substreams and stream slices, not to collapse everything into one stream. [[CDK Yandex Metrica](https://discuss.airbyte.io/t/1867); [Iterate across multiple endpoints](https://discuss.airbyte.io/t/1069); [SubStreams for Non HTTP Source](https://discuss.airbyte.io/t/1706)] --- So, best practice with the information available: - Implement three streams with substream / nested-stream relationships. - Use caching on parent streams to avoid re-fetching within a sync. - For “only updated parent records trigger child calls” across syncs, make both parent and child incremental and override the child’s
stream_slices
to read the parent in incremental mode instead of full refresh.
t
@kapa.ai why i cant see request for errors in current version of airbyte cloud ? How to check what calls airbyte is making?
k
A message from kapa.ai
Airbyte’s logs do not currently expose full HTTP request/response details for each API call in the way you’re looking for, and the knowledge sources don’t describe a built‑in way in Airbyte Cloud to see every underlying HTTP request. What you can do today: 1. Use connection logs in the UI For each sync, Airbyte records detailed logs you can browse in the UI: - Go to the connection’s Job history. - Click the three dots next to a sync → View logs. - Search for
Error
,
Exception
, or
Fail
to locate error details. [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs)] You can also: - Download logs (three dots → Download logs) to inspect them locally. [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs)] These logs show connector behavior and errors, but not every raw HTTP request/response. 2. Understand current limitations around HTTP call logging There is an open GitHub issue requesting exactly what you’re asking for: logging the actual HTTP calls (URL, headers, query params, cursor, etc.) for debugging custom connectors. [[Log HTTP calls issue](https://github.com/airbytehq/airbyte/issues/52077)] A maintainer notes that: - The Python CDK already has infrastructure for debug‑level logging (via
DEBUG=true
), and - There’s been discussion about adding a log‑level option to connector CLIs to emit more detailed HTTP logs, but this is not yet implemented as a general feature. [[Log HTTP calls comments](https://github.com/airbytehq/airbyte/issues/52077)] So, based on the sources, there is no documented, supported way in Airbyte Cloud today to see every HTTP request/response for a connector run. 3. Workarounds (from community feedback) In that same GitHub issue, a user describes a workaround: routing traffic through an HTTP proxy (e.g., nginx) with logging enabled to capture the actual requests while debugging a connector. [[Log HTTP calls comments](https://github.com/airbytehq/airbyte/issues/52077)] This is not an official feature, but it’s one way people inspect calls when they control the network path (more applicable to self‑managed than Cloud). --- Summary - In current Airbyte Cloud, you can see sync logs and error messages, but not a full trace of each HTTP request/response. - Detailed HTTP call logging is a requested but not yet delivered feature. - For now, use the Job history → View logs / Download logs for error investigation, and consider a proxy‑based approach only if you’re on a deployment where you control networking.