https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I'm trying to create a new custom c...
# replication-ask-ai
a
@kapa.ai I'm trying to create a new custom connector using the builder for Branch.io, specifically the cross-events export API. I have properly configured it to be asynchronous job and set up the required fields. However when testing it I'm getting an error. Traceback logs in thread
k
Thinking completed Based on the knowledge sources, it looks like you may be hitting the same issue described in a GitHub issue where the Connector Builder fails to read 
.gz
files in the download tab of an asynchronous stream. [GitHub Issue] The Branch.io cross-events export API likely returns a 
.csv.gz
file as the download target — which matches exactly the scenario in that open issue. What's Happening The user in that issue configured: • HTTP Response Format: 
gzip
• Unzipped data format: 
CSV
...but still encountered an error during the download phase, even though creation and polling worked fine. This appears to be a known bug in the Connector Builder (as of version 6.48.15) where 
.gz
files cannot be properly read in the download tab. [GitHub Issue] What You Can Try 1. Verify your Download tab configuration matches the expected format: ◦ HTTP Response Format: 
gzip
◦ Unzipped data format: 
CSV
◦ URL field: 
{{ download_target }}
(referencing the URL extracted during polling) [Download Tab Docs] 1. Check the "Other Requests" panel after testing to confirm that creation and polling are succeeding, and that the download URL is being correctly extracted. [Testing Async Streams] 2. Share your traceback logs (as you mentioned you'll post in thread) — the specific error message will help narrow down whether this is the same 
.gz
reading bug or a different configuration issue. Current Limitation The knowledge sources indicate this is an open issue with no confirmed fix yet. If you are indeed hitting the 
.gz
download bug, there is currently no documented workaround in the Connector Builder UI for this specific case. You may want to: • Add a comment or upvote the GitHub issue to signal its impact. • Consider building a fully custom connector (outside the Builder UI) if you need this to work immediately. Please share your traceback logs and I can help further diagnose the issue! 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai below is the traceback logs
Copy code
Traceback (most recent call last):
  File "/app/airbyte_cdk/sources/streams/concurrent/partition_reader.py", line 79, in process_partition
    for record in partition.read():
                  ~~~~~~~~~~~~~~^^
  File "/app/airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py", line 91, in read
    for stream_data in self._retriever.read_records(
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self._schema_loader.get_json_schema(), self._stream_slice
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/app/airbyte_cdk/sources/declarative/retrievers/async_retriever.py", line 91, in read_records
    yield from self.record_selector.filter_and_transform(
    ...<4 lines>...
    )
  File "/app/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 121, in filter_and_transform
    for data in normalized_data:
                ^^^^^^^^^^^^^^^
  File "/app/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 137, in _normalize_by_schema
    yield from records
  File "/app/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 162, in _transform
    for record in records:
                  ^^^^^^^
  File "/app/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 154, in _filter
    yield from records
  File "/app/airbyte_cdk/sources/declarative/async_job/job_orchestrator.py", line 580, in fetch_records
    yield from self._job_repository.fetch_records(job)
  File "/app/airbyte_cdk/sources/declarative/requesters/http_job_repository.py", line 225, in fetch_records
    "polling_response": self._get_polling_response_interpolation_context(job),
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/app/airbyte_cdk/sources/declarative/requesters/http_job_repository.py", line 313, in _get_polling_response_interpolation_context
    polling_response_context = dict(self._polling_job_response_by_id[job.api_job_id()].json())
ValueError: dictionary update sequence element #0 has length 5; 2 is required
k
Thinking completed The knowledge sources do not contain enough information to provide a confident diagnosis of this specific error. However, I can share what the traceback reveals and relate it to what is documented. What the Error Means The error occurs in the download phase of the async stream:
Copy code
ValueError: dictionary update sequence element #0 has length 5; 2 is required
This happens in 
http_job_repository.py
when the CDK tries to build the 
polling_response
interpolation context by calling 
.json()
on the polling response and converting it to a 
dict
. The error suggests that the polling response body is not a valid JSON object (key-value pairs) — instead, it appears to be returning something like a list of strings or a non-dict-compatible structure (e.g., a list where the first element has length 5, not 2 key-value pairs). Likely Root Cause The Branch.io polling response may be returning a JSON array or some other non-object structure at the top level, rather than a JSON object (
{}
). The CDK expects the polling response to be a dictionary, so it can be used for interpolation via 
{{ polling_response }}
. What to Check 1. Inspect your polling response using the "Other Requests" panel in the Connector Builder UI after a test run. Look at the raw response body from the polling request. [Testing Async Streams] 2. Verify the Status Extractor field path is correctly pointing into the polling response structure. If the response is nested or array-based, the field path needs to reflect that. [Polling Tab Docs] 3. Verify the Download Target Extractor field path is correctly pointing to the download URL within the polling response. [Polling Tab Docs] Important Note This error is separate from the 
.gz
download bug mentioned in the GitHub issue. This error is crashing before the download phase even begins — it fails while trying to prepare the 
polling_response
context to construct the download request. So resolving this error is a prerequisite before you'd encounter any 
.gz
-related issues. [GitHub Issue] Could you share what your polling response body looks like from the "Other Requests" panel? That would help pinpoint the exact misconfiguration. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai the polling response is as below
Copy code
[
    {
        "job_id": "redacted",
        "created_date": "2026-10-13 12:24:12.913814",
        "completion_date": "2026-10-13 12:32:19.789",
        "status": "SUCCEEDED",
        "download_links": [
            "<https://branch-exports-web-usw2.s3.us-west-2.amazonaws.com/branch_agg_async_api/y%3D2026/m%3D10/d%3D13/app_id%3D644074824343244968/job_id%3Dredacted/task_id%3D0/filename-1.csv.gz>",
            "<https://branch-exports-web-usw2.s3.us-west-2.amazonaws.com/branch_agg_async_api/y%3D2026/m%3D10/d%3D13/app_id%3D644074824343244968/job_id%3Dredacted/task_id%3D0/filename-2.csv.gz>"
        ]
    }
]
k
Thinking completed The root cause is now clear. Your polling response is a JSON array (wrapped in 
[...]
) rather than a JSON object (
{...}
). The CDK's 
http_job_repository.py
calls 
.json()
and tries to convert it to a 
dict
, but since it's a list, it fails with:
Copy code
ValueError: dictionary update sequence element #0 has length 5; 2 is required
The "length 5" refers to the string 
"job_id"
— the first key of the first object in the array — which has 5 characters, not the 2 elements (key, value) that 
dict()
expects. The Problem The CDK expects the polling response to be a top-level JSON object (
{}
), but Branch.io returns a top-level JSON array (
[]
) containing one object. What to Try Unfortunately, the knowledge sources do not explicitly document a built-in way to handle a top-level array polling response in the Connector Builder's async stream configuration. [Polling Tab Docs] However, here are some things worth checking: 1. Status Extractor field path: Try setting the field path to 
[0]/status
or 
0/status
to index into the first element of the array, rather than just 
status
. This may or may not resolve the underlying 
dict()
conversion issue, as that happens before field extraction. 2. Download Target Extractor field path: Similarly, try 
[0]/download_links
or 
0/download_links
to point to the array of URLs. 3. Multiple download links: Your polling response also returns two 
.csv.gz
download URLs in the 
download_links
array. The knowledge sources do not document how the Connector Builder handles multiple download URLs from a single job — this may be an additional limitation to be aware of. [Download Tab Docs] Recommendation Given these two compounding issues: • Top-level array in polling response (not supported by CDK's 
dict()
conversion) • Multiple 
.gz
download URLs per job • The existing open bug with 
.gz
file reading [GitHub Issue] This connector may be beyond what the Connector Builder currently supports. You may need to build a fully custom Python connector using the Airbyte CDK directly, where you have full control over how the polling response is parsed and how multiple download URLs are iterated. I'd also recommend filing or commenting on a GitHub issue describing the top-level array polling response limitation, as it appears to be an unsupported edge case in the current CDK implementation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai below are my findings 1. Status Extractor field path: using just 
status
seem to have worked. Trying 
[0]/status
or 
0/status
would have kept the pipeline hanging as it is unable to find the status. 2. Download Target Extractor field path: 
[0]/download_links
or 
0/download_links
would avoid that ValueError failure but there is no record being fetched Any further advice you can provide on what we could try?
k
Thinking completed Thank you for sharing those findings — that's very helpful for narrowing things down. What We Know So Far • 
status
works for the Status Extractor (the CDK is somehow resolving it from the array's first element) • 
[0]/download_links
avoids the 
ValueError
but returns no records • The download links are an array of multiple 
.csv.gz
URLs The Core Remaining Problems There are likely two compounding issues at this point: 1. Multiple Download URLs Your 
download_links
field is itself an array of URLs:
Copy code
"download_links": [
    "https://...filename-1.csv.gz",
    "https://...filename-2.csv.gz"
]
The Download Target Extractor is designed to extract a single download URL or identifier. The knowledge sources do not document support for iterating over multiple download URLs from a single polling response. [Download Tab Docs] 2. The .gz File Reading Bug Even if the download URL is correctly extracted, there is an open bug where the Connector Builder fails to read 
.csv.gz
files in the download tab. [GitHub Issue] What You Could Still Try • Try setting the Download Target Extractor field path to 
[0]/download_links/0
to target just the first URL in the array. This would at least isolate whether the no-records issue is caused by the multi-URL array or the 
.gz
reading bug. Honest Assessment The knowledge sources do not contain enough information to confirm a working solution for your specific case. Given the combination of: 1. Top-level array polling response 2. Multiple 
.gz
download URLs per job 3. The open 
.gz
reading bug in the Connector Builder ...this connector is very likely beyond what the Connector Builder currently supports reliably. Building a fully custom Python connector using the Airbyte CDK directly is probably the most viable path forward at this point, as it would give you full control over array parsing, URL iteration, and 
.gz
decompression. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai I noticed in Download target Extractor there is another checkbox within called Record Expander. What is that feature and would that help with this particular case?
k
Thinking completed Great observation! The Record Expander is a documented component in the Airbyte CDK/YAML configuration. Here's what it does and whether it could help. What is Record Expander? The 
RecordExpander
component expands records by extracting items from a nested array field, emitting each item as a separate record. It supports wildcards (
*
) for matching multiple arrays. [YAML Reference] For example, if a record contains:
Copy code
{
  "download_links": [
    "url-1.csv.gz",
    "url-2.csv.gz"
  ]
}
With 
expand_records_from_field: ["download_links"]
, it would emit two separate records — one per URL. Would It Help Your Case? Potentially partially, but with caveats: • For the multiple download URLs problem: Record Expander could theoretically expand your 
download_links
array so that each URL is treated as a separate record/target. This directly addresses the issue of having multiple 
.csv.gz
URLs per job. [YAML Reference] • Useful options to consider: ◦ 
merge_parent: true
— merges parent record fields (like 
job_id
, 
status
) into each expanded item, so you retain context per URL ◦ 
on_no_records
— controls behavior if 
download_links
is missing or empty (
skip
or 
emit_parent
) Important Caveats The knowledge sources do not explicitly document how Record Expander behaves specifically within the Download Target Extractor context of an async stream. It's unclear whether the expanded items would each be used as separate download targets or just as separate output records. Additionally, the 
.gz
file reading bug remains an open issue regardless of how the download URLs are extracted. [GitHub Issue] Recommendation It's worth trying — configure the Record Expander with: • 
expand_records_from_field: ["download_links"]
(or 
[0]/download_links
given your top-level array response) • 
merge_parent: true
to retain job metadata on each expanded record Then test and observe whether records are returned. This would at least help isolate whether the no-records issue is the multi-URL array problem or the 
.gz
reading bug. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
@kapa.ai Field Path: 
download_links
--> Error: "ValueError dictionary update sequence element #0 has length 5; 2 is required" Field Path: 
0/download_links
or 
[0]/download_links
--> Test passes, but connection get no records loaded And we tried using Epland Records From Field instead, Expand Recrods From Field: 
download_links
--> Error: "ValueError dictionary update sequence element #0 has length 5; 2 is required" Expand Recrods From Field: 
0/download_links
or 
[0]/download_links
--> Test passes, but connection get no records loaded
k
Thinking completed Thank you for the thorough testing. The results are now very telling. What the Results Tell Us The consistent pattern across all configurations is: The fact that both the Download Target Extractor and Expand Records From Field produce identical results strongly suggests the problem is not with how the URLs are being extracted — it's what happens after extraction, i.e., the actual downloading and reading of the 
.csv.gz
files. Most Likely Remaining Cause This points squarely at the open 
.gz
file reading bug in the Connector Builder. Even when the download URL is correctly resolved, the Connector Builder cannot successfully decompress and parse the 
.csv.gz
content, resulting in zero records rather than an explicit error. [GitHub Issue] Honest Conclusion The knowledge sources do not contain enough information to provide a working workaround for this within the Connector Builder UI. Based on everything tested, you have hit two hard limitations: 1. The top-level array polling response requiring the 
0/
prefix workaround 2. The 
.gz
download reading bug preventing any records from loading Recommended Path Forward Given these findings, I would strongly recommend: 1. File a GitHub issue (or comment on the existing one) with your specific Branch.io findings — the top-level array polling response is a distinct additional bug worth reporting separately. 2. Build a custom Python connector using the Airbyte CDK directly, where you have full control over: ◦ Parsing the top-level array polling response ◦ Iterating over multiple download URLs ◦ Manually handling 
.gz
decompression The Connector Builder does not currently appear to support your specific combination of API response structures reliably. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
k
@kapa.ai the log for using 
[0]/download_links
for expand records from field is:
Copy code
2026-10-15 14:21:36 replication-orchestrator INFO ----- START REPLICATION -----
2026-10-15 14:21:36 replication-orchestrator INFO
2026-10-15 14:21:36 replication-orchestrator INFO Running destination...
2026-10-15 14:21:36 replication-orchestrator INFO Writing messages to protocol version 0.2.0
2026-10-15 14:21:36 replication-orchestrator INFO Reading messages from protocol version 0.2.0
2026-10-15 14:21:36 replication-orchestrator INFO Closing io.micronaut.context.propagation.instrument.execution.ContextPropagatingExecutorService@7951a08c
2026-10-15 14:21:36 replication-orchestrator INFO Closing io.micronaut.context.propagation.instrument.execution.ContextPropagatingExecutorService@7e168d90
2026-10-15 14:21:36 replication-orchestrator INFO Closing io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination@cf0e8b4
2026-10-15 14:21:36 destination ERROR io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform. Streams without a status message: [kw_branch_io_cross_event_new]
2026-10-15 14:21:36 destination ERROR 	at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116)
2026-10-15 14:21:36 destination ERROR 	at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:313)
2026-10-15 14:21:36 destination ERROR 	at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117)
2026-10-15 14:21:36 destination ERROR 	at io.airbyte.cdk.load.task.DestinationTaskLauncher$WrappedTask.execute(DestinationTaskLauncher.kt:124)
2026-10-15 14:21:36 destination ERROR 	at io.airbyte.cdk.load.task.TaskScopeProvider$launch$job$1.invokeSuspend(TaskScopeProvider.kt:35)
2026-10-15 14:21:36 destination ERROR 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
2026-10-15 14:21:36 destination ERROR 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
2026-10-15 14:21:37 replication-orchestrator ERROR Error closing resource io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination@cf0e8b4; recording failure but continuing.
2026-10-15 14:21:37 replication-orchestrator INFO Closing io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteSource@4c14ed24
2026-10-15 14:21:37 replication-orchestrator INFO Closing io.airbyte.container.orchestrator.worker.RecordSchemaValidator@26db7540
2026-10-15 14:21:37 replication-orchestrator INFO Closing io.airbyte.container.orchestrator.persistence.SyncPersistenceImpl@4dcdc172
2026-10-15 14:21:37 replication-orchestrator ERROR Killing orchestrator because of an Exception
k
Thinking completed This log reveals a different error from what we've been debugging. The failure is now:
Copy code
io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not 
receive a terminal stream status message.
Streams without a status message: [kw_branch_io_cross_event_new]
What This Means This is a separate, known bug where 
source-declarative-manifest
does not emit terminal 
STREAM_STATUS
messages required by CDK v3 destinations. [GitHub Issue] Critically, this error is happening at the destination level, not the source level. It means: • The source did complete reading (input was fully read) • But the destination never received the required terminal stream status message • This is unrelated to the 
.gz
or field path issues we were debugging earlier What This Tells Us About Your Previous Tests The "zero records" results you saw earlier may actually have been caused by this same terminal status bug silently failing the sync, rather than a 
.gz
reading problem. The 
.gz
bug may not be the blocker after all. Current Workarounds (from the GitHub issue) [GitHub Issue] 1. Downgrade your destination to a non-CDK v3 version: ◦ BigQuery → v2.x ◦ S3 → v0.6.7 or earlier (pre-v1.0.0) 1. Build a custom Python connector using the traditional Python CDK, which properly emits 
STREAM_STATUS
messages and works with any destination version. Important Context • This bug has been open since at least June 2025 • It was temporarily resolved by a code revert but the permanent fix was never implemented • It affects all 
source-declarative-manifest
connectors (including Connector Builder-based connectors) paired with CDK v3 destinations • There is no YAML-level configuration option to force stream status emission Recommended Next Steps 1. First, try downgrading your destination to a non-CDK v3 version to confirm this is the primary blocker — this will tell you whether the 
.gz
reading and field path issues are also real blockers or were masked by this error all along. 2. If downgrading resolves the terminal status error and records still don't load, then the 
.gz
issue is a real secondary blocker. 3. If downgrading resolves everything, you have a working connector but are pinned to an older destination version until Airbyte ships a permanent fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by