https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I'm trying to do a GET request on a...
# replication-ask-ai
h
@kapa.ai I'm trying to do a GET request on a URL which returns a gzip compression which has a JSONL file within it. When trying to fetch the data, I get this error constantly:
Copy code
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py", line 103, in read                                                                                                                                                                                                                                                         │
│     for stream_data in self._retriever.read_records(                                                                                                                                                                                                                                                                                                                                                                                     │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 525, in read_records                                                                                                                                                                                                                                                                    │
│     for stream_data in self._read_pages(record_generator, self.state, _slice):                                                                                                                                                                                                                                                                                                                                                           │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 396, in _read_pages                                                                                                                                                                                                                                                                     │
│     for current_record in records_generator_fn(response):                                                                                                                                                                                                                                                                                                                                                                                │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 595, in _parse_records                                                                                                                                                                                                                                                                  │
│     yield from self._parse_response(                                                                                                                                                                                                                                                                                                                                                                                                     │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 281, in _parse_response                                                                                                                                                                                                                                                                 │
│     yield from self.record_selector.select_records(                                                                                                                                                                                                                                                                                                                                                                                      │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 90, in select_records                                                                                                                                                                                                                                                                    │
│     yield from self.filter_and_transform(                                                                                                                                                                                                                                                                                                                                                                                                │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 121, in filter_and_transform                                                                                                                                                                                                                                                             │
│     for data in normalized_data:                                                                                                                                                                                                                                                                                                                                                                                                         │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 137, in _normalize_by_schema                                                                                                                                                                                                                                                             │
│     yield from records                                                                                                                                                                                                                                                                                                                                                                                                                   │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 162, in _transform                                                                                                                                                                                                                                                                       │
│     for record in records:                                                                                                                                                                                                                                                                                                                                                                                                               │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 154, in _filter                                                                                                                                                                                                                                                                          │
│     yield from records                                                                                                                                                                                                                                                                                                                                                                                                                   │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/extractors/dpath_extractor.py", line 72, in extract_records                                                                                                                                                                                                                                                                   │
│     for body in self.decoder.decode(response):                                                                                                                                                                                                                                                                                                                                                                                           │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/decoders/composite_raw_decoder.py", line 199, in decode                                                                                                                                                                                                                                                                       │
│     yield from parser.parse(data=io.BytesIO(response.content))                                                                                                                                                                                                                                                                                                                                                                           │
│   File "/home/airbyte/.pyenv/versions/3.10.18/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/decoders/composite_raw_decoder.py", line 96, in parse                                                                                                                                                                                                                                                                         │
│     yield json.loads(line.decode(encoding=self.encoding or "utf-8"))                                                                                                                                                                                                                                                                                                                                                                     │
│ UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte                                                                                                                                                                                                                                                                                                                                               │
How can I fix this?
k
Fix: UnicodeDecodeError on gzip + JSONL Response The error occurs because the CDK is attempting to parse the raw (still-compressed) gzip response directly as JSONL. The 
0x8b
byte is the second byte of the gzip magic number (
0x1f 0x8b
), confirming the response has not been decompressed before JSONL parsing. Solution You need to nest a 
JsonlDecoder
inside a 
GzipDecoder
in your stream's YAML configuration. The 
GzipDecoder
will decompress the response first, then pass the result to the inner decoder. [ZIP and gzip]
Copy code
retriever:
  type: SimpleRetriever
  record_selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path: []
  decoder:
    type: GzipDecoder
    decoder:
      type: JsonlDecoder
  requester:
    ...
The 
GzipDecoder
supports nesting the following inner decoders: 
CsvDecoder
, 
GzipDecoder
, 
JsonDecoder
, 
JsonItemsDecoder
, and 
JsonlDecoder
. [GzipDecoder reference] This pattern is also used in the Amplitude connector, which handles ZIP → GZip → JSONL nesting:
Copy code
decoder:
  type: ZipfileDecoder
  decoder:
    type: GzipDecoder
    decoder:
      type: JsonlDecoder
For your case (single gzip layer containing JSONL), only the 
GzipDecoder
wrapping 
JsonlDecoder
is needed. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
h
@kapa.ai That's how my decoder looks like:
Copy code
- type: DeclarativeStream
retriever:
  type: SimpleRetriever
  record_selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path: []
  requester:
    type: HttpRequester
    http_method: GET
    url: >-
      https://<url>
    request_headers: {}
  decoder:
    type: GzipDecoder
    decoder:
      type: JsonlDecoder
name: test
k
Your decoder configuration (
GzipDecoder
wrapping 
JsonlDecoder
) is structurally correct. However, the error trace confirms that the raw gzip bytes (
0x8b
) are reaching the 
JsonlDecoder
directly, meaning the 
GzipDecoder
decompression step is being skipped. There are two likely causes: 1. YAML Indentation Issue Your pasted YAML shows 
retriever:
and 
name:
at the same indentation level as 
type: DeclarativeStream
, which would be invalid. Make sure the full stream is properly indented:
Copy code
streams:
  - type: DeclarativeStream
    name: test
    retriever:
      type: SimpleRetriever
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      requester:
        type: HttpRequester
        http_method: GET
        url_base: "https://<url>"
        request_headers: {}
      decoder:
        type: GzipDecoder
        decoder:
          type: JsonlDecoder
2. Content-Encoding Header Conflict If the server returns a 
Content-Encoding: gzip
header, the 
requests
library may auto-decompress the body, causing the CDK to mishandle the 
GzipDecoder
pipeline. A known workaround (referenced in a related GitHub issue) is to wrap the decoder in a second 
GzipDecoder
layer:
Copy code
decoder:
  type: GzipDecoder
  decoder:
    type: GzipDecoder
    decoder:
      type: JsonlDecoder
[GitHub issue #56988] I'd recommend verifying your YAML indentation first, as that is the most common cause of decoders not being applied. If the problem persists, try the double 
GzipDecoder
workaround. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
h
@kapa.ai The indentation is fine. The server does NOT return any Content-Encoding header, but just
Copy code
"Content-Type": "application/octet-stream",
k
This is the same root cause as a known CDK issue: the 
GzipDecoder
only activates when it detects a recognized gzip Content-Type (like 
application/gzip
or 
application/x-gzip
). With 
application/octet-stream
, it skips decompression and passes the raw bytes straight to the inner 
JsonlDecoder
. [GitHub issue #56988] The confirmed workaround is to wrap your decoder in a second 
GzipDecoder
, which causes the CDK to use its default gzip content type and decompress correctly:
Copy code
decoder:   type: GzipDecoder   decoder:     type: GzipDecoder     decoder:       type: JsonlDecoder
This was tested on real data and confirmed working for the same scenario (gzip-compressed response with no 
Content-Encoding
header and a non-standard Content-Type). [GitHub issue #56988] > Note: This is a workaround, not a permanent fix. The proper fix would be on the CDK side to default the content type to 
gzip
for 
GzipDecoder
regardless of the response headers. If this workaround doesn't resolve your issue, it may be worth filing or referencing a CDK bug report. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by