@kapa.ai, i want to get csv files from google cloud storage, but they are in a zip file in the bucket. How to parse the files?

The

source-gcs

connector supports reading CSV files from ZIP archives in GCS buckets. Here's what you need to know: How It Works When

source-gcs

lists files in your bucket, it automatically detects

.zip

files and extracts their contents using the

ZipHelper

class. The extracted files are then parsed according to your configured format (e.g., CSV). [stream_reader.py] Setup 1. Configure your

source-gcs

connector to point to your GCS bucket. 2. Set the Globs pattern to match your ZIP files (e.g.,

**/*.zip

or

folder/myfile.zip

). 3. Set the file format to CSV. The connector will automatically unzip the archive and parse the inner CSV files. Important: File Naming Make sure your ZIP files use a single

.zip

extension (e.g.,

mock_data.zip

), not a compound extension like

mock_data.csv.zip

. There was a known bug where files named with compound extensions (e.g.,

mock_data.csv.zip

) were not recognized as ZIP archives, causing the connector to attempt reading the raw binary as CSV and failing with a UTF-8 encoding error. [GitHub Issue] This bug was fixed in version 0.10.9 by changing the ZIP detection logic from an exact match to an

endswith(".zip")

check. [GCS Changelog] So if you're on version 0.10.9 or later, compound extensions like

.csv.zip

should also work correctly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai if the zip file contains several csv files, what happen?

When a ZIP file contains multiple CSV files, the

source-gcs

connector will extract and yield each file individually. The

ZipHelper.get_gcs_remote_files()

method iterates over all files found in the extracted archive and creates a separate

GCSUploadableRemoteFile

entry for each one. [zip_helper.py]

for dirpath, _, filenames in os.walk(extract_dir):
    for unzipped_file in filenames:
        file_path = os.path.join(dirpath, unzipped_file)
        yield GCSUploadableRemoteFile(...)

A few things worth noting: • Nested directories are supported: Files inside subdirectories within the ZIP are also discovered, thanks to the use of

os.walk()

(recursive traversal). [PR #74781] • Multiple ZIPs work correctly: If you have several ZIP files in your bucket, each ZIP gets its own temporary directory, so files from different ZIPs don't interfere with each other. [PR #74781] • Filename collisions across ZIPs are handled: Because each ZIP is extracted into its own isolated temp directory, two ZIPs containing a file with the same name (e.g., both containing

data.csv

) won't overwrite each other. [PR #74781] These fixes were introduced in a recent patch, so make sure you're on an up-to-date version of the connector to benefit from this behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I get the following error: ... The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)]) The above exception was the direct cause of the following exception: Traceback (most recent call last): File ...

The knowledge sources provided do not contain information about this specific OpenSSL error in the context of

source-gcs

or any other Airbyte connector. This error appears to be related to SSL/TLS certificate or encryption issues (OpenSSL), but I cannot find any relevant documentation or discussion about it in the available knowledge sources to give you a confident, evidence-based answer. I'd recommend: 1. Joining the Airbyte Community Slack to ask other users who may have encountered this. 2. If you're an Airbyte Cloud customer, opening a support ticket with the full stack trace. 3. Searching or filing an issue on the Airbyte GitHub repository. If you can share more context about which connector version you're using and how the error is triggered, that may help narrow down the cause. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I get this error when setuping the source: The glob is extracts/email_logs_.*\.zip Traceback (most recent call last): File "/airbyte/integration_code/source_gcs/stream_reader.py", line 95, in get_matching_files bucket = self.gcs_client.get_bucket(self.config.bucket) ^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_gcs/stream_reader.py", line 78, in gcs_client return self._initialize_gcs_client() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_gcs/stream_reader.py", line 57, in _initialize_gcs_client credentials = self._get_credentials() ^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_gcs/stream_reader.py", line 66, in _get_credentials return service_account.Credentials.from_service_account_info(json.loads(self.config.credentials.service_account)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/google/oauth2/service_account.py", line 243, in from_service_account_info signer = _service_account_info.from_dict( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/google/auth/_service_account_info.py", line 57, in from_dict signer = crypt.RSASigner.from_service_account_info(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/google/auth/crypt/base.py", line 109, in from_service_account_info return cls.from_string( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/google/auth/crypt/_cryptography_rsa.py", line 133, in from_string private_key = serialization.load_pem_private_key( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 494, in _handle_key_loading_error raise ValueError( ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>]) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 84, in _check_list_files file = next(iter(stream.get_files())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_gcs/stream_reader.py", line 114, in get_matching_files self._handle_file_listing_error(exc, prefix, logger) File "/airbyte/integration_code/source_gcs/stream_reader.py", line 118, in _handle_file_listing_error raise ErrorListingFiles( airbyte_cdk.sources.file_based.exceptions.ErrorListingFiles: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance. source=gcs bucket=evs_sfmc prefix=extracts/email_logs_. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 63, in check_availability_and_parsability file = self._check_list_files(stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 90, in _check_list_files raise CheckAvailabilityError(FileBasedSourceError.ERROR_LISTING_FILES, stream=stream.name) from exc airbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance. stream=email_logs_sent

The root cause of your error is clear from the stack trace — it's not a permissions or glob issue. The error originates here:

ValueError: Could not deserialize key data. The data may be in an incorrect format...

This happens in

_get_credentials()

when the connector tries to load your service account JSON: [stream_reader.py]

return service_account.Credentials.from_service_account_info(
    json.loads(self.config.credentials.service_account)
)

The private key inside your service account JSON cannot be parsed by OpenSSL. This is a credential configuration problem, not a glob or file format problem. Common causes: 1. Corrupted or incorrectly copied service account JSON — The private key field (

"private_key"

) may have been mangled (e.g., newlines replaced with spaces, or extra characters added) when pasting it into the Airbyte UI. 2. Unsupported key type — The key uses an algorithm or format not supported by the version of the

cryptography

library in use (e.g., EC keys with explicit parameters instead of named curves). 3. Incomplete JSON — The service account JSON may be truncated or malformed. What to try: 1. Re-download the service account JSON key from the Google Cloud Console and paste it fresh into the

Service Account Information

field — do not manually edit it. 2. Ensure you are pasting the entire JSON content, including the

private_key

field with proper

\n

line breaks. 3. If you generated a new key, make sure it is an RSA key (the default), not an EC key. The glob pattern

extracts/email_logs_.*.zip

is not the issue here — the connector never even reaches the file listing stage because credential initialization fails first. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I've copy/pasted the content of the json file in the Service Account Information field, but I still get the error. Can I edit the content in a text editor to be sure it works? The content is multiline, but the gcs source service account information field is a one line input

You've identified a real issue! The

service_account

field in the GCS connector should support multi-line input, but if the UI is rendering it as a single-line text box, newlines in your private key can get stripped — which would corrupt the key and cause exactly the OpenSSL error you're seeing. According to Airbyte's connector specification, string fields can be annotated with

multiline: true

to display a multi-line text box in the UI, which preserves line breaks. This is specifically important for things like RSA private keys. [Connector Specification Reference] Looking at the GCS source connector code, the

service_account

field is defined as: [config.py]

service_account: str = Field(
    title="Service Account Information.",
    airbyte_secret=True,
    description=(...),
)

It does not have

multiline: true

set, which may explain why the field appears as a single-line input. What you can do: 1. Ensure

\n

is preserved as literal newlines — When pasting the JSON, the

private_key

field must contain actual newline characters (or

\n

escape sequences within the JSON string). If the UI strips them, the key becomes invalid. 2. Validate your JSON before pasting — Open the downloaded

.json

file in a text editor, copy the entire raw content without any modifications, and paste it directly. Do not reformat or minify it. 3. Check the private key format — Inside the JSON, the

private_key

value should look like:"private_key": "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"The

\n

characters here are JSON escape sequences for newlines — this is correct and expected in a single-line JSON string value. If the UI field is truly single-line and stripping newlines, this is a known limitation of fields without

multiline: true

. In that case, I'd recommend raising this on the Airbyte GitHub repository or checking if a newer version of the connector has addressed this. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai In my zip files, I have several csv (Sent.csv, Unsubs.csv, SendJobs.csv...etc.), but when I update the schema, I get only one stream. How to get 1 stream per file?

The knowledge sources I have available don't contain specific information about how

source-gcs

maps extracted files from ZIP archives to streams, or how to configure it to create one stream per CSV file. What I can confirm from the knowledge sources is that the connector does extract and yield each file individually from a ZIP archive [zip_helper.py] — but how those individual files are then mapped to streams in the Airbyte UI is not covered in the available documentation. For this kind of configuration question, I'd recommend: 1. Checking the GCS connector documentation for stream configuration details. 2. Asking in the Airbyte Community Slack where other users may have set up similar multi-file ZIP configurations. 3. If you're an Airbyte Cloud customer, opening a support ticket referencing your use case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Can I use the glob to specify a path within a zip file like "email_logs.zip/sent.csv"?

No, the glob pattern in

source-gcs

cannot be used to specify a path within a ZIP file. The glob pattern operates on the GCS bucket's file/object structure — it matches blob names (i.e., the paths of files stored in GCS), not paths inside ZIP archives. [Path Patterns] For example, a glob like

**/*.zip

or

extracts/email_logs_*.zip

matches the ZIP file objects in your bucket. Once a ZIP is matched, the connector automatically extracts all files inside it — you cannot use the glob to selectively pick only

sent.csv

from within the ZIP. [stream_reader.py] The extraction of files from within the ZIP is handled internally by

ZipHelper

, which yields every file found inside the archive. [zip_helper.py] So if you want only specific CSV files from your ZIPs, there is no supported way to filter them via the glob pattern — all files inside a matched ZIP will be extracted and processed. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.