@kapa.ai I'm trying to update my S3 source and I get this: "Failed to save S3 due to the following error: Error creating stream config object." The test logs show no error.

The knowledge sources do not contain enough information to directly explain the "Error creating stream config object" error you're encountering when saving an S3 source in Airbyte. However, based on what the knowledge sources do cover, here are a few things worth checking: Possible causes to investigate 1. Missing

format

field: If you're on source-s3 v4.0.4+, the

streams.*.format

field is now required. If you're configuring via the API without providing it, you may hit config validation errors. [S3 Migration Guide] 2. Invalid endpoint format: If you're specifying a custom endpoint (e.g.,

s3.us-east-1.amazonaws.com

without the

https://

prefix), this can cause failures. On Airbyte Cloud, the endpoint must be secure and not start with

http://

. [S3 Migration Guide] 3. Legacy config migration: If you upgraded to source-s3 v4.0.0+, the config format changed significantly (e.g.,

streams

array is now required). If you modified the config before running a sync post-upgrade, you may need to manually update the fields to match the new format. [S3 Migration Guide] Recommended next steps • Double-check that your stream configuration includes a valid

format

object (e.g.,

{"filetype": "csv"}

). • Ensure the

Bucket

field contains only the bucket name, not a path (paths should go in

Globs

). • If you're using the API, verify your config matches the v4 schema with a

streams

array. If the issue persists, I'd recommend reaching out to Airbyte support with the full connector version and your configuration (with secrets redacted), as the knowledge sources don't contain a direct match for this specific error message. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The issue seems to be caused by the use of a double pipe (||) as a delimiter

That's a plausible root cause. Based on the knowledge sources, the S3 source connector's CSV format configuration specifies that the delimiter must be exactly 1 character long: [CSV spec]

delimiter: str = Field(
    default=",",
    min_length=1,
    description="The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\\t'.",
    order=0,
)

The description explicitly states: "This may only be a 1-character string." A double pipe (

||

) is two characters, which would fail this validation constraint (

min_length=1

combined with the single-character requirement), and would likely cause the stream config object creation to fail. To fix this, you would need to use a single-character delimiter. If your data uses

||

as a separator, unfortunately the S3 CSV connector does not support multi-character delimiters. You would need to either: • Pre-process your files to use a single-character delimiter (e.g.,

|

), or • Use a different file format if possible. The knowledge sources do not provide a workaround for multi-character delimiters within the connector itself. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai well that's fixed, but now when I try to refresh my schema I just get "An unexpected error occurred. Please report this if the issue persists. (HTTP 500)". There's only one file in my bucket and it's a zip file with multiple csvs

The HTTP 500 error during schema refresh with a ZIP file containing multiple CSVs is a known area of complexity for the S3 connector. The knowledge sources don't directly explain what's causing your specific HTTP 500 error, but here are some relevant things to be aware of: How S3 handles ZIP files The S3 source connector does support reading files inside ZIP archives. It detects ZIP files by checking if the filename ends with

.zip

, then uses a

ZipFileHandler

to parse the archive's central directory and list the inner files. [stream_reader.py] Potential issue: Invalid or corrupt ZIP There is a known bug where if a

.zip

file is corrupt or not a valid ZIP archive, the connector can crash with a

TypeError

(which could surface as an HTTP 500). A fix was implemented to handle this more gracefully by skipping invalid ZIP files with a warning. [fix PR] To check: Verify that your ZIP file is a valid, non-corrupt archive by opening it locally. Things to verify 1. ZIP file validity: Confirm the ZIP file can be extracted locally without errors. 2. File naming: Ensure the file ends with

.zip

(not a compound extension like

.csv.zip

, which has a separate known bug in

source-gcs

but not

source-s3

). [GCS issue] 3. Inner file format: Make sure the CSVs inside the ZIP are valid and match the format configuration (including your delimiter). 4. Connector version: Make sure you're on a recent version of the S3 connector that includes the invalid ZIP handling fix. Unfortunately, the knowledge sources don't contain enough detail to pinpoint the exact cause of your HTTP 500 during schema refresh. If the issue persists after verifying the above, I'd recommend checking the full server-side logs or opening a support ticket with Airbyte. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The CSVs didn't match the config. Now it works, but when I try to sync I get this error : "Init container error encountered while processing workload for id: 3c6273fa-8853-454e-9a98-7674b763508c_80095341_0_sync. Encountered exception of type: class com.fasterxml.jackson.module.kotlin.KotlinInvalidNullException. Exception message: Instantiation of [simple type, class io.airbyte.config.Field] value failed for JSON property name due to missing (therefore NULL) value for creator parameter name which is a non-nullable type at [Source: REDACTED (

StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION

disabled); line: 1, column: 1209] (through reference chain: io.airbyte.config.ConfiguredAirbyteCatalog["streams"]->java.util.ArrayList[0]->java.util.ArrayList[0]->io.airbyte.config.Field["name"])."

Sorry @Swan Sauvegrain, I'm currently experiencing some technical difficulties. Please try again in a few minutes. If this issue persists, please reach out to the administrators of this Slack workspace.

@kapa.ai The CSVs didn't match the config. Now it works, but when I try to sync I get this error : "Init container error encountered while processing workload for id: 3c6273fa-8853-454e-9a98-7674b763508c_80095341_0_sync. Encountered exception of type: class com.fasterxml.jackson.module.kotlin.KotlinInvalidNullException. Exception message: Instantiation of [simple type, class io.airbyte.config.Field] value failed for JSON property name due to missing (therefore NULL) value for creator parameter name which is a non-nullable type at [Source: REDACTED (

StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION

disabled); line: 1, column: 1209] (through reference chain: io.airbyte.config.ConfiguredAirbyteCatalog["streams"]->java.util.ArrayList[0]->java.util.ArrayList[0]->io.airbyte.config.Field["name"])."

Hello! I'm here to help. Do you have any follow-up questions about your S3 source setup, or is there something else Airbyte-related I can assist you with? 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The CSVs didn't match the config. Now it works, but when I try to sync I get this error : "Init container error encountered while processing workload for id: 3c6273fa-8853-454e-9a98-7674b763508c_80095341_0_sync. Encountered exception of type: class com.fasterxml.jackson.module.kotlin.KotlinInvalidNullException. Exception message: Instantiation of [simple type, class io.airbyte.config.Field] value failed for JSON property name due to missing (therefore NULL) value for creator parameter name which is a non-nullable type at [Source: REDACTED (

StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION

disabled); line: 1, column: 1209] (through reference chain: io.airbyte.config.ConfiguredAirbyteCatalog["streams"]->java.util.ArrayList[0]->java.util.ArrayList[0]->io.airbyte.config.Field["name"])."

The knowledge sources do not contain enough information to directly explain this specific error (

KotlinInvalidNullException

for a null

name

field in

io.airbyte.config.Field

). However, the error message is clear about what's happening: during sync initialization, Airbyte is trying to deserialize your

ConfiguredAirbyteCatalog

and one of the

Field

objects in your stream configuration has a

null

value for its

name

property, which is defined as non-nullable in the Airbyte config model. This is conceptually similar to a known class of issues where null values in catalog/stream configuration fields cause NPEs or Kotlin null-safety exceptions during initialization. [NPE fix PR] What to try 1. Refresh your schema again and re-save the connection. It's possible the catalog stored for your connection has a malformed field entry (e.g., a field with a blank or missing name, possibly introduced during the earlier schema refresh issues you experienced). 2. Reset the connection: Delete and re-create the connection from scratch. This forces Airbyte to re-discover the catalog cleanly. 3. Check your CSV headers: Since the error points to a

Field

with a null

name

, it's worth verifying that none of your CSV files inside the ZIP have empty or missing column headers — these could be getting picked up as unnamed fields during schema inference. 4. Check your Airbyte version: If you're on an older platform version, there may be known bugs around null field handling that have since been patched. Unfortunately, the knowledge sources don't contain a direct fix or workaround for this exact error. If the above steps don't resolve it, I'd recommend opening a support ticket or posting in the Airbyte Community Slack with your connector version and platform version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai there are line breaks in some of my CSVs' text fields. This breaks the schema refresh. Is there any way to get around this?

Yes! The S3 source connector has a built-in setting specifically for this. The "Allow newlines in values" (

newlines_in_values

) option controls whether newline characters are permitted inside CSV field values. [CSV spec] In your S3 source CSV format settings, set: • Allow Newlines in Values →

true

> ⚠️ Note: The documentation warns that enabling this option may affect performance. [S3 file format settings] This is the standard supported workaround for line breaks inside CSV text fields. After enabling it, try refreshing your schema again. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai this isn't in the doc or in the form anymore. Is it a version issue?

Based on the knowledge sources, the "Allow Newlines in Values" (

newlines_in_values

) field is defined in the legacy CSV spec file, which is part of the older (pre-v4) connector code. [CSV spec] When the connector was migrated to v4.0.0 (using the file-based CDK), the configuration UI was significantly revamped. [S3 Migration Guide] The new v4 CSV format settings documented in the official docs list these options: [S3 file format settings] • Header Definition, Delimiter, Double Quote, Encoding, Escape Character, False Values, Null Values, Quote Character, Skip Rows After/Before Header, Strings Can Be Null, True Values "Allow Newlines in Values" is not listed among the v4 format settings, which suggests it may have been dropped from the UI in the v4 migration. The knowledge sources do not contain enough information to confirm definitively whether this option is still available under the hood in v4 or has been fully removed. I'd recommend: 1. Checking the Airbyte S3 connector changelog for any mention of this option being deprecated. 2. Posting in the Airbyte Community to ask if there's an equivalent workaround in v4. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.