https://linen.dev logo
Join Slack
Powered by
Hi everyone :wave: I’m trying to use PyAirbyte to...
# help-connector-development
b
Hi everyone 👋 I’m trying to use PyAirbyte to move data from an S3 source to an Iceberg destination. The sync from the source to DuckDB cache works fine, but when loading into Iceberg, I’m getting an error saying that the namespace cannot be null. The issue is that the S3 source doesn’t seem to have any field or config option to populate the namespace, so I’m not sure how to handle this requirement on the destination side. Has anyone faced this before or found a workaround? Any guidance would be really appreciated. Thanks in advance! 🙏
k
Airbyte’s namespace behavior depends on what the source and destination support and how the connection is configured. From the docs: - If the source does not support namespaces, all data is written into the destination’s default namespace. [[Namespaces – examples](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#examples)] - If the destination does not support namespaces, any namespace preference on the connection is ignored. [[Namespaces – examples](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#examples)] - To use namespaces at all, both source and destination connectors must support them and be recent enough. [[Namespaces – requirements](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#requirements)] In your case, S3 as a source likely does not provide a namespace, so Airbyte will fall back to the default namespace configured on the destination. The docs don’t describe any way to “force” a namespace from the S3 source itself. Given that, the only documented levers are: 1. Set a default namespace on the destination (e.g., schema/database/etc., depending on the connector). This is what Airbyte uses when the source has no namespace. [[Namespaces – destination-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] 2. In the connection, use a Custom or Destination-defined namespace option rather than relying on 
${SOURCE_NAMESPACE}
, since S3 doesn’t provide one. [[Namespaces – custom](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] However, the knowledge sources you provided do not mention the Iceberg destination or its specific requirement that “namespace cannot be null”, nor how PyAirbyte exposes these settings. Because of that, I can’t say exactly which PyAirbyte call or Iceberg config you need to change. So, based only on the docs: - Ensure your Iceberg destination has a non-null default namespace configured. - In the connection, avoid relying on 
${SOURCE_NAMESPACE}
and instead choose Destination-defined or a Custom namespace string. If you still get the “namespace cannot be null” error after setting a default namespace on the Iceberg destination, that behavior is not covered in the available documentation, and I don’t have enough information in these sources to provide a more precise workaround.
b
@kapa.ai this is the error i am getting Completed 
source-s3 -> DuckDBCache
sync at 
00:24:28
(
payment_details
stream). Writing PyAirbyte performance stats to file: /tmp/airbyte/logs/airbyte-stats.log streams=[ConfiguredAirbyteStream(stream=AirbyteStream(name='payment_details', json_schema={'type': 'object', 'properties': {'_ab_source_file_last_modified': {'type': 'string'}, '_ab_source_file_url': {'type': 'string'}, 'country_id': {'type': ['null', 'string']}, 'bus_date': {'type': ['null', 'string']}, 'pos_venue_id': {'type': ['null', 'string']}, 'order_id': {'type': ['null', 'string']}, 'src_sys_id': {'type': ['null', 'string']}, 'card_type': {'type': ['null', 'string']}, 'payment_type': {'type': ['null', 'string']}, 'transaction_id': {'type': ['null', 'string']}, 'vendor_id': {'type': ['null', 'string']}, 'transaction_seq_nu': {'type': ['null', 'string']}, 'cash_recycler_tiny_code': {'type': ['null', 'string']}, 'cash_recycler_name': {'type': ['null', 'string']}, 'cash_recycler_serial_number': {'type': ['null', 'string']}, 'payment_amount': {'type': ['null', 'string']}, 'account_id': {'type': ['null', 'string']}, 'payment_amt_rewards_points': {'type': ['null', 'string']}, 'voucher_number': {'type': ['null', 'string']}, 'payment_collector_uid': {'type': ['null', 'string']}, 'active_fl': {'type': ['null', 'string']}}}, supported_sync_modes=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>], source_defined_cursor=True, default_cursor_field=['_ab_source_file_last_modified'], source_defined_primary_key=[['order_id']], namespace='default', is_resumable=True, is_file_based=False), sync_mode=<SyncMode.incremental: 'incremental'>, cursor_field=['_ab_source_file_last_modified'], destination_sync_mode=<DestinationSyncMode.overwrite: 'overwrite'>, primary_key=[['order_id']], generation_id=None, minimum_generation_id=None, sync_id=None, namespace='default')] _ab_source_file_last_modified _ab_source_file_url country_id bus_date ... active_fl _airbyte_raw_id _airbyte_extracted_at _airbyte_meta 0 2025-12-08T132210.914000Z payment_details.csv 94310 2025-07-05 ... 1 0693cb23-c25a-7144-8000-62c19bfb93dd 2025-12-13 002428.141 {} 1 2025-12-08T132210.914000Z payment_details.csv 4059 2025-07-11 ... 1 0693cb23-c25c-767d-8000-bc7413bf0a9e 2025-12-13 002428.141 {} 2 2025-12-08T132210.914000Z payment_details.csv 9847 2025-01-30 ... 1 0693cb23-c25c-7810-8000-766f3af216d5 2025-12-13 002428.141 {} 3 2025-12-08T132210.914000Z payment_details.csv 8775 2025-07-01 ... 1 0693cb23-c25c-7981-8000-f3c5407d2d23 2025-12-13 002428.141 {} 4 2025-12-08T132210.914000Z payment_details.csv 95640 2025-05-24 ... 1 0693cb23-c25c-7ae2-8000-da24b898990b 2025-12-13 002428.141 {} [5 rows x 24 columns] Writing 
destination-s3-data-lake
logs to file: /tmp/airbyte/logs/destination-s3-data-lake/destination-s3-data-lake-log-KCAHGFCKQ.log Connection check succeeded for 
destination-s3-data-lake
. ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Sync Progress: DuckDBCache -> destination-s3-data-lake ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────ERROR: Stream not found: namespace=null, name=payment_details java.lang.IllegalArgumentException: Stream not found: namespace=null, name=payment_details at io.airbyte.cdk.load.command.DestinationCatalog.getStream(DestinationCatalog.kt:35) at io.airbyte.cdk.load.message.DestinationMessageFactory.fromAirbyteMessage(DestinationMessage.kt:670) at io.airbyte.cdk.load.message.ProtocolMessageDeserializer.deserialize(DestinationMessageDeserializer.kt:50) at io.airbyte.cdk.load.task.internal.ReservingDeserializingInputFlow.collect(ReservingDeserializingInputFlow.kt:44) at io.airbyte.cdk.load.task.internal.DefaultInputConsumerTask$execute$3.invokeSuspend(InputConsumerTask.kt:280) at io.airbyte.cdk.load.task.internal.DefaultInputConsumerTask$execute$3.invoke(InputConsumerTask.kt) at io.airbyte.cdk.load.task.internal.DefaultInputConsumerTask$execute$3.invoke(InputConsumerTask.kt) at io.airbyte.cdk.load.util.CoroutineUtilsKt.use(CoroutineUtils.kt:22) at io.airbyte.cdk.load.task.internal.DefaultInputConsumerTask.execute(InputConsumerTask.kt:279) at io.airbyte.cdk.load.task.DestinationTaskLauncher$WrappedTask.execute(DestinationTaskLauncher.kt:163) at io.airbyte.cdk.load.task.TaskScopeProvider$launch$job$1.invokeSuspend(TaskScopeProvider.kt:35) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101) at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589)
k
A message from kapa.ai
The error you’re seeing: > 
Stream not found: namespace=null, name=payment_details
matches a known issue pattern with the S3 Data Lake destination when the incoming records have a 
null
namespace but the destination catalog expects a non‑null namespace. [[S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] From the logs you shared, the stream is configured in DuckDB with:
Copy code
text namespace='default' ... streams=[..., namespace='default')]
but by the time it reaches 
destination-s3-data-lake
, the destination is looking for:
Copy code
text namespace=null, name=payment_details
So there’s a mismatch between: - the namespace in the catalog the destination was started with, and - the namespace on the records it is receiving. The public docs only describe the general namespace behavior: - If a source/stream has no namespace, Airbyte falls back to the destination’s default namespace when using Destination-defined or Source-defined modes. [[Namespaces – overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] - Namespaces are configured per connection as: - Destination-defined (use destination default), - Custom (single user‑defined namespace, optionally with 
${SOURCE_NAMESPACE}
), - Source-defined (mirror source namespace when present, otherwise fall back to destination default). [[Namespaces – options](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] However, the knowledge sources do not show: - how PyAirbyte exposes the connection‑level namespace setting, or - any Iceberg‑specific or S3‑data‑lake‑specific configuration knobs for this error. Because of that, I can’t give you an exact PyAirbyte code snippet or a guaranteed fix, only what follows from the docs and the linked issue: 1. Ensure the connection uses a non‑null namespace consistently In the connection that goes from 
DuckDBCache
to 
destination-s3-data-lake
, set the namespace mode to either: - Destination-defined (so everything goes to the destination’s default namespace), or - Custom with a literal value like 
default
(do not use 
${SOURCE_NAMESPACE}
here, since that can be null for S3‑based streams). [[Namespaces – custom & destination-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] The goal is that both the catalog and the records use the same non‑null namespace (e.g. 
default
) so the destination can find the stream. 2. Avoid relying on 
${SOURCE_NAMESPACE}
for S3 S3 sources generally don’t provide a meaningful namespace; when the source namespace is unknown, behavior falls back to the destination default. [[Namespaces – source-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] Using 
${SOURCE_NAMESPACE}
in a custom format can therefore result in 
null
namespaces, which is exactly what triggers the error in the S3 Data Lake destination. [[S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] 3. If the error persists The GitHub issue suggests there have been platform‑level bugs around connection‑level namespace overrides that affected S3 Data Lake and were fixed in newer versions. [[S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] The knowledge sources don’t specify which exact versions PyAirbyte or the S3 Data Lake destination must be on, so I can’t say more than: if you’re on an older Airbyte platform/connector version, upgrading may be necessary. Because the provided docs don’t include PyAirbyte‑specific examples or Iceberg/S3‑data‑lake configuration details, I can’t be more precise than recommending: - configure a fixed, non‑null namespace (e.g. 
default
) for the DuckDB → S3 Data Lake connection, and - avoid 
${SOURCE_NAMESPACE}
for this flow.
b
@kapa.ai How about this?? ERROR: Failed to initialize connector operation io.airbyte.cdk.ConfigErrorException: Failed to initialize connector operation at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:31) at picocli.CommandLine.executeUserObject(CommandLine.java:2030) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) at picocli.CommandLine.execute(CommandLine.java:2174) at io.airbyte.cdk.AirbyteDestinationRunner$Companion.run(AirbyteConnectorRunner.kt:288) at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeDestination.main(S3DataLakeDestination.kt:15) Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.cdk.load.command.DefaultDestinationCatalogFactory] Message: class io.airbyte.protocol.models.v0.ConfiguredAirbyteCatalog schema violation: Validation error(s) : streams.0.stream.namespace: Null value is not allowed. (code: 1021) From: streams.0.<items>.<#/definitions/ConfiguredAirbyteStream>.stream.<#/definitions/AirbyteStream>.namespace.<nullable>
k
A message from kapa.ai
The new error shows the root cause much more clearly: > 
ConfiguredAirbyteCatalog schema violation: ... streams.0.stream.namespace: Null value is not allowed. (code: 1021)
This is coming from the destination’s catalog validation: the S3 Data Lake destination (and other bulk destinations) now require that each stream’s 
namespace
in the ConfiguredAirbyteCatalog is non‑null. [[Airbyte protocol](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-protocol#messages)] This aligns with the platform change that **stream identifiers (name + namespace) must be unique and fully specified**; you can no longer rely on “no namespace” and have multiple connections write into the same destination stream. [[Release 1.6 – unique streams](https://docs.airbyte.com/release_notes/v-1.6#unique-stream-prefixes-or-namespaces-are-now-mandatory)] From the namespace docs: - Airbyte supports three connection‑level namespace modes: Custom, Destination-defined, and Source-defined. [[Namespaces – options](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] - When the source doesn’t provide a namespace (typical for S3), behavior should fall back to the destination’s default namespace if you use Destination-defined or Source-defined. [[Namespaces – source-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] Your error means that, for at least one stream (
payment_details
), the catalog being passed into 
destination-s3-data-lake
has:
Copy code
json "namespace": null
and the destination rejects that. Based strictly on the docs and the related S3 Data Lake issue you’re effectively hitting [[S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)], the way to avoid this is: 1. Ensure the connection defines a non‑null namespace for the stream For the DuckDB → S3 Data Lake (or DuckDB → Iceberg) connection, configure the namespace mode so that the resulting 
ConfiguredAirbyteCatalog
has a concrete namespace string, e.g. `default`: - Use Custom namespace and set it to a literal like 
default
(do not use 
${SOURCE_NAMESPACE}
here, since S3 doesn’t provide one and it can become null). [[Namespaces – custom](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] - Or use Destination-defined, making sure the destination itself has a default namespace configured (schema/database/etc., depending on connector). [[Namespaces – destination-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] The key is: the catalog that reaches the destination must have 
namespace: "something"
for every stream, not 
null
. 2. Avoid null namespaces from S3 Since S3 as a source doesn’t provide a namespace, don’t rely on 
${SOURCE_NAMESPACE}
in any namespace template for this flow; that’s how you end up with 
null
in the catalog. [[Namespaces – source-defined](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] 3. If you still see the error The GitHub issue for S3 Data Lake shows similar stack traces (
Stream not found: namespace=null, name=...
) and mentions that some of this behavior was tied to older platform versions and bugs around connection‑level namespace overrides. [[S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] The provided knowledge doesn’t specify exact PyAirbyte or connector versions required, so I can only say: if you’re on an older Airbyte platform/connector, upgrading may be necessary. Because the knowledge sources don’t include PyAirbyte‑specific code examples, I can’t show the exact Python call, but conceptually you need to configure the DuckDB → S3 Data Lake / Iceberg connection so that its ConfiguredAirbyteStream.stream.namespace is a non‑null string (e.g. 
default
) before running the sync.
b
@kapa.ai I tried all the approaches but couldn't find the solution can you trying seeing the get_source and get_destination for these two
k
A message from kapa.ai
I’m sorry, but the knowledge sources you’ve provided do not contain any information about: - PyAirbyte’s 
get_source
/ 
get_destination
APIs, or - concrete code examples for configuring S3 → DuckDB → S3 Data Lake / Iceberg with PyAirbyte. The docs only describe the concept of namespaces and how they’re handled at the platform/connection level (Custom, Destination-defined, Source-defined), plus a related S3 Data Lake issue where 
namespace=null
causes “Stream not found” and catalog validation errors.[[Namespaces core](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace); [S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] They do not show: - how 
get_source()
or 
get_destination()
should be called in PyAirbyte, - how to pass a namespace override via those functions, or - any Iceberg‑specific configuration. Because of that, I can’t reliably “inspect” or reconstruct the correct 
get_source
/ 
get_destination
usage for your exact case without guessing, which I must avoid. What the docs do make clear is: - Every 
ConfiguredAirbyteStream.stream.namespace
must be non‑null for destinations like S3 Data Lake; 
null
is rejected by schema validation.[[Namespaces core](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace); [S3 data lake issue](https://github.com/airbytehq/airbyte/issues/58133)] - When the source doesn’t provide a namespace (typical for S3), you must rely on a Custom or Destination-defined namespace so the catalog has a concrete value (e.g. 
"default"
).[[Namespaces core](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces#what-is-a-namespace)] But how to express that specifically in PyAirbyte’s 
get_source
/ 
get_destination
calls is not documented in the materials I have. To move forward without guessing, you’d need either: - PyAirbyte’s own docs or examples for 
get_source
/ 
get_destination
, or - the exact Python snippet you’re using, so someone with access to the full PyAirbyte API can advise how to inject a non‑null namespace into the connection/catalog.
4 Views
Open in Slack
PreviousNext
Powered by