https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Does Airbyte Helm 2.1.0 have a new ...
# replication-ask-ai
t
@kapa.ai Does Airbyte Helm 2.1.0 have a new restriction of only 100 connections for OSS deployments?
k
No, Airbyte Helm chart V2 (including version 2.1.0) does not introduce a restriction of 100 connections for OSS/Community deployments. The knowledge sources do not mention any such limitation. The number "100" that appears in the documentation refers to a typical usage reference point, not a hard limit: > "As a reference point, the typical Airbyte user has 5–20 connectors and 10–100 connections configured... resulting in at most 100 concurrent jobs." [Scaling Airbyte] This is simply describing average usage patterns, not imposing a cap. The same documentation notes that scaling issues are generally only encountered when running over a thousand connections. The actual breaking changes introduced in Airbyte 2.1.0 are: • Helm chart V1 is no longer supported — you must migrate to Helm chart V2. [Airbyte 2.1] • Authentication is now enabled by default for new deployments. • Management endpoints moved to port 8085. • The 
AIRBYTE_ROLE
environment variable has been removed. None of these changes impose a connection count restriction on OSS deployments. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
🙏 1
a
2026-05-07 210932 source ERROR In app events reports: reports are limited to 31 days per query. Select a narrower date range and submit your query again. 2026-05-07 210932 source ERROR Encountered an exception while reading stream retargeting_in_app_events Traceback (most recent call last): File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py”, line 130, in read stream_is_available, reason = stream_instance.check_availability(logger, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/core.py”, line 269, in check_availability return self.availability_strategy.check_availability(self, logger, source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py”, line 56, in check_availability is_available, reason = self.handle_http_error(stream, logger, source, error) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py”, line 85, in handle_http_error raise error File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py”, line 50, in check_availability get_first_record_for_slice(stream, stream_slice) File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py”, line 40, in get_first_record_for_slice return next(records_for_slice) ^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 482, in read_records yield from self._read_pages( File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 498, in _read_pages request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 524, in _fetch_next_page response = self._send_request(request, request_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 422, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/backoff/_sync.py”, line 105, in retry ret = target(*args, **kwargs) @kapa.ai get this this in appsflyer
k
The error you're seeing is related to the 
retargeting_in_app_events
 stream in the Airbyte AppsFlyer source connector. The message "reports are limited to 31 days per query" is actually the expected behavior enforced by the AppsFlyer API itself. Looking at the connector source code, the 
InAppEvents
stream (which 
RetargetingInAppEvents
inherits from) explicitly sets a 31-day interval limit: [source-appsflyer source.py]
Copy code
class InAppEvents(RawDataMixin, IncrementalAppsflyerStream):
    intervals = 31
    cursor_field = "event_time"
This 
intervals = 31
setting means the connector chunks date ranges into 31-day slices when making API requests. The 
RetargetingInAppEvents
class inherits this behavior:
Copy code
class RetargetingInAppEvents(InAppEvents):
    def path(...) -> str:
        return f"raw-data/export/app/{self.app_id}/in-app-events-retarget/v5"
The error you're encountering suggests the connector is sending a request with a date range wider than 31 days to the AppsFlyer API. This could happen if: 1. The stream state or stream slice is not being chunked correctly. 2. You are running an older version of the connector where this chunking may not work as expected. What you can try: • Ensure you are using a recent version of the 
source-appsflyer
connector, as there have been several bug fixes and updates. • Set a 
start_date
that is relatively recent (within the last 31 days) to avoid large initial syncs that might trigger this issue. • Note that the connector enforces a 90-day maximum lookback for raw data streams, so start dates older than 90 days will be automatically adjusted. [source-appsflyer source.py] The 31-day chunking is a hard constraint imposed by the AppsFlyer API for in-app events reports, and the connector is designed to respect it by splitting requests accordingly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
all stream are in cremental append y i am geting this, @kapa.ai
k
Based on the previous conversation, you're asking why you're still getting the "reports are limited to 31 days per query" error even though your streams are set to incremental append mode. The issue is that the 31-day limit error is not caused by the sync mode (incremental vs. full refresh). It's caused by the date range being sent in the API request being wider than 31 days. Here's what happens in incremental mode with the AppsFlyer connector: 1. On the first sync (initial backfill), the connector calculates stream slices from your 
start_date
to 
end_date
, chunked into 31-day intervals. [source-appsflyer source.py] 2. The error is occurring during the availability check (before actual data reading), where the CDK tries to fetch the first record using a stream slice. If that initial slice spans more than 31 days, the API rejects it. 3. The connector also enforces a 90-day cap on 
start_date
— so even if you set an older start date, it gets reset to 90 days ago. [source-appsflyer source.py] This means the initial backfill window could be up to 90 days, which needs to be chunked into ~3 slices of 31 days each. The error in your traceback is happening at the 
check_availability
 step, not during actual data reading — meaning the connector is failing before it even starts syncing. This is a known issue with how the CDK's availability check interacts with the stream slicing logic. Recommended actions: • Try setting your 
start_date
to a date within the last 31 days to reduce the initial sync window. • Check if you're on the latest version of the 
source-appsflyer
connector, as there have been open issues and PRs related to this connector's date range handling. [GitHub Issue #60994] The knowledge sources do not contain enough detail to confirm whether a specific connector version fully resolves this availability check issue. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
but all pipline was working now it ma getting this @kapa.ai
k
Since your pipeline was previously working and is now suddenly failing, this points to something that changed recently rather than a configuration issue. Based on the context of your AppsFlyer connector error, here are the most likely causes: 1. Your sync state has drifted / gap in syncs If your connection was paused or failed for a period, the next incremental sync would try to catch up from the last cursor value. If that gap is larger than 31 days, the connector may attempt a slice wider than what the AppsFlyer API allows for in-app events streams. What to try: Reset the stream state for 
retargeting_in_app_events
to a recent date (within the last 31 days) via the Airbyte UI under Connection → Status → Reset stream state. 2. Connector version change The knowledge sources note that the 
source-appsflyer
connector has had several updates and known issues around date range handling. [GitHub Issue #60994] If the connector was auto-updated, behavior may have changed. What to try: Check your connector version in the Airbyte UI and compare it to what was running before. 3. The 90-day cap being applied unexpectedly The connector globally caps 
start_date
to 90 days ago for all streams, including raw data streams. [source-appsflyer source.py] If something reset your stream state, the connector may be trying to backfill 90 days of data, which then needs to be chunked into 31-day slices — and the availability check may be failing before that chunking kicks in. Immediate workaround: Set your 
start_date
in the connector configuration to a date within the last 31 days, trigger a sync, and then adjust it back once the state is re-established. The knowledge sources don't contain enough detail to pinpoint the exact root cause of a previously-working pipeline suddenly breaking, so I'd recommend checking the Airbyte connector logs for any version changes or state resets around the time the issue started. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
2026-05-07 124227 source INFO Start date over 90 days, using start_date: 2026-02-05T230000-08:00 2026-05-07 124227 source INFO Using start_date: 2026-02-05T230000-08:00, end_date: 2026-05-07T001227.208148-07:00 2026-05-07 124227 source ERROR In app events reports: reports are limited to 31 days per query. Select a narrower date range and submit your query again. 2026-05-07 124227 source ERROR Encountered an exception while reading stream organic_in_app_events Traceback (most recent call last): File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py”, line 130, in read stream_is_available, reason = stream_instance.check_availability(logger, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/core.py”, line 269, in check_availability return self.availability_strategy.check_availability(self, logger, source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py”, line 56, in check_availability is_available, reason = self.handle_http_error(stream, logger, source, error) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @kapa.ai
k
The logs confirm exactly what was discussed earlier. The connector is applying the 90-day cap and setting 
start_date
to 2026-02-05, which is approximately 90 days before the sync date. The 
organic_in_app_events
stream then fails during the availability check because that check is sending a request spanning more than 31 days to the AppsFlyer API. The root cause is clear from the connector code: the 
streams()
method globally caps 
start_date
to 90 days ago for all streams, but the availability check doesn't respect the 31-day chunking that the 
IncrementalAppsflyerStream
applies during actual data reading. [source-appsflyer source.py]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90)
start_date = parse_date(config.get("start_date") or pendulum.today(timezone), timezone)
config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
This 90-day window (~3 months) is wider than the 31-day API limit for in-app events, and the availability check fires before the chunking logic splits it into valid slices. Workarounds 1. Set 
start_date
to within the last 31 days in your connector configuration (e.g., 30 days ago). This prevents the 90-day cap from being triggered and keeps the initial request within the API's allowed range. 2. Reset the stream state for 
organic_in_app_events
(and 
retargeting_in_app_events
) to a recent date via Airbyte UI → Connection → Status → Reset stream state, then set 
start_date
to a recent date before re-syncing. 3. Check for a connector update — there was an open issue and a PR (#61520) specifically addressing this 90-day cap being incorrectly applied to all streams. [GitHub Issue #60994] However, that issue was closed as stale, so it's unclear if the fix was merged. The underlying problem is a known bug in the connector where the global 90-day cap conflicts with the 31-day per-request limit for raw data streams during the availability check phase. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
last sucesss was wokring with same date. Now that i am getting. @kapa.ai
k
This is a key detail — if the same 
start_date
configuration was working before and is now failing, the issue is almost certainly not your configuration. Something changed on either the connector or the state side. Based on the knowledge sources, here are the most likely explanations: 1. Stream state was reset or corrupted If the stream state for 
organic_in_app_events
or 
retargeting_in_app_events
was reset (manually or due to a failed sync), the connector falls back to the configured 
start_date
. If that 
start_date
is older than 90 days, the connector resets it to 90 days ago — which then triggers the 31-day API limit error during the availability check. [source-appsflyer source.py] 2. Connector was auto-updated The 
source-appsflyer
connector has had frequent dependency updates (nearly every week). [AppsFlyer Changelog] An auto-update could have changed behavior. What to do right now 1. Check your stream state in the Airbyte UI under Connection → Status. If the cursor for 
organic_in_app_events
or 
retargeting_in_app_events
shows an old date or is missing, that's the cause. 2. Manually set the stream state to a date within the last 31 days via Connection → Status → Edit state, then re-run the sync. 3. Set 
start_date
 in the connector config to a date within the last 31 days as a temporary fix to get syncs running again. The knowledge sources do not contain enough information to confirm exactly what changed between your last successful sync and now, but a state reset combined with the 90-day cap is the most consistent explanation given the logs you've shared. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
*Akshayah C.*Getting this error 📄
Error in airbyte: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field group is expected to be one of these: NULL, STRING. If this is a complex type, check if offending field (path: group) adheres to schema: {id=50bafb55-a0e7-41b0-8d7c-7b0f4ca9446d} With Airbyte, the issue is almost always in how the destination connector handles Avro/Parquet schema normalization. Here’s the specific breakdown: Why S3 worked but GCS doesn’t: Airbyte’s S3 destination in Parquet mode uses JSON → Parquet direct conversion (via Jackson), which is schema-lenient. The GCS destination’s Parquet writer goes through JSON → Avro → Parquet, which is strict about union types. So the 
group
field being an object 
{"id": "50bafb55..."}
slips through on S3 but explodes on GCS. @kapa.ai i am using version 1.6 and does it get fixed on airbyte 2.0
k
The error you're seeing is a well-known issue with Airbyte's JSON → Avro → Parquet conversion pipeline used by the GCS destination. The 
group
field is arriving as a JSON object 
{"id": "50bafb55-..."}
but the Avro schema expects it to be either 
NULL
or 
STRING
. [GCS Parquet Docs] This is consistent with multiple reported cases across different sources (MongoDB, MSSQL, Postgres, Pinterest) where a field's actual data type doesn't match the Avro schema's expected union type. [Forum: Failed to convert JSON to Avro] Regarding your specific questions: On the S3 vs GCS difference you described: The knowledge sources confirm that the GCS destination's Parquet output goes through JSON → Avro → Parquet, which is strict about union types. [GCS Parquet Docs] However, the knowledge sources do not contain enough information to confirm or deny whether S3 uses a different (more lenient) conversion path as you described. On whether it's fixed in Airbyte 2.0: The knowledge sources do not contain information about this specific fix being addressed in Airbyte 2.0. What the GCS changelog does show is that version 0.2.11 added a fix for "Json to Avro conversion when there is field name clash from combined restrictions (
anyOf
, 
oneOf
, 
allOf
fields)" and version 0.2.16 added "handler to process null values in arrays." [GCS Changelog] The current GCS connector version is 0.4.9. Practical workarounds based on the knowledge sources: • Use JSON or CSV output format instead of Parquet/Avro as a temporary workaround, since the conversion issue only affects Avro/Parquet. [Forum: MSSQL CDC sync fails] • Upgrade to the latest GCS connector version (0.4.9), as many fixes have been shipped since older versions. [GitHub Issue #64505] The knowledge sources do not confirm whether this specific object-type-in-string-field issue is resolved in any particular Airbyte version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai does thos work same in s3 beacuse s3 is not breaking
k
Based on the knowledge sources available, here's what can be confirmed: The S3 destination (version 1.0.0+) introduced significant improvements to Avro/Parquet conversion that make it more resilient to type mismatches: > "Conversion failures are captured: values not matching the client schema will no longer break syncs using Avro or Parquet formats." [S3 Migration Guide] This means that in the newer S3 destination, instead of breaking the sync, a field that can't be converted is nulled out, and the failure is recorded in 
_airbyte_meta.changes[]
. This is why S3 doesn't break while GCS does — the S3 destination has more fault-tolerant conversion handling. The GCS destination's Parquet writer still goes through the strict JSON → Avro → Parquet pipeline, which throws an exception when a field's actual type (e.g., an object 
{id: "50bafb55..."}
) doesn't match the expected Avro union type (e.g., 
NULL, STRING
). [GCS Output Schema] In summary: • S3 (v1.0.0+): Tolerates conversion failures — nulls the bad field and continues the sync. • GCS: Strict conversion — throws an exception and breaks the sync. The knowledge sources do not confirm whether the GCS destination has received the same fault-tolerant conversion improvements as the S3 destination. Upgrading to the latest GCS connector version (0.4.9) is recommended, as it may include some of these improvements. [GitHub Issue #64505] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
does the s3 also work the same way @kapa.ai
k
Yes, based on the knowledge sources, the S3 destination (v1.0.0+) works in exactly the way described in the previous answer. Specifically: • Conversion failures no longer break syncs in Avro or Parquet formats. Instead, the bad value is nulled out and the failure is recorded in 
_airbyte_meta.changes[]
. [S3 Migration Guide] • The 
_airbyte_meta
field captures these changes with details about which field was affected, the type of change (
NULLED
or 
TRUNCATED
), and the reason. [S3 Migration Guide] For example, if an array item can't be converted to the expected type, it gets nulled: > _"The nulled fields represent conversion failures and will appear in 
_airbyte_meta.changes[]
."_ [S3 Array Types] So in your case, if the 
group
field arrives as an object 
{id: "50bafb55-..."}
but the schema expects 
NULL
or 
STRING
, the S3 destination (v1.0.0+) would null the field and continue the sync, rather than throwing an exception and breaking it. This is the key behavioral difference from the GCS destination. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
for sync schme avro to parquet does s3 do @kapa.ai
k
Based on the knowledge sources, here is how the S3 destination handles the Avro and Parquet output formats: Avro • Data is serialized using binary encoding. • The source JSON schema is first converted to an Avro schema, then the JSON object is converted to an Avro record. [S3 Supported Output Schema] Parquet • The conversion pipeline is: JSON → Avro → Parquet. • The JSON schema is first converted to an Avro schema, then to an Avro record, and finally outputted to Parquet format. [S3 Parquet Docs] Key improvement in S3 v1.0.0+ As discussed earlier, the S3 destination introduced fault-tolerant conversion: • Conversion failures no longer break syncs — bad values are nulled and tracked in 
_airbyte_meta.changes[]
. [S3 Migration Guide] • Date/time types were simplified — all time types are now converted to integral logical types instead of the old union-of-types approach. [S3 Date and Time Types] • Array handling was also improved for both Avro and Parquet formats. [S3 Array Types] So yes, S3 uses the same JSON → Avro → Parquet pipeline as GCS, but with more resilient error handling in newer versions. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
l
When I try to create more than 100 connections, the API/UI returns a 
422
error like: "Cannot create connection: organization has reached the maximum limit of 100 connections. organizationId=00000000-0000-0000-0000-000000000000"
👀 1
a
i took the Db of airbyte and temporal and temporal_visiblity and moved to new location and resorted to Airbyte 1.6.2, now getting this 2026-05-23T032343.183Z DEBUG CREATE TABLE namespaces( partition_id INTEGER NOT NULL, id BYTEA NOT NULL, name VARCHAR(255) UNIQUE NOT NULL, notification_version BIGINT NOT NULL, data BYTEA NOT NULL, data_encoding VARCHAR(16) NOT NULL, is_global BOOLEAN NOT NULL, PRIMARY KEY(partition_id, id) ); {“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:161"} 2026-05-23T032343.184Z WARN Duplicate update, most likely due to previous partially succeeded update attempt. Ignoring it and continue. {“error”: “pq: relation \“namespaces\” already exists”, “logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:171"} 2026-05-23T032343.184Z DEBUG CREATE TABLE namespace_metadata ( partition_id INTEGER NOT NULL, notification_version BIGINT NOT NULL, PRIMARY KEY(partition_id) ); {“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:161"} 2026-05-23T032343.188Z WARN Duplicate update, most likely due to previous partially succeeded update attempt. Ignoring it and continue. {“error”: “pq: relation \“namespace_metadata\” already exists”, “logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:171"} 2026-05-23T032343.188Z DEBUG INSERT INTO namespace_metadata (partition_id, notification_version) VALUES (54321, 1); {“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:161"} 2026-05-23T032343.194Z ERROR Unable to update SQL schema. {“error”: “error executing statement: pq: duplicate key value violates unique constraint \“namespace_metadata_pkey\“”, “logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/tools/sql/handler.go:77”} @kapa.ai
k
The error you're seeing is a known issue when migrating Airbyte's 
temporal
and 
temporal_visibility
databases to a new location. The Temporal auto-setup script is trying to run schema migrations on databases that already have the schema applied, and it's hitting duplicate key/relation errors. Specifically: • 
pq: relation "namespaces" already exists
→ WARN, ignored and continued • 
pq: duplicate key value violates unique constraint "namespace_metadata_pkey"
→ ERROR, stops the process The WARN-level duplicates are handled gracefully, but the 
INSERT INTO namespace_metadata
statement fails because the row already exists from your restored database. Why this happens When you restore the 
temporal
database from a dump, the schema and seed data (including the 
namespace_metadata
row) are already present. The Temporal auto-setup script then tries to re-run the same 
INSERT
statement, which violates the primary key constraint. [Forum: Cannot start with empty DBs] This is consistent with what others have experienced when moving Airbyte databases — the Temporal pod is sensitive to the state of the database it connects to. [GitHub Issue #59730] What to try 1. Set 
SKIP_SCHEMA_SETUP=true
 for the Temporal container. Since your databases are already fully migrated from the source instance, you don't need the auto-setup to run again. This is the recommended approach when databases are pre-populated. [Forum: Cannot start with empty DBs] 2. Verify the restore was complete — ensure all three databases (
airbyte
, 
temporal
, 
temporal_visibility
) were fully restored before starting Airbyte. [GitHub Issue #49304] 3. Ensure the database user has the correct permissions — the Temporal user needs 
GRANT ALL
on the 
temporal
and 
temporal_visibility
databases. [Forum: Cannot start with empty DBs] The knowledge sources do not contain enough detail to confirm whether 
SKIP_SCHEMA_SETUP
is exposed as a configurable environment variable in Airbyte 1.6.2's Helm chart or Docker Compose setup, so you may need to check your deployment's configuration options for that setting. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
[Fx] PROVIDE *temporal.ServicesMetadata[group = “services”] <= go.temporal.io/server/temporal.FrontendServiceProvider() [Fx] PROVIDE *temporal.ServicesMetadata[group = “services”] <= go.temporal.io/server/temporal.InternalFrontendServiceProvider() [Fx] PROVIDE *temporal.ServicesMetadata[group = “services”] <= go.temporal.io/server/temporal.WorkerServiceProvider() [Fx] PROVIDE *cluster.Config <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider() [Fx] PROVIDE config.Persistence <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider() [Fx] PROVIDE *dynamicconfig.Collection <= go.temporal.io/server/common/dynamicconfig.init.func1() [Fx] PROVIDE pingable.Pingable[group = “deadlockDetectorRoots”] <= fx.Annotate(go.temporal.io/server/common/dynamicconfig.init.func2(), fx.ResultTags([“group:\“deadlockDetectorRoots\“”]) [Fx] PROVIDE *pprof.PProfInitializerImpl <= go.temporal.io/server/common/pprof.NewInitializer() [Fx] PROVIDE []trace.SpanExporter <= go.temporal.io/server/temporal.init.func2() [Fx] SUPPLY []temporal.ServerOption [Fx] RUN supply: stub([]temporal.ServerOption) in 9.67µs [Fx] RUN provide: go.temporal.io/server/temporal.ServerOptionsProvider() in 20.841819ms [Fx] Error returned: received non-nil error from function “go.temporal.io/server/temporal”.ServerOptionsProvider /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go182 sql schema version compatibility check failed: version mismatch for keyspace/database: “temporal_new”. Expected version: 1.16 cannot be greater than Actual version: 0.0 [Fx] ERROR Failed to initialize custom logger: could not build arguments for function “go.uber.org/fx”.(*module).constructCustomLogger.func2 /home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/module.go294 failed to build fxevent.Logger: could not build arguments for function “go.temporal.io/server/temporal”.init.func7 /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go1008 failed to build log.Logger: received non-nil error from function “go.temporal.io/server/temporal”.ServerOptionsProvider /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go182 sql schema version compatibility check failed: version mismatch for keyspace/database: “temporal_new”. Expected version: 1.16 cannot be greater than Actual version: 0.0 Unable to create server. Error: could not build arguments for function “go.uber.org/fx”.(*module).constructCustomLogger.func2 (/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/module.go:294): failed to build fxevent.Logger: could not build arguments for function “go.temporal.io/server/temporal”.init.func7 (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1008): failed to build log.Logger: received non-nil error from function “go.temporal.io/server/temporal”.ServerOptionsProvider (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:182): sql schema version compatibility check failed: version mismatch for keyspace/database: “temporal_new”. Expected version: 1.16 cannot be greater than Actual version: 0.0. @kapa.ai
k
The error is clear:
Copy code
sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal_new". 
Expected version: 1.16 cannot be greater than Actual version: 0.0
This means the Temporal server is trying to connect to a database called 
temporal_new
 that has no schema applied (version 
0.0
), but the Temporal server expects schema version 
1.16
. What went wrong When you moved your databases to the new location, it appears the 
temporal
database was either: • Renamed or pointed to a different database name (
temporal_new
instead of 
temporal
) • Not properly restored — the restore completed but the schema migrations were not applied (empty schema at version 
0.0
) This is different from your previous error where the schema already existed. Here, the target database exists but is empty/uninitialized. How to fix Option 1: Fix the database name Check your Helm 
values.yaml
or environment variables for 
DBNAME
and 
VISIBILITY_DBNAME
. They should point to your restored databases, not a new empty one. [Forum: Cannot start with empty DBs]
Copy code
DBNAME=temporal
VISIBILITY_DBNAME=temporal_visibility
If something is pointing to 
temporal_new
, correct it back to 
temporal
. Option 2: Re-run schema setup on the empty database If 
temporal_new
is intentional and is a freshly created empty database, the Temporal auto-setup needs to initialize it. Ensure 
SKIP_SCHEMA_SETUP=false
so the schema migrations run. [Forum: Cannot start with empty DBs] Option 3: Verify the restore Confirm the 
temporal
database was fully restored by checking the schema version:
Copy code
SELECT * FROM schema_version;
A properly restored database should show version 
1.6
or higher, not 
0.0
. [Forum: Airbyte-temporal started failed] The database user also needs 
GRANT ALL
privileges on the temporal databases for schema setup to succeed. [Forum: Unable to create SQL database] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
“SELECT * FROM schema_version;” Password for user postgres: version_partition | db_name | creation_time | curr_version | min_compatible_version -------------------+----------+----------------------------+--------------+------------------------ 0 | temporal | 2025-05-24 025327.920857 | 1.16 | 1.0 (1 row) @kapa.ai
k
Your 
temporal
database schema is at version 1.16, which is exactly what the Temporal server expects. So the 
temporal
database itself is correctly restored and initialized. This means the issue from the previous error — 
version mismatch for keyspace/database: "temporal_new". Expected version: 1.16 cannot be greater than Actual version: 0.0
— is pointing to a different database named 
temporal_new
, not your restored 
temporal
database. Next steps 1. Check your configuration to find where 
temporal_new
is being referenced. Look at your Helm 
values.yaml
or environment variables for 
DBNAME
— it should be set to 
temporal
, not 
temporal_new
. [Forum: Cannot start with empty DBs] 2. Check 
temporal_visibility
as well:
Copy code
\c temporal_visibility
SELECT * FROM schema_version;
This should also show a valid version (typically 
1.1
). [Forum: Airbyte-temporal started failed] 3. If 
temporal_new
is an unintended empty database that was created by a misconfigured 
DBNAME
variable, you can either drop it or correct the configuration to point back to 
temporal
. Your restored 
temporal
database is fine — the problem is a misconfiguration pointing Temporal to the wrong (empty) database. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
temporal_new this name of DB where i restored the temporal @kapa.ai
@kapa.ai
k
Got it — so 
temporal_new
is the actual name of the database where you restored the Temporal data, and it has the correct schema at version 
1.16
. The problem is that your Temporal configuration is still pointing to 
temporal
(or some other name) instead of 
temporal_new
. What you need to do Update your Temporal configuration to use 
temporal_new
as the database name. In your Helm 
values.yaml
or environment variables, set: [Forum: Cannot start with empty DBs]
Copy code
temporal:
  extraEnv:
    - name: DBNAME
      value: 'temporal_new'
    - name: VISIBILITY_DBNAME
      value: 'temporal_visibility'  # or whatever name you used for visibility DB
Or if using Docker Compose / environment variables directly:
Copy code
DBNAME=temporal_new
VISIBILITY_DBNAME=temporal_visibility
Also verify 1. Check 
temporal_visibility
schema version as well:
Copy code
\c temporal_visibility_new   -- or whatever name you used
SELECT * FROM schema_version;
It should show version 
1.1
or higher. [Forum: Airbyte-temporal started failed] 2. Ensure the database user has full permissions on `temporal_new`:
Copy code
GRANT ALL PRIVILEGES ON DATABASE temporal_new TO <your_airbyte_user>;
[Forum: Unable to create SQL database] 3. Since the schema is already applied, set 
SKIP_SCHEMA_SETUP=true
 (or 
SKIP_DB_CREATE=true
) to prevent the auto-setup from trying to re-run migrations on the already-initialized database. [Forum: Cannot start with empty DBs] Once the 
DBNAME
variable points to 
temporal_new
, the version mismatch error should be resolved. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
extraEnv: - name: “SQL_TLS” value: “true” - name: “SQL_TLS_DISABLE_HOST_VERIFICATION” value: “true” - name: “SQL_TLS_ENABLED” value: “true” - name: “SQL_HOST_VERIFICATION” value: “false” - name: SKIP_DB_CREATE value: “true” - name: “SKIP_SCHEMA_SETUP” value: “true” - name: DBNAME value: temporal_new - name: VISIBILITY_DBNAME value: temporal_visibility_new @kapa.ai this i have already. do i restore again as i added this later name: “SKIP_SCHEMA_SETUP” value: “true” after start of airbyte
k
Based on your configuration, you already have the correct setup:
Copy code
extraEnv:
  - name: DBNAME
    value: temporal_new
  - name: VISIBILITY_DBNAME
    value: temporal_visibility_new
  - name: SKIP_DB_CREATE
    value: "true"
  - name: SKIP_SCHEMA_SETUP
    value: "true"
Do you need to restore again? No, you should not need to restore again — as long as: 1. Your 
temporal_new
database has schema version 
1.16
(which you already confirmed ✅) 2. Your 
temporal_visibility_new
database also has a valid schema version Check 
temporal_visibility_new
first:
Copy code
\c temporal_visibility_new
SELECT * FROM schema_version;
It should show version 
1.1
or higher. [Forum: Airbyte-temporal started failed] Regarding SKIP_SCHEMA_SETUP being added after startup Since 
SKIP_SCHEMA_SETUP=true
was added after Airbyte had already started, the auto-setup script may have already attempted (and partially failed) to re-run migrations on your restored databases. However, since your schema is already at the correct version (
1.16
), and you now have both 
SKIP_DB_CREATE=true
and 
SKIP_SCHEMA_SETUP=true
set, a fresh restart of the Temporal pod with these settings should skip all schema operations and connect directly to your restored databases. [Forum: Cannot start with empty DBs] Also verify the database user has full privileges on both databases:
Copy code
GRANT ALL PRIVILEGES ON DATABASE temporal_new TO <your_airbyte_user>;
GRANT ALL PRIVILEGES ON DATABASE temporal_visibility_new TO <your_airbyte_user>;
[Forum: Unable to create SQL database] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
if keep the fresh temporal and termpora_visility and resote just airbyte DB will it work or fail @kapa.ai
k
Based on the knowledge sources and your specific situation, here's what would happen: Restoring only the airbyte DB with fresh temporal and temporal_visibility It will partially work, but with caveats: What will work ✅ • The 
airbyte
database contains your sources, destinations, connections, and configuration. Restoring it means your Airbyte UI will show all your existing connectors and connections. • Fresh 
temporal
and 
temporal_visibility
databases will be initialized cleanly by the Temporal auto-setup (no schema conflicts). [Forum: Unable to create SQL database] What will be lost ❌ • Temporal workflow state — all in-progress or scheduled sync workflows stored in 
temporal
and 
temporal_visibility
will be gone. Your connections will exist in the UI but their Temporal workflows will need to be re-triggered. • Job history — sync job history is stored in the 
airbyte
database, so that should be preserved. However, any active/running jobs at the time of migration will be lost from Temporal's perspective. Important note on fresh Temporal DBs With fresh (empty) 
temporal
and 
temporal_visibility
databases, you must ensure: • 
SKIP_DB_CREATE=false
and 
SKIP_SCHEMA_SETUP=false
so Temporal auto-setup initializes the schema • The database user has 
CREATEDB
permissions or the databases are pre-created with 
GRANT ALL
to the Airbyte user [Forum: Cannot start with empty DBs]
Copy code
CREATE DATABASE temporal WITH OWNER = postgres;
CREATE DATABASE temporal_visibility WITH OWNER = postgres;
[GitHub Issue #49304] Bottom line: This approach is simpler and avoids the schema version mismatch issues you've been facing. You'll just need to manually re-trigger your connections after startup to recreate the Temporal workflows. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
Snapchat integration @kapa.ai it give 2 days back data why it has delay anyone else reported.
k
Yes, this is a known and documented behavior in the Snapchat Marketing connector. The 2-day delay is intentional and was explicitly added as a fix. Why there is a 2-day lookback delay In version 1.5.29 (released 2026-02-24), the following fix was shipped: > _"fix(source-snapchat-marketing): add lookback_window P2D to mitigate UTC timezone mismatch"_ [Snapchat Marketing Changelog] The 
P2D
means a 2-day lookback window was added to handle timezone mismatches between the connector and the Snapchat API. Specifically: • The Snapchat API uses UTC timestamps • The connector was missing data due to timezone offset differences (e.g., records created late in the day in one timezone but attributed to the previous day in UTC) • The 2-day lookback was added as a safety buffer to ensure no records are missed Has anyone else reported it? Yes — there is an open GitHub issue (#54139) specifically about the 
end_datetime
defaulting to midnight (
00:00:00.000Z
) instead of end of day (
23:59:59.999Z
), which causes records updated later in the day to be missed. [GitHub Issue #54139] The 2-day lookback in v1.5.29 was added as a mitigation for this class of issue. What you can expect • The 2-day delay is by design in the current connector version • Data will eventually be synced — it just arrives with a ~2 day lag • If you need more up-to-date data, this is a known limitation of the current implementation You may want to follow or comment on GitHub Issue #54139 to track progress on a proper fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
The 2-day delay is by design in @kapa.ai did it got fixed.
k
Based on the knowledge sources available, I cannot confirm whether the 2-day lookback delay has been fixed. The changelog shows that version 1.5.29 (2026-02-24) added the 
P2D
lookback window as a fix for the UTC timezone mismatch. [Snapchat Marketing Changelog] Looking through all subsequent versions up to the latest 1.5.41 (2026-06-30), none of the changelog entries mention: • Removing the 2-day lookback window • Fixing the UTC timezone mismatch in a different way • Any changes to the 
lookback_window
behavior The only related fix after v1.5.29 was in v1.5.16 (2025-08-11): > "Fix start time for stats streams" [Snapchat Marketing Changelog] But this predates v1.5.29 and is unrelated to the lookback window. The knowledge sources do not contain enough information to confirm whether the 2-day delay has been removed or resolved in any version. The 
P2D
lookback window appears to still be in place as of the latest version. I'd recommend: • Checking the open issue #54139 for updates • Joining the Airbyte Community Slack to ask directly 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
github.com/airbytehq/airbyte/pull/73634 @kapa.ai in this it say 2PD. what it means like in incremental append instead of picking from cusor it is always picking 2 say of data . so why will o get any delay, my data should be current.
k
You're right to question this. Let me clarify what 
P2D
lookback actually means: What P2D lookback window means 
P2D
is an ISO 8601 duration meaning 2 days. In the context of 
DatetimeBasedCursor
, the lookback window subtracts 2 days from the last cursor value before starting the next sync: > "The Lookback window specifies a duration that is subtracted from the last cutoff date before starting to sync." [Connector Builder Docs] So your data should NOT be delayed You are correct — with incremental append + 
P2D
lookback: • ✅ Your current/today's data is still synced • ✅ The connector picks up from 
cursor - 2 days
to 
now
• ✅ No 2-day delay on current data What actually happens is: • Last sync cursor = 
2026-06-28
• Next sync starts from = 
2026-06-26
(2 days back) • Next sync ends at = 
now
(today) Why P2D was added The 2-day lookback was added to re-fetch the last 2 days of already-synced data to catch any records that were missed due to UTC timezone mismatches. [Snapchat Marketing Changelog] What this means for you • You should not be seeing a 2-day delay in your data • You will see the last 2 days re-synced on every run (duplicates handled by dedup if configured) • If you are seeing a genuine 2-day delay in current data, it is likely a different issue unrelated to the 
P2D
lookback window Can you clarify what you mean by "2-day delay"? Is today's data missing, or are you seeing duplicate records from 2 days ago? 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
but my connector is alwys have delay of 2 day data. what is that issue @kapa.ai
k
The knowledge sources do not contain enough information to confirm the exact cause of a consistent 2-day data delay in your specific case. However, based on what is available, here are the most likely causes: Most likely cause: end_datetime defaulting to midnight There is an open GitHub issue (#54139) specifically reporting this problem: > _"When the 
end_date
parameter is not specified for incremental syncs, the 
end_datetime
partition defaults to midnight (
00:00:00.000Z
). This causes issues because any records updated on the same day (but after midnight) are excluded from the sync."_ [GitHub Issue #54139] This means if you sync today, the connector only picks up data up to midnight of today, effectively always being behind by up to a day or more. The fix that was proposed
Copy code
end_datetime:
  type: MinMaxDatetime
  datetime: '{{ (config.get("end_date") or now_utc().strftime("%Y-%m-%d")) + "T23:59:59.999Z" }}'
  datetime_format: '%Y-%m-%dT%H:%M:%S.%fZ'
This was proposed to change 
end_datetime
from 
00:00:00.000Z
to 
23:59:59.999Z
. [GitHub Issue #54139] A related fix was shipped in v1.5.13 (2025-07-02): > "Fix end datetime to include today" [Snapchat Marketing Changelog] What you should do 1. Check your connector version — ensure you are on at least v1.5.13 or higher (latest is v1.5.41) which includes the 
end_datetime
fix 2. Set an explicit 
end_date
 in your connector config if you are not already doing so 3. Comment on GitHub Issue #54139 describing your exact delay pattern to get more targeted help from the Airbyte team If you are already on v1.5.13+ and still seeing a consistent 2-day delay, the knowledge sources do not contain enough information to explain why, and I'd recommend raising it directly on the GitHub issue. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai Does aitbyte always deposit time in UTC?
k
Yes, Airbyte generally stores and transports timestamps in UTC across most of its components. Here's a breakdown: Airbyte and UTC Timestamps Backend & APIs All timestamps are stored in UTC on the backend. The API, SDK, and MCP server all read and return timestamps as ISO 8601 UTC strings (e.g., 
2026-03-18T13:32:00Z
). [Time Zones Docs] Destinations • S3 / GCS (Avro/Parquet): Timestamps with timezone are stored in UTC. The 
is_adjusted_to_utc
field in Parquet metadata is always 
true
. [S3 Migration Guide] • S3 Data Lake / GCS Data Lake (Iceberg): 
time with timezone
and 
timestamp with timezone
types are converted to UTC before writing. [S3 Data Lake] • Avro (blob storage): Timestamps are stored as 
timestamp-micros
— microseconds from 
1970-01-01T00:00:00Z
(UTC). [JSON to Avro Conversion] Important Caveat: timestamp_without_timezone Airbyte's type system distinguishes between: • 
timestamp_with_timezone
→ always converted to UTC • 
timestamp_without_timezone
→ stored as-is, no timezone conversion applied [Supported Data Types] Known Exceptions / Bugs There have been reported issues where timezone conversion was not always applied correctly: • MySQL source: Configured server timezone (e.g., 
Europe/Rome
) was ignored during sync, with timestamps inserted as-is without UTC conversion. [GitHub Issue #52656] • MySQL CDC vs Full Refresh: 
datetime
fields (without timezone) were handled differently between CDC and full refresh syncs. [GitHub Issue #54718] • S3 `time_of_day_with_timezone`: Timezones were not respected when converting until a bugfix was shipped. [S3 Migration Guide] Summary So while Airbyte aims to use UTC throughout, the behavior depends on whether the field is typed as 
with_timezone
or 
without_timezone
, and some source connectors have had known bugs with timezone handling. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by