https://linen.dev logo
Join Slack
Powered by
with 1.4.0 , airbyte was changed, now we got auto ...
# ask-ai
a
with 1.4.0 , airbyte was changed, now we got auto schema discovery setting, was it used to run before every sync in past before 1.4.0?
k
Based on the knowledge sources provided, I can answer your question about the auto schema discovery setting in Airbyte version 1.4.0 and how it compares to previous versions. Prior to version 1.4.0, Airbyte's behavior regarding schema discovery was different: 1. For Airbyte Cloud users: Schema discovery was run before every sync, but with a 15-minute cooldown period. If a sync was requested within 15 minutes of the previous schema check, it would use the previous result. 2. For self-managed (Open Source) users: Schema discovery was performed at most once every 24 hours before syncing. This meant that source schema changes might not be automatically propagated if less than 24 hours had passed since the previous schema check. With the release of Airbyte version 1.4.0, there have been changes to the schema discovery process: 1. For Airbyte Cloud users: Schema discovery is now always run as part of the sync. This ensures that the sync always uses the latest schema, which improves the functionality of auto-propagation and backfill features. 2. For self-managed (Open Source) users: A new configuration option called 
DISCOVER_REFRESH_WINDOW_MINUTES
has been introduced. This allows users to set the minimum number of minutes Airbyte will wait before refreshing the schema for sources. The default value is set to 1440 minutes (24 hours), which is similar to the previous behavior. These changes are documented in the [Airbyte 1.4.0 Release Notes](https://docs.airbyte.com/release_notes/v-1.4) and the [July 2024 Release Notes](https://docs.airbyte.com/release_notes/july_2024). It's worth noting that the auto schema discovery setting you're referring to is likely the new "Detect and Propagate Schema Changes" option, which allows users to choose how Airbyte handles schema changes. This feature provides more control over how schema changes are managed and propagated.
a
@kapa.ai do you have proof that it was 24 hours before 1.4.0? any source links
k
A message from kapa.ai
Yes, I have proof that schema discovery was performed at most once every 24 hours for self-managed (Open Source) Airbyte instances before version 1.4.0. Here's the relevant information from the knowledge sources: [Schema Change Management](https://docs.airbyte.com/using-airbyte/schema-change-management) states: "For self-managed users, Airbyte checks for any changes in your source schema immediately before syncing, at most every 24 hours. This means your source schema changes may not be automatically propagated if it has been less than 24 hours since the previous schema check." This documentation clearly indicates the 24-hour interval for schema checks in self-managed Airbyte instances before the changes introduced in version 1.4.0. Additionally, the [Airbyte 1.4.0 Release Notes](https://docs.airbyte.com/release_notes/v-1.4) corroborate this information by mentioning the new configuration option: "You may now use a new toggle, 
DISCOVER_REFRESH_WINDOW_MINUTES
, to set the minimum number of minutes Airbyte will wait to refresh the schema for your sources. By setting a larger number, you run automatic schema detection less frequently, which can result in gains in sync performance. The default configuration in Airbyte Open Source is 
1440
, which refreshes schemas every 24 hours." This new configuration option was introduced to allow users to adjust the schema refresh interval, with the default value set to 1440 minutes (24 hours), matching the previous behavior. These sources provide clear evidence that the schema discovery interval was indeed 24 hours for self-managed Airbyte instances before version 1.4.0.
Open in Slack
PreviousNext
Powered by