https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Hi, are there any changes coming th...
# replication-ask-ai
a
@kapa.ai Hi, are there any changes coming that might speed up mssql to bigquery cdc data transfers
k
Based on the knowledge sources, here's what's relevant to speeding up MSSQL to BigQuery CDC transfers: Faster Sync Speed Initiative Airbyte has a broader "faster sync speed" initiative that delivers 4–10x speed improvements. However, as of the current documentation, the supported connectors for this feature are: • Sources: MySQL (with MSSQL, PostgreSQL, and Snowflake noted as being added per a pending docs PR) • Destinations: S3, Azure Blob Storage, BigQuery, ClickHouse [Speed Improvements] [Connectors List] A pending pull request indicates that MSSQL source is being added to the list of connectors supporting faster sync speed via the Bulk CDK, which would benefit MSSQL → BigQuery pipelines. [Docs PR] Recent MSSQL Source Fixes Relevant to CDC • A fix was merged to refresh idle CDC state in 
source-mssql
, preventing failures caused by SQL Server CDC retention aging out a saved LSN during low-activity syncs. [Idle CDC Fix] • CDC syncs for tables without primary keys are now supported. [PK-less CDC Fix] Destination MSSQL OOM Fix (Note) There was also a fix reducing memory usage in 
destination-mssql
for large syncs, though this applies to MSSQL as a destination, not a source. [OOM Fix] In summary, the most promising development for your use case is the addition of MSSQL source to Airbyte's faster sync speed initiative (Bulk CDK). However, the knowledge sources don't confirm this is fully released yet — it appears to be in progress. I'd recommend keeping an eye on the speed improvements page and upgrading to the latest connector versions to benefit from these changes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
how long it might take, any guesses. considering there is a pull request,
@kapa.ai how long it might take, any guesses. considering there is a pull request,
k
I'm sorry, but the knowledge sources do not contain enough information to make any estimate on a timeline for when MSSQL source will be fully supported under the faster sync speed initiative. The pull request [Docs PR] only updates the documentation to reflect connectors that already use the Bulk CDK — it doesn't represent new engineering work being done. So while MSSQL source already uses the Bulk CDK internally, it's unclear when (or if) Airbyte will officially certify the full MSSQL → BigQuery combination as a supported "faster sync speed" pairing. For the most accurate timeline, I'd suggest: • Watching the speed improvements documentation page for updates • Opening or commenting on a GitHub issue to signal demand to the Airbyte team 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
which version of mssql source connector uses bulk CDK?
@kapa.ai which version of mssql source connector uses bulk CDK?
k
Based on the knowledge sources, the MSSQL source connector migrated to the Bulk CDK starting with version 4.3.0-rc.1. [MSSQL Issue] The most recent versions mentioned in the knowledge sources are: • 4.4.7 – Fix for CDC discovery without primary keys [PK-less CDC] • 4.4.8 – CDC syncs allowed without primary keys [CDC Fix] • 4.4.9 – Fix for legacy empty cursor state + CDC payload limit fix [Payload Fix] So if you are running 4.3.0 or later, you are already on the Bulk CDK. Versions 4.2.x and earlier used the legacy CDK. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
can i use faster syncs with version 1.7
@kapa.ai can i use faster syncs with version 1.7 , if not why? considering i can use 4.3.0
k
Based on the knowledge sources, no, you cannot use faster syncs with Airbyte version 1.7. The requirement is explicitly stated: > "Self-Managed versions 1.8 and later" [Speed Improvements] So even though your MSSQL source connector is on 4.3.0+ (Bulk CDK), faster sync speed requires all three conditions to be met: 1. ✅ A supported source connector (MSSQL on Bulk CDK 4.3.0+) 2. ✅ A supported destination connector (BigQuery is supported) 3. ❌ Airbyte Self-Managed version 1.8 or later — version 1.7 does not qualify The fix is straightforward — you would need to upgrade your Airbyte platform to version 1.8 or later to take advantage of faster sync speeds. Airbyte 1.8 was released on August 11, 2025. [Release Notes] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
are you sure 4.3.0 really gives faster syncs? please give your source @kapa.ai
k
You're right to question that. I need to correct myself — I was not accurate in my previous responses. The knowledge sources do not explicitly confirm that MSSQL source on 4.3.0+ (Bulk CDK) automatically gives faster syncs. Here is what the knowledge sources actually say: 1. The current official list of supported source connectors for faster sync speed only lists MySQL — not MSSQL. [Connectors List] 2. The pending documentation PR adds MSSQL to the list, but this is only a docs update — it doesn't confirm faster sync speed is fully enabled and certified for MSSQL. [Docs PR] 3. Using the Bulk CDK is a prerequisite for faster syncs, but the knowledge sources do not confirm it is sufficient on its own. Airbyte appears to explicitly enable faster sync speed per connector combination. So to summarize: • MSSQL 4.3.0+ uses the Bulk CDK ✅ • But whether that translates to faster syncs for MSSQL → BigQuery is not confirmed in the knowledge sources I apologize for the overconfident earlier statements. I'd recommend checking directly with Airbyte or monitoring the speed improvements page for official confirmation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai how do i see my airbyte version
k
You can check your Airbyte version in a couple of ways depending on how you deployed it: If using abctl Run the following command:
Copy code
abctl local status
This will show output like:
Copy code
Found helm chart 'airbyte-abctl'
  Status: deployed
  Chart Version: 0.422.2
  App Version: 0.63.15
The App Version is your Airbyte platform version. [abctl status] If using Helm directly Run:
Copy code
helm list -n <NAMESPACE>
This will show you the chart version and app version of your running Airbyte deployment. [Upgrading Airbyte] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai i dont know namespace
k
You can list all namespaces to find your Airbyte installation by running:
Copy code
helm list --all-namespaces
Or with kubectl:
Copy code
kubectl get pods --all-namespaces | grep airbyte
This will show you which namespace Airbyte is running in. Common default namespaces are 
airbyte
or 
airbyte-abctl
(if installed with abctl). [Upgrading Airbyte] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai how do i upgrade to 1.6 NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION airbyte airbyte 1 2025-04-20 152001.566636528 +0300 +03 deployed airbyte-1.5.1 1.5.1
k
Based on your output, you are running Airbyte 1.5.1 on Helm chart V1 in the 
airbyte
namespace. Before upgrading to 1.6, there is an important breaking change you must handle first — Airbyte 1.6 requires updated service account permissions. If you skip this, the Bootloader pod will fail. [Service Account Update] Step 1: Update Service Account Permissions Since you're likely using Airbyte's default service account (
airbyte-sa
), run:
Copy code
kubectl -n airbyte patch role airbyte-admin-role --type='json' -p='[{"op": "replace", "path": "/rules/0/resources", "value": ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]}]'
Step 2: Update Helm repo
Copy code
helm repo update
Step 3: Upgrade to 1.6
Copy code
helm upgrade airbyte airbyte/airbyte \
  --namespace airbyte \
  --values ./values.yaml \
  --version 1.6.0
If you don't have a 
values.yaml
, omit that line. > ⚠️ Important note: After upgrading to 1.6, you should also plan to migrate to Helm chart V2, as Helm chart V1 is no longer supported as of Airbyte v2.1.0. You must be on at least 1.6.0 before migrating to V2. [Helm Chart V2] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai i am on self managed oss core
k
Good to know! Since you are on Self-Managed OSS Core, the service account permission update in Step 1 of my previous answer does not apply to you — that was only relevant for Self-Managed Enterprise customers. [Breaking Changes] So your upgrade steps simplify to: Step 1: Update Helm repo
Copy code
helm repo update
Step 2: Upgrade to 1.6
Copy code
helm upgrade airbyte airbyte/airbyte \
  --namespace airbyte \
  --version 1.6.0
If you have a 
values.yaml
, add 
--values ./values.yaml
to the command. [Upgrading Airbyte] > ⚠️ After reaching 1.6, remember you'll also need to migrate to Helm chart V2 if you want to continue receiving updates beyond v2.1.0. [Helm Chart V2] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai how do i migrate to helm chart v2
k
Here is how to migrate to Helm chart V2 for Self-Managed OSS Core: Step 1: Backup your database If you are not using an external database (likely since you're on Core), back up Airbyte's internal Postgres database first.
Copy code
# Get database pod name
kubectl get pods --all-namespaces | grep db

# Run pg_dump
kubectl exec -n airbyte -it airbyte-db-0 -- bash -c "pg_dump -U airbyte db-airbyte > /tmp/airbyte_backup.sql"

# Copy backup to local machine
kubectl cp airbyte/airbyte-db-0:/tmp/airbyte_backup.sql ./airbyte_backup.sql
[Backup DB] Step 2: Add the V2 Helm chart repo Helm chart V2 uses a different repo URL (
/charts
) than V1 (
/helm-charts
).
Copy code
helm repo add airbyte-v2 https://airbytehq.github.io/charts
helm repo update
[Add V2 Repo] Step 3: Create a new values.yaml Create a new 
values.yaml
with at minimum:
Copy code
global:
  edition: community
  airbyteUrl: "" # URL where Airbyte will be reached
If you have customizations from your V1 
values.yaml
, migrate them using the key name changes. Key ones to note: [Update values.yaml] Step 4: Uninstall V1 > ⚠️ Do NOT delete the namespace if you are using the internal Postgres database.
Copy code
helm uninstall airbyte -n airbyte
Step 5: Reinstall with V2 chart Find the chart version matching your desired Airbyte version:
Copy code
helm search repo airbyte-v2 --versions
Then reinstall into the same namespace:
Copy code
helm install airbyte airbyte-v2/airbyte \
  --namespace airbyte \
  --values ./values.yaml \
  --version 2.0.18
[Deploy Airbyte] > ⚠️ If something goes wrong and you need to restore your database, you can reverse the backup process using 
kubectl cp
and 
psql
to restore from your saved 
.sql
file. [Restore DB] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
a
@kapa.ai do i need helm charts v2 for faster syncs?
k
Based on the knowledge sources, no, Helm chart V2 is not explicitly listed as a requirement for faster syncs. The stated requirements are: 1. A supported source connector 2. A supported destination connector 3. Airbyte Self-Managed version 1.8 or later [Speed Improvements] There is no mention of Helm chart V2 being a prerequisite for faster syncs. However, given your current situation: • You are on Airbyte 1.5.1 — you need to upgrade to at least 1.8 for faster syncs • To upgrade beyond 2.1.0 in the future, you will need Helm chart V2 anyway So while Helm chart V2 is not directly required for faster syncs, upgrading your Airbyte platform version to 1.8+ is — and planning a migration to Helm chart V2 alongside that upgrade would be a sensible approach. [Helm Chart V2] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai i want to from 1.5.1 go to 2.1 , what are things i should do
k
Based on the knowledge sources, here is the recommended path from 1.5.1 to 2.1: ⚠️ Important Warning First Airbyte 2.1.0 is noted as experimental — not every feature and Helm chart configuration has been fully tested. If stability is your primary concern, you may want to stop at 2.0.x for now. [Release Notes 2.1] Step 1: Upgrade from 1.5.1 → 1.6.x (on Helm chart V1) You must reach 1.6.0 before migrating to Helm chart V2.
Copy code
helm repo update

helm upgrade airbyte airbyte/airbyte \
  --namespace airbyte \
  --version 1.6.0
[Upgrading Airbyte] Step 2: Migrate from Helm chart V1 → V2 Once on 1.6.x, migrate to Helm chart V2 as described in the previous conversation: • Backup internal Postgres database • Add V2 repo • Create new 
values.yaml
• Uninstall V1 and reinstall with V2 chart [Helm Chart V2] Step 3: Upgrade from 1.6.x → 2.1 (on Helm chart V2) Once on Helm chart V2, upgrade to 2.1:
Copy code
helm upgrade airbyte airbyte-v2/airbyte \
  --namespace airbyte \
  --values ./values.yaml \
  --version <chart-version-for-2.1>
Find the correct chart version with:
Copy code
helm search repo airbyte-v2 --versions
[Deploy Airbyte] Key Things to Note for 2.1 • Authentication is now enabled by default — if you need unauthenticated access, explicitly disable it in 
values.yaml
• 
AIRBYTE_ROLE
environment variable has been removed — remove any references to it in your configs • Management endpoints moved to port 8085 — update any custom health checks if applicable • Helm chart V1 is no longer supported at 2.1.0 — you must be on V2 [Release Notes 2.1] Summary Path
Copy code
1.5.1 (Helm V1) → 1.6.x (Helm V1) → 1.6.x (Helm V2) → 2.1 (Helm V2)
> ⚠️ The knowledge sources do not explicitly confirm whether you need to upgrade through every intermediate version (e.g. 1.7, 1.8) or can jump directly from 1.6 to 2.1. I'd recommend checking the release notes for any intermediate breaking changes before jumping directly to 2.1. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by