https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> failed `--low-resource-mode` upgrad...
# replication-ask-ai
r
@kapa.ai failed 
--low-resource-mode
upgrade on Airbyte 2.1.0 (self-hosted, single-node kind on EC2, abctl v0.30.4). Install now fails on every attempt and has left MinIO missing. Sharing in case it's known or someone's found a way through. What happened: originally installed without 
--low-resource-mode
, later re-ran with the flag. Now every 
abctl local install
(with or without it) fails at ~23s, before pods start:
Copy code
unable to install helm: failed to create patch: The order in patch list:
[map[name:CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST value:0] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST value:0] map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST valueFrom:map[configMapKeyRef:map[key:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST name:airbyte-abctl-airbyte-env]]] ...]
doesn't match $setElementOrder list: [...]
Looks triggered by duplicated 
..._REQUEST
entries (
value:0
vs 
valueFrom
) on the 
airbyte-abctl-airbyte-env
configmap. Impact: the failed upgrade left MinIO's statefulset/service/pod gone (PVC still Bound). Server now throws 
UnknownHostException: airbyte-minio-svc
and discovers fail → schema refresh broken in UI. Config DB intact + backed up; sync data unaffected. Everything Running except MinIO (note: only 
create-bucket
, no 
airbyte-minio-0
server pod):
Copy code
airbyte-abctl-bootloader              0/1   Completed
airbyte-abctl-cron-...                1/1   Running
airbyte-abctl-manifest-server-...     1/1   Running
airbyte-abctl-server-...              1/1   Running
airbyte-abctl-temporal-...            1/1   Running
airbyte-abctl-worker-...              1/1   Running
airbyte-abctl-workload-api-server-... 1/1   Running
airbyte-abctl-workload-launcher-...   1/1   Running
airbyte-db-0                          1/1   Running
airbyte-minio-create-bucket           1/1   Running
Tried: deleting the cron deployment (fails earlier at the configmap patch); abctl is already latest (v0.30.4). Q: Is this a known issue with toggling 
--low-resource-mode
on an existing install? And how do I get a clean install to complete and restore MinIO from here, without a destructive uninstall?
k
Known Issue: --low-resource-mode Toggle Causing Patch Order Failures Yes, this is a known issue. The error you're seeing — duplicated 
..._REQUEST
entries with both 
value:0
and 
valueFrom
in the patch list — is exactly the bug documented for toggling 
--low-resource-mode
on an existing install. [Troubleshoot Deploy] [GitHub Issue #65129] Specifically, the docs note: > This error can occur when installing using 
abctl local install
and subsequently ran 
abctl local install --low-resource-mode
. The fix is listed as "Fixing" (in progress via abctl PR #114), so it is not yet resolved in v0.30.4. Workarounds Reported by the Community Based on what others have tried in the GitHub issues: 1. Pin to a known-good chart version — one user resolved a similar situation by downgrading the chart:abctl local install --chart-version=1.7.2[GitHub Issue #65129] 2. Fresh install (uninstall + reinstall) — some users found that a clean uninstall and reinstall resolved the patch conflict, though this is destructive and you've indicated you want to avoid that. 3. Manually patch the configmap — one user mentioned manually patching the 
airbyte-abctl-airbyte-env
configmap to resolve the duplicate entries, though this is complex and not officially documented. Regarding the Missing MinIO The knowledge sources do not contain specific guidance on restoring a partially-removed MinIO statefulset/service without a full reinstall. However, since your PVC is still 
Bound
, your data should be intact. A few observations from the issue context: • The 
airbyte-minio-create-bucket
job running without 
airbyte-minio-0
suggests the StatefulSet/Service was removed mid-upgrade but the PVC survived. • If you can get a successful 
abctl local install
to complete (e.g., via the chart version pin above), Helm should re-create the MinIO StatefulSet and Service, and the existing PVC should be reattached automatically since it's still 
Bound
. Suggested path forward (non-destructive): 1. Try pinning to chart version 1.7.2 first:abctl local install --chart-version=1.7.2 [your other flags, WITHOUT --low-resource-mode] 2. If that succeeds, MinIO should be restored from the existing PVC. 3. Avoid toggling 
--low-resource-mode
until the upstream fix is released. I want to be clear that the MinIO restoration path is my inference based on how Helm and Kubernetes PVC binding works — the knowledge sources do not explicitly document a recovery procedure for this specific partial-failure state. If the chart version pin doesn't resolve the patch conflict, you may need to engage the #C021JANJ6TY for further assistance. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai Whats the impact of downgrading to chart-version=1.7.2? Will that impact being able to us Airbyte 2.1.0 and abctl v0.30.4
k
Impact of Downgrading to Chart Version 1.7.2 This is an important concern, and the knowledge sources are fairly clear on the implications. Helm Chart V1 is No Longer Supported in Airbyte 2.1 Chart version 1.7.2 is a Helm chart V1 chart. As of Airbyte 2.1.0, Helm chart V1 is explicitly no longer supported: > It's not possible to deploy this or any future version of Airbyte with Helm chart V1. [Airbyte 2.1 Release Notes] So downgrading to chart version 1.7.2 is not compatible with Airbyte 2.1.0. You would effectively be rolling back your Airbyte platform version as well, not just the chart. What You'd Need Instead To run Airbyte 2.1.0, you must use Helm chart V2. With abctl v0.30.0+, Helm chart V2 is used by default. [Upgrade to Helm Chart V2] The chart version 1.7.2 workaround was reported working for users on Airbyte 1.7.x, not 2.1.x. [GitHub Issue #65129] Summary Given your situation, the 1.7.2 downgrade is not a viable path if staying on Airbyte 2.1.0 is a requirement. Unfortunately, the knowledge sources don't provide a clean non-destructive recovery path for your specific patch-conflict + missing MinIO state on 2.1.0. I'd strongly recommend raising this directly in the #C021JANJ6TY where the team has been actively responding to similar upgrade issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
r
@kapa.ai can you help me draft a summarised message to send to get people up to speed on the issue
k
Here's a concise summary you could post to the Airbyte Slack or GitHub: Subject: abctl v0.30.4 + Airbyte 2.1.0 — 
--low-resource-mode
toggle breaks install, MinIO missing Environment: • Airbyte: 2.1.0 (self-hosted, single-node kind on EC2) • abctl: v0.30.4 • Helm chart: V2 (default) What happened: Originally installed without 
--low-resource-mode
, then re-ran with the flag. This triggered a known Helm patch ordering bug (duplicate 
..._REQUEST
entries with both 
value:0
and 
valueFrom
in the configmap patch list). Now every 
abctl local install
— with or without the flag — fails at ~23s before pods start. Error:
Copy code
unable to install helm: failed to create patch: The order in patch list:
[map[name:CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST value:0] ... map[name:CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST valueFrom:...]]
doesn't match $setElementOrder list: [...]
Secondary impact: The failed upgrade left MinIO's StatefulSet/Service/Pod gone — only the PVC remains (
Bound
). Server is throwing 
UnknownHostException: airbyte-minio-svc
, causing discover failures and broken schema refresh in the UI. Config DB is intact and backed up; sync data unaffected. What's been tried: • Re-running install with/without 
--low-resource-mode
— same error • Deleting the cron deployment — fails earlier at the configmap patch • 
--chart-version=1.7.2
— not viable, V1 chart is incompatible with Airbyte 2.1.0 Ask: • Is there a non-destructive way to resolve the configmap patch conflict on Helm chart V2 / Airbyte 2.1.0? • Can a successful install re-attach the existing MinIO PVC and restore the StatefulSet? Feel free to adjust the tone or trim further depending on where you're posting it. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai Resized the box and dropped 
--low-resource-mode
— the patch-order error is gone and the install now runs, with all pods healthy except MinIO. The problem: the MinIO statefulset is never created by the 2.1.0 chart, even though 
pv/airbyte-minio-pv
and 
pvc/airbyte-minio-pv-claim-airbyte-minio-0
are both Bound. Every install run warns: "Found MinIO physical volume. Consider migrating it to local storage." Only 
airbyte-db
statefulset exists; no 
airbyte-minio
statefulset at all. How do I get the chart to recreate the MinIO statefulset against the existing physical-volume PVC? (Want to avoid losing the PVC data if possible.)
k
I'm sorry, but the knowledge sources do not contain enough information to give you a confident, specific answer here. What the warning "Found MinIO physical volume. Consider migrating it to local storage" strongly suggests is that the Airbyte 2.1.0 / Helm chart V2 has intentionally moved away from MinIO as internal storage, and the chart no longer creates a MinIO StatefulSet by design — hence the migration hint. However, the knowledge sources do not document the details of this architectural change or provide a procedure for either: • Forcing the V2 chart to recreate the MinIO StatefulSet against an existing PVC, or • The recommended migration path from MinIO PV to the new local storage approach This is a gap in the available documentation I have access to. What I'd recommend 1. Check the Airbyte 2.1.0 release notes and Helm chart V2 changelog directly for any mention of MinIO deprecation or the local storage migration path — the warning message itself is a strong signal there's a documented migration procedure somewhere. 2. Raise this specific scenario in the Airbyte Slack with the summary we drafted earlier, updated to reflect your current state: ◦ Patch-order error resolved ✅ ◦ All pods healthy except MinIO ✅ ◦ MinIO StatefulSet not created by 2.1.0 chart (by design?) ◦ PV + PVC both 
Bound
, data intact ◦ Warning: "Found MinIO physical volume. Consider migrating it to local storage" 1. Do not delete the PVC until you have clarity on the migration path — since the PV/PVC are still 
Bound
, your data is safe for now. The migration warning is the key clue here and likely points to an intentional change in 2.1.0 that the Airbyte team will be best placed to clarify. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai Release notes flag 2.1.0 as experimental — is the recommended path to fix MinIO on 2.1, or move to 2.0? My config DB has already migrated to 2.1.0 and I can't lose my connections
k
This is an important question and the knowledge sources give partial but meaningful guidance. What the Release Notes Actually Say About 2.1.0 Being Experimental > This version of Airbyte is experimental. Airbyte changed the way we assemble new versions and haven't tested every feature and Helm chart configuration in version 2.1.0. If stability is your primary concern, you may elect to skip this version. [Airbyte 2.1 Release Notes] So skipping 2.1.0 is explicitly offered as an option — but only if you haven't already migrated. Your Specific Constraint: Config DB Already on 2.1.0 The knowledge sources do not document a downgrade path from 2.1.0 back to 2.0. Database migrations in Airbyte are typically one-way, and attempting to point a 2.1.0-migrated config DB at a 2.0 install would very likely corrupt or lose your connections. The knowledge sources do not contradict this — they simply don't address downgrading at all. Rolling back to 2.0 is almost certainly not safe at this point. The Practical Reality Recommendation Stay on 2.1.0 and pursue the MinIO/local storage migration path. The knowledge sources don't document that procedure, so your best next step is to contact Airbyte directly — the release notes themselves explicitly state: > If you still use Self-Managed Enterprise, do not upgrade. Speak to your Airbyte representative... [Airbyte 2.1 Release Notes] While that specific note is for Enterprise users, it underscores that Airbyte is aware 2.1.0 has rough edges and is actively supporting users through them. I'd update the Slack message we drafted with your current state and explicitly flag that your config DB has already migrated — that context is critical for whoever helps you. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
5 Views
Open in Slack
PreviousNext
Powered by