<@U04S8PLJWMT> why cron service won't start on kub...
# replication-ask-ai
d
@kapa.ai why cron service won't start on kubenretes - No bean of type [io.airbyte.config.secrets.persistence.SecretPersistence] present within context │ │ airbyte-cron-9c6df6cd4-w7d6x * [$LocalTestingSecretPersistence$Definition$Intercepted] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present │ │ airbyte-cron-9c6df6cd4-w7d6x * [AwsSecretManagerPersistence] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present │ │ airbyte-cron-9c6df6cd4-w7d6x * [GoogleSecretManagerPersistence] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present │ │ airbyte-cron-9c6df6cd4-w7d6x * [NoOpSecretPersistence] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present │ │ airbyte-cron-9c6df6cd4-w7d6x * [VaultSecretPersistence] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present │ │ airbyte-cron-9c6df6cd4-w7d6x * [AzureKeyVaultPersistence] a candidate of [SecretPersistence] is disabled because: │ │ airbyte-cron-9c6df6cd4-w7d6x - Required property [airbyte.secret.persistence] not present
k
The knowledge sources don’t describe this exact
airbyte.secret.persistence
error for the cron service, but they do show closely related issues and how they were fixed. From the logs you pasted, Micronaut is trying to create a
SecretPersistence
bean, but all candidates are disabled because the required property
airbyte.secret.persistence
is missing. That means the cron pod is starting with a configuration where the secret‑persistence type is not set. Relevant points from the docs and issues: - Airbyte supports multiple secret persistence backends (local DB, AWS Secrets Manager, Google Secret Manager, Vault, Azure Key Vault). These are controlled by configuration flags / values. [[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte)] - When using external secret managers, you must explicitly configure the secret manager type and its settings in
values.yaml
(Helm) or environment variables; otherwise the corresponding beans are disabled. [[Secret mgmt values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)] - A very similar Micronaut “No bean of type SecretPersistence” error on worker startup was resolved by adding the missing secret‑manager configuration in
values.yaml
. [[GitHub issue 60915](https://github.com/airbytehq/airbyte/issues/60915)] Given that, the cron service is failing because no secret‑persistence implementation is enabled in its environment. To fix it, you need to define the secret‑persistence configuration consistently for all services (including cron): 1. If you are using the Helm chart (recommended) In your
values.yaml
, configure
global.secretsManager
(or
secretsManager
for older chart layouts) according to the backend you want, for example AWS Secrets Manager: [[Secret mgmt values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)]
Copy code
yaml    global:      secretsManager:        type: awsSecretManager          # or AWS_SECRET_MANAGER depending on chart version        secretName: "airbyte-config-secrets"        awsSecretManager:          region: <aws-region>          authenticationType: credentials  # or instanceProfile
Or Google Secret Manager:
Copy code
yaml    global:      secretsManager:        type: googleSecretManager        secretName: "airbyte-config-secrets"        googleSecretManager:          projectId: <project-id>          region: ""              # optional          credentialsSecretKey: gcp.json
Make sure the corresponding Kubernetes secrets (
airbyte-config-secrets
,
gcp-cred-secrets
, etc.) are created as described in the docs. [[Configure K8s secrets](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configure-kubernetes-secrets)] 2. If you intend to use the default (Kubernetes secrets only, no external manager) Ensure that the configuration does not partially enable an external secret manager. If you previously set any secret‑manager–related values, remove or fully disable them so that Airbyte falls back to its default behavior of using Kubernetes secrets. [[External connector secret mgmt](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] 3. Apply the updated configuration and restart - Update your Helm values and run
helm upgrade
so the cron deployment gets the same secret‑persistence configuration as the other services. - Verify that the cron pod now starts without the
No bean of type SecretPersistence
error. If you can share how you deployed Airbyte (Helm chart version and your
values.yaml
secrets/secretManager section), I can help pinpoint the exact change needed.
d
Yes, @kapa.ai nothing changes actually. secretsManager: enabled: true type: AWS_SECRET_MANAGER awsSecretManager: region: ${AWS_REGION} authenticationType: instanceProfile tags: - key: AirbyteManaged value: "true" - key: Application value: "Airbyte" - key: "Owner" value: "Airbyte" It was and worked before. Same nodes,same pods.
k
A message from kapa.ai
The knowledge sources don’t mention this exact
airbyte.secret.persistence
error on the cron service, so I can’t be fully certain, but they do show how secret persistence is wired and what must be set for the beans to be enabled. From your log: > all candidates of
SecretPersistence
are disabled because: >
Required property [airbyte.secret.persistence] not present
That means Micronaut doesn’t see any secret‑persistence type configured in the cron pod’s environment. From the docs: - Secret persistence is controlled by a config flag (
SECRET_PERSISTENCE
) and/or the Helm
secretsManager
block. [[Configuring Airbyte](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-airbyte)] - For Helm, you must configure
secretsManager
/
global.secretsManager
with a type such as
AWS_SECRET_MANAGER
,
awsSecretManager
, etc., and the chart will translate that into the internal secret‑persistence configuration. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values); [External connector secrets](https://docs.airbyte.com/platform/2.0/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] Your snippet:
Copy code
yaml secretsManager:   enabled: true   type: AWS_SECRET_MANAGER   awsSecretManager:     region: ${AWS_REGION}     authenticationType: instanceProfile     tags:       - key: AirbyteManaged         value: "true"       - key: Application         value: "Airbyte"       - key: "Owner"         value: "Airbyte"
Two important nuances from the docs: 1. Chart layout / key path matters In the 2.x docs and Helm v2 examples, the block is under
global.secretsManager
, not top‑level `secretsManager`:
Copy code
yaml    global:      secretsManager:        enabled: true        type: AWS_SECRET_MANAGER        secretName: "airbyte-config-secrets"        awsSecretManager:          region: <aws-region>          authenticationType: credentials
[[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] If your chart version expects
global.secretsManager
but you only have
secretsManager
, some pods (like cron) may not get the env vars that set
airbyte.secret.persistence
/
SECRET_PERSISTENCE
, even if other pods did before. 2. The required K8s secret must exist For AWS, the docs require a Kubernetes secret (by default
airbyte-config-secrets
) with the AWS access key and secret key, unless you use
instanceProfile
. [[External connector secrets](https://docs.airbyte.com/platform/2.0/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] If anything changed around that secret name or content, the chart might not be enabling the AWS secret persistence bean. Given that it “worked before” and now cron specifically complains: Concrete checks to do: 1. Confirm the correct key path for your Helm chart version - If you’re on Helm chart v2 / Airbyte 2.x, ensure the config is under
global.secretsManager
exactly as in the docs, not only under
secretsManager
. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] - If you recently upgraded chart or Airbyte version, compare your
values.yaml
with the current example in the docs and align the structure. 2. Verify that cron actually sees the env vars -
kubectl describe pod <cron-pod> -n <namespace>
and look for env vars related to secrets (e.g.
SECRET_PERSISTENCE
,
AWS_*
, or any `AIRBYTE_`/`airbyte.secret.persistence`). - If they’re missing on cron but present on other pods, it’s almost certainly a Helm values / key‑path issue. 3. Ensure
enabled: true
and type are consistent
- In the 2.0 docs,
enabled
is shown under
global.secretsManager
. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] - Make sure you don’t have another
global.secretsManager
block elsewhere in
values.yaml
that overrides this one for some components. Because the docs don’t show the internal mapping from
type: AWS_SECRET_MANAGER
to
airbyte.secret.persistence
, I can’t point to the exact env var name, but the error clearly indicates that whatever mapping the chart normally does is not happening for the cron pod. If you can paste: - Your full
values.yaml
secrets‑related section (including whether it’s under
global:
), and - Your Helm chart version / Airbyte version, I can map it directly against the doc examples and tell you the minimal change needed.
d
In v2 helm chart we are using
AWS_SECRET_MANAGER
based on docs on how to migrate to v2 @kapa.ai https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community
k
A message from kapa.ai
You’re right to use
AWS_SECRET_MANAGER
with the v2 Helm chart—that’s what the migration docs show. The issue is likely where and how it’s configured in
values.yaml
, not the value itself. From the v2 docs: 1. The migration table shows that the old env var
SECRET_PERSISTENCE
maps to
global.secretsManager.type
in Helm v2. [[Chart v2 mapping](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] 2. The secret‑manager configuration for v2 is expected under
global.secretsManager
, not a top‑level
secretsManager
key. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] Example from the docs (AWS Secrets Manager):
Copy code
yaml    global:      secretsManager:        type: AWS_SECRET_MANAGER        secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.        awsSecretManager:          region: <aws-region>          authenticationType: credentials  # or instanceProfile          tags:            - key: team              value: deployments
Given your snippet:
Copy code
yaml secretsManager:   enabled: true   type: AWS_SECRET_MANAGER   awsSecretManager:     region: ${AWS_REGION}     authenticationType: instanceProfile     tags:       - key: AirbyteManaged         value: "true"       - key: Application         value: "Airbyte"       - key: "Owner"         value: "Airbyte"
this is missing the
global:
prefix
. With Helm v2, that means the chart will not populate the internal config that sets
SECRET_PERSISTENCE
/
airbyte.secret.persistence
, so Micronaut disables all
SecretPersistence
beans and your cron pod fails. ### What to change Move your block under
global.secretsManager
and include `secretName`:
Copy code
yaml global:   secretsManager:     enabled: true     type: AWS_SECRET_MANAGER     secretName: "airbyte-config-secrets"   # or your existing secret name     awsSecretManager:       region: ${AWS_REGION}       authenticationType: instanceProfile       tags:         - key: AirbyteManaged           value: "true"         - key: Application           value: "Airbyte"         - key: "Owner"           value: "Airbyte"
This matches the v2 docs and the env‑var mapping table. [[Chart v2 mapping](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file); [Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] Then: 1.
helm upgrade
(or your
abctl
command) with the updated
values.yaml
. 2. Check the cron pod: it should now start without the
No bean of type SecretPersistence
/
airbyte.secret.persistence
error. If it still fails, the next thing to inspect is the cron pod’s env vars (
kubectl describe pod ...
) to confirm that the chart is now setting the secret‑persistence configuration.
d
We are not missing @kapa.ai, our structure is global.secretsManager.enabled: true and following other params like auth trough InstanceProfile.
k
A message from kapa.ai
The knowledge sources don’t mention this exact
airbyte.secret.persistence
error on the cron service, so I can’t be fully certain, but they do show how secret persistence is supposed to be wired in Helm v2. Key points from the docs: - In Helm v2, external secret management is configured via
global.secretsManager
. The chart then sets the internal secret‑persistence configuration (what Micronaut sees). [[Chart v2 upgrade](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file); [Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] - Valid
type
values for v2 are:
VAULT
,
GOOGLE_SECRET_MANAGER
,
AWS_SECRET_MANAGER
,
AZURE_KEY_VAULT
,
TESTING_CONFIG_DB_TABLE
. [[Chart v2 upgrade](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] - Example AWS config (v2 docs):
Copy code
yaml   global:     secretsManager:       enabled: false       type: "" # one of: VAULT, GOOGLE_SECRET_MANAGER, AWS_SECRET_MANAGER, AZURE_KEY_VAULT, TESTING_CONFIG_DB_TABLE       secretName: "airbyte-config-secrets"        awsSecretManager:         region: <aws-region>         authenticationType: credentials ## Use "credentials" or "instanceProfile"         tags:           - key: team             value: deployments           - key: business-unit             value: engineering         kms: ## Optional - ARN for KMS Decryption.
[[Chart v2 upgrade](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file); [Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] You’ve confirmed: -
global.secretsManager.enabled: true
-
global.secretsManager.type: AWS_SECRET_MANAGER
-
global.secretsManager.awsSecretManager.authenticationType: instanceProfile
Given that, the cron error: > Required property [airbyte.secret.persistence] not present means that, for the cron pod specifically, the chart is still not producing the Micronaut property it expects (internally mapped from
global.secretsManager
). From the docs and related issues, the only concrete, documented levers you have are: 1. Ensure the full block matches the v2 schema Double‑check that your block includes at least:
Copy code
yaml    global:      secretsManager:        enabled: true        type: AWS_SECRET_MANAGER        secretName: "airbyte-config-secrets"   # or your secret name        awsSecretManager:          region: <aws-region>          authenticationType: instanceProfile
secretName
is required in all AWS examples in the docs. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values); [Enterprise impl guide](https://docs.airbyte.com/platform/2.0/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] 2. Verify that cron actually receives the env/config The docs don’t show the exact env var name for
airbyte.secret.persistence
, but they do show that
SECRET_PERSISTENCE
is the internal knob for secret persistence. [[Configuring Airbyte](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-airbyte)] To confirm the chart is wiring things correctly for cron: - Run
kubectl describe pod <cron-pod> -n <namespace>
and inspect: - Does it have any env var like
SECRET_PERSISTENCE
or similar? - Does it have the AWS region / IAM‑related env vars you expect? If other pods (e.g.
server
,
worker
) have those env vars but
cron
does not, then the issue is specifically that the Helm chart is not propagating
global.secretsManager
to cron in your deployment. 3. Cross‑check chart version vs docs The v2 docs and examples you’re following are for Helm chart v2 (Core). [[Chart v2 upgrade](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] If your chart version is older/newer than what the docs target, there might be a mismatch in how
global.secretsManager
is consumed by the cron deployment. The knowledge sources don’t provide more detail here, so I can’t say more without guessing. Because the docs don’t expose the internal mapping from
global.secretsManager
to
airbyte.secret.persistence
, I can’t definitively tell you why cron alone is missing that property. The safest, doc‑aligned steps are: - Ensure your
global.secretsManager
block exactly matches the v2 examples (including
secretName
). - Confirm via
kubectl describe
that cron gets the same secret‑related env/config as other pods. - If it still fails and other pods work, this may be a chart bug or regression not covered in the docs; in that case, you’d likely need to open an issue with your full
values.yaml
and chart version, similar to the existing AWS Secrets Manager issue. [[AWS SM issue](https://github.com/airbytehq/airbyte/issues/67588)] If you can share the full
global.secretsManager
section and your exact chart version, I can compare it line‑by‑line against the documented schema and call out any concrete mismatches.
d
I can see config works fine, inside of the container @kapa.ai I can see, variables like SECRET_PERSISTENCE=AWS_SECRET_MANAGER
k
A message from kapa.ai
You’re right: if
SECRET_PERSISTENCE=AWS_SECRET_MANAGER
is present inside the cron container, then the basic mapping from Helm → env var is working. The error is still: > Required property
[airbyte.secret.persistence]
not present The knowledge sources don’t document this
airbyte.secret.persistence
property name explicitly, so I can’t say with certainty how it’s derived internally. They only expose the public knobs: -
SECRET_PERSISTENCE
env var / config key. [[Configuring Airbyte](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-airbyte)] -
global.secretsManager.type
in Helm v2, which maps from the old
SECRET_PERSISTENCE
. [[Chart v2 mapping](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] Given what we do know from the docs and your info: 1. Cron has
SECRET_PERSISTENCE=AWS_SECRET_MANAGER
So the high‑level secret persistence type is set correctly. 2. Micronaut still says
airbyte.secret.persistence
is missing
That suggests the internal Micronaut property it expects (
airbyte.secret.persistence
) is not being set from
SECRET_PERSISTENCE
in this service, even though the env var is present. 3. All SecretPersistence beans are disabled The log shows every candidate (
AwsSecretManagerPersistence
,
GoogleSecretManagerPersistence
,
VaultSecretPersistence
, etc.) is disabled because
airbyte.secret.persistence
is missing, not because AWS‑specific config is wrong. From the docs, the only supported, documented way to influence this is: - Set
SECRET_PERSISTENCE
(which you have). [[Configuring Airbyte](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-airbyte)] - Or, in Helm v2, set
global.secretsManager.type
to
AWS_SECRET_MANAGER
and configure the AWS block. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] Since both appear correct and the env var is visible in the container, but Micronaut still doesn’t see
airbyte.secret.persistence
, this looks like behavior that is not covered in the docs and may be a bug or regression in the cron service wiring for your chart/version. Given the limits of the knowledge sources, the most I can recommend without guessing is: - Capture and share (in an issue or support channel): - Your Helm chart version and Airbyte version. - The full
global.secretsManager
section from
values.yaml
. - The full cron pod env (
kubectl describe pod <cron-pod>
). - Reference that the cron pod has
SECRET_PERSISTENCE=AWS_SECRET_MANAGER
but still logs
Required property [airbyte.secret.persistence] not present
. There is a related open issue where AWS Secrets Manager config stopped being honored after upgrading to chart v2, which suggests there may indeed be chart‑level problems around secret persistence in v2. [[AWS SM error issue](https://github.com/airbytehq/airbyte/issues/67588)] The docs don’t provide enough detail on the internal
airbyte.secret.persistence
mapping to give a precise fix beyond what you’ve already done.
d
What I don't see in
env
that on how we authorise aws.
@kapa.ai I don't see authenticationType on it. Seems like not complete environment variables.
k
A message from kapa.ai
You’re right to focus on the env vars: if
SECRET_PERSISTENCE
is present but AWS‑specific settings (like
authenticationType
) are missing or incomplete, the AWS Secret Manager persistence bean may still not initialize correctly. From the Helm v2 docs, when using AWS Secrets Manager you must configure all of these under
global.secretsManager
in
values.yaml
(chart v2): [[Chart v2 upgrade](https://docs.airbyte.com/platform/2.0/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file); [Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)]
Copy code
yaml global:   secretsManager:     enabled: true     type: AWS_SECRET_MANAGER     secretName: "airbyte-config-secrets"   # required     awsSecretManager:       region: <aws-region>                 # required       authenticationType: instanceProfile  # or "credentials"       tags:         - key: AirbyteManaged           value: "true"         - key: Application           value: "Airbyte"         - key: Owner           value: "Airbyte"       # kms: <optional-kms-arn>
The chart then turns this into the internal configuration (including
SECRET_PERSISTENCE
and AWS‑specific properties) for all services, cron included. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] Given your observation that
authenticationType
is not visible in the cron container: 1. Confirm the full block is under
global.secretsManager
exactly as above
, including: -
enabled: true
-
type: AWS_SECRET_MANAGER
-
secretName: ...
-
awsSecretManager.region
-
awsSecretManager.authenticationType
2. Check whether the cron pod gets the same config as other pods: - Run
kubectl describe pod <cron-pod> -n <ns>
and compare its env with a working pod (e.g.
server
or
worker
). - If
SECRET_PERSISTENCE
is there but AWS‑specific env/config is missing only on cron, that indicates the chart is not propagating the full
global.secretsManager
config to cron. The docs don’t show the exact env var names for
authenticationType
or how they map into Micronaut properties, so I can’t tell you precisely which env var should appear; they only document the
values.yaml
schema and that
awsSecretManager.authenticationType
is required. [[Secret mgmt values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/secrets#values)] If: -
global.secretsManager
in
values.yaml
matches the doc examples, and - other pods work with AWS Secrets Manager, but - cron still shows incomplete secret‑manager config and logs
Required property [airbyte.secret.persistence] not present
, then this is likely behavior not covered by the docs (possibly a chart or service wiring issue). In that case, the next step is to open an issue with: - Your chart + Airbyte versions, - The full
global.secretsManager
block, - Env from a working pod vs the cron pod, similar to the existing AWS Secrets Manager issue. [[AWS SM error issue](https://github.com/airbytehq/airbyte/issues/67588)]