https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I get this erros when trying to cle...
# replication-ask-ai
i
@kapa.ai I get this erros when trying to clear the data, the clear data operation does not even start 
2026-07-16 07:56:17,456 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 6]	WARN	i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=cca3b0c1-a8f7-3586-9d42-40b19e029f98, activityType=CheckProgress, attempt=4
java.lang.NullPointerException: null
at io.airbyte.workers.temporal.scheduling.activities.CheckRunProgressActivityImpl.checkProgress(CheckRunProgressActivityImpl.kt:21)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
This error is coming from the worker pod
k
I'm sorry, but the knowledge sources do not contain enough information to provide a confident answer about this specific error. The error you're seeing is:
Copy code
java.lang.NullPointerException: null
    at io.airbyte.workers.temporal.scheduling.activities.CheckRunProgressActivityImpl.checkProgress(CheckRunProgressActivityImpl.kt:21)
This is a 
NullPointerException
occurring in the 
CheckRunProgressActivity
during a Clear operation, on attempt 4 of the Temporal activity. This is different from the errors documented in the available knowledge sources. What I can tell you based on general context from the knowledge sources: 1. Similar NPE patterns have appeared in older Airbyte versions (0.39.x–0.40.x) and were often resolved by upgrading Airbyte and/or its components. [Forum Post] 2. For failed clear operations, the Airbyte documentation recommends manually dropping the tables in the destination so that Airbyte can continue syncing accurately. [Clearing Data] 3. One user resolved a similar "clear data not completing" issue by upgrading the postgres connector version to latest and allocating more worker pods. [GitHub Issue] Suggested next steps: • Check your Airbyte version and ensure all components (including worker pods) are up to date. • Review the full worker pod logs for more context around what 
checkProgress
is trying to access when it hits the null value. • If the clear continues to fail, consider manually dropping the affected destination tables as a workaround. • If you're on a self-managed deployment, consider opening a GitHub issue or reaching out to Airbyte support with the full logs for a more targeted diagnosis. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
i
@kapa.ai another question, Im interested in using azure storage account instaed of minio for airbyte, what are the way I can offer it credentisl using the values file ?
k
Based on the knowledge sources, when configuring Azure Blob Storage in your 
values.yaml
instead of MinIO, you have two ways to provide credentials: Option 1: Reference a Kubernetes Secret (recommended) Store your Azure connection string in a Kubernetes secret and reference it by key: 1. Create the Kubernetes secret:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  azure-blob-store-connection-string: ## DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...==;EndpointSuffix=core.windows.net
2. Reference it in `values.yaml`:
Copy code
global:
  storage:
    type: "Azure"
    secretName: airbyte-config-secrets
    bucket:
      log: airbyte-container
      state: airbyte-container
      workloadOutput: airbyte-container
    azure:
      connectionStringSecretKey: azure-blob-store-connection-string
[Storage Docs] Option 2: Inline connection string directly in values.yaml You can also provide the connection string directly (without a separate Kubernetes secret):
Copy code
global:
  storage:
    secretName: ""
    type: azure
    bucket:
      log: airbyte-container
      state: airbyte-container
      workloadOutput: airbyte-container
      activityPayload: airbyte-container
    azure:
      connectionString: <azure storage connection string>
[External Logging] Key notes: • Use 
connectionStringSecretKey
 when your connection string is stored in a Kubernetes secret (more secure). • Use 
connectionString
 to inline the value directly in 
values.yaml
(less secure). • Adding external storage details disables the default internal MinIO instance (
airbyte/minio
). • Airbyte recommends using a single bucket/container across all three values (
log
, 
state
, 
workloadOutput
). [Storage Docs] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
i
@kapa.ai What If I can't use connection string keys and can only use azure credentials like a service principal
k
I'm sorry, but the knowledge sources do not contain enough information to confirm whether the 
values.yaml
storage configuration supports service principal credentials (Tenant ID, Client ID, Client Secret) as a direct alternative to a connection string for the state/logging storage integration. The knowledge sources only document the connection string approach for Azure Blob Storage in the 
values.yaml
configuration — either inlined or referenced via a Kubernetes secret key. [Storage Docs] What I can suggest: • The Azure Blob Storage connector (source/destination) does support service principal / Entra ID authentication, but that is separate from the platform-level storage configuration. [Destination Docs] • For the platform storage configuration, I'd recommend: a. Checking the Airbyte GitHub repository for the latest 
values.yaml
schema to see if service principal fields are available. b. Reaching out to Airbyte support or opening a GitHub issue to request this capability if it's not yet supported. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
i
@kapa.ai I'm wondering if a azure blob storage connector might get duplicated data from a ussual, non-failed syncs?
k
Yes, duplicates can occur in normal (non-failed) syncs with the Azure Blob Storage destination, depending on the sync mode you use. Why duplicates can happen 1. Inclusive cursor behavior: In 
Incremental | Append
mode, Airbyte reads records where 
cursor >= last_synced_cursor_value
(not strictly greater than). This means records sitting exactly on the cursor boundary get re-read on the next sync and appended again. [Inclusive Cursors] 2. Cursor doesn't advance: If your cursor column has the same value across many rows (e.g. a 
dateupdated
that never changes), the cursor never moves forward and every sync re-reads and re-appends all those rows. [GitHub Discussion] 3. Full Refresh Append: This mode always appends the full dataset on every sync, so duplicates accumulate naturally over time. [Sync Modes] Important limitation for Azure Blob Storage The Azure Blob Storage destination does not support deduped sync modes (
Incremental Append + Deduped
and 
Full Refresh Overwrite + Deduped
are both unsupported). [Azure Blob Storage Docs] This means you cannot rely on Airbyte itself to deduplicate the data at the destination level. Your options to handle duplicates are: • Pick a cursor column that reliably advances on every row update (e.g. a real 
updated_at
timestamp or a monotonically increasing ID). • Deduplicate downstream in your processing layer after loading from Azure Blob Storage (e.g. using a window function like 
ROW_NUMBER() OVER (PARTITION BY <pk> ORDER BY <cursor> DESC) = 1
). [GitHub Discussion] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by