https://linen.dev logo
Join Slack
Powered by
Hello team :octavia-hello: I'm trying to run Strip...
# replication-troubleshooting
g
Hello team ๐Ÿ‘‹ I'm trying to run Stripe ingestions to BigQuery in 
Incremental | Deduped history
mode, but the state is never persisted to the DB, even once the connector is in success ๐Ÿ˜• The impact is that data syncs always ingest the whole history as if I was in 
Full Refresh
mode. Is someone facing the same issue? How do you think I could fix this? Thanks in advance for your help! ๐Ÿ˜ƒ๐Ÿ™ Airbyte version: 
0.40.22
Stripe source connector version: 
0.1.40
Stripe source connector lookback window: 
1 day
Stripe source connector Data request time increment: 
1 day
โœ… 1
m
What streams are you trying to sync using Stripe? Could you use a more recent data just to make sure the sync works properly?
g
Thanks for your help Marcos! I read the following streams: โ€ข balance_transactions โ€ข charges โ€ข disputes โ€ข transfers โ€ข refunds โ€ข payment_intents Each in a separate connection. The syncs work properly ๐Ÿ™‚ Also, Salesforce connections work in Incremental mode without any problem, hence my surprise ๐Ÿ˜ฎ
m
I ran using our test account and the state was saved correctly:
Copy code
[
  {
    "streamDescriptor": {
      "name": "disputes"
    },
    "streamState": {
      "created": 1610996305
    }
  },
  {
    "streamDescriptor": {
      "name": "transfers"
    },
    "streamState": {
      "created": 1610995824
    }
  },
  {
    "streamDescriptor": {
      "name": "balance_transactions"
    },
    "streamState": {
      "created": 1669080098
    }
  },
  {
    "streamDescriptor": {
      "name": "refunds"
    },
    "streamState": {
      "created": 1619595629
    }
  },
  {
    "streamDescriptor": {
      "name": "payment_intents"
    },
    "streamState": {
      "created": 1669072785
    }
  },
  {
    "streamDescriptor": {
      "name": "charges"
    },
    "streamState": {
      "created": 1669072785
    }
  }
]
Can you try to find the log line similar to:
Copy code
2022-11-22 18:33:05 INFO i.a.w.g.DefaultReplicationWorker(prepStateForLaterSaving):481 - State capture: Updated state to: Optional[io.airbyte.config.State@351ae747[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"disputes"},"stream_state":{"created":1610996305}}},{"type":"STREAM","stream":{"stream_descriptor":{"name":"transfers"},"stream_state":{"created":1610995824}}},{"type":"STREAM","stream":{"stream_descriptor":{"name":"balance_transactions"},"stream_state":{"created":1669080098}}},{"type":"STREAM","stream":{"stream_descriptor":{"name":"refunds"},"stream_state":{"created":1619595629}}},{"type":"STREAM","stream":{"stream_descriptor":{"name":"payment_intents"},"stream_state":{"created":1669072785}}},{"type":"STREAM","stream":{"stream_descriptor":{"name":"charges"},"stream_state":{"created":1669072785}}}]]]
g
Hello Marcos, I'm impressed by your reactivity, thanks a ton ๐Ÿ™ I can indeed find such a log for my 
disputes
connection:
Copy code
2022-11-23 17:07:29 replication-orchestrator > State capture: Updated state to: Optional[io.airbyte.config.State@60743cdb[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"disputes"},"stream_state":{"created":1667992678}}}]]]
However, the connection_id is 
e47abe67-9871-454a-a4ec-24c2e70f7f6c
And when I look at the state table, this connection_id is not mentioned (as we can see on the screenshot)
I tried to reproduce the bug on my airbyte installation (docker-compose up), and the state was properly saved. The bug appears only in my company environment. I found a variable whose value was different: 
USE_STREAM_CAPABLE_STATE
is true on my local env, and seems false in my company kubernetes environment. Do you know how I could modify it?
local env log:
Copy code
2022-11-23 16:23:00 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/119/0 --log-driver none --name destination-bigquery-write-119-0-daqpt --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/destination-bigquery:1.1.11 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.22 -e WORKER_JOB_ID=119 airbyte/destination-bigquery:1.1.11 write --config destination_config.json --catalog destination_catalog.json
company kubernetes env log:
Copy code
2022-11-23 17:23:40 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):50 - Using default value for environment variable USE_STREAM_CAPABLE_STATE: 'false'
m
Do you know how I could modify it? (edited)
Probably you must enable it
g
Do I just need to modify an environment variable in the helm chart? Or should it be managed somewhere else?
I'll try to set it explicitly in the environment variables, and have good hope that this will fix my issue ๐Ÿ™‚ I'll confirm it here when tested ๐Ÿ‘
๐Ÿ™ 1
I confirm setting 
USE_STREAM_CAPABLE_STATE
to true fixed my issue ๐ŸŽ‰๐Ÿฅณ
โœ… 1
I don't have the context, but am wondering why true is not the default value for this parameter?
I wasted several days pinpointing this issue, maybe it could save others time if we changed the default value, what's your take on this Marcos?
m
@Grember Yohan probably it is an on-going project from connector team and donโ€™t think right now it is the correct time to make it default
๐Ÿ‘ 1
u
Hello Grember Yohan, it's been a while without an update from us. Are you still having problems or did you find a solution?
g
Hello Marcos, no problem regarding states since we set 
USE_STREAM_CAPABLE_STATE
to true ๐Ÿ‘Œ
โœ… 1
3 Views
Open in Slack
PreviousNext
Powered by