https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • y
    airbyte-worker is taking too much memory, it just increases to 8 gb until it uses all
    k
    • 2
    • 11
  • o
    🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves
  • t
    Hello, when making any change in the connection, such as adding a table, or removing it, it always asks to restore the data, and then it does the sync. I don't want the sync to start, if I uncheck it so as not to reset the data, can it cause a problem?, or is there any other way?
    k
    • 2
    • 3
  • m
    good morning airbyte team! i am working on creating a custom connector for a firestore source. the code seems to work as expected locally, but when i try to build to the docker image i encounter the following error:
    Copy code
    #0 305.5 [output clipped, log limit 1MiB reached]
#0 305.5   Running setup.py clean for grpcio
#0 306.2 Successfully built source-google-firestore genson PyYAML pendulum
#0 306.2 Failed to build grpcio
------
ERROR: failed to solve: executor failed running [/bin/sh -c pip install --prefix=/install .]: exit code: 1
    trying to build with gradle results in
    Copy code
    * What went wrong:
Execution failed for task ':airbyte-commons:compileJava'.
> error: invalid source release: 17
    i have confirmed that i have openjdk-17 and python 3.11.9. has anyone else encountered these and fixed?
    k
    • 2
    • 8
  • k
    Hello, I am trying to move the data from postgres to databricks, but somehow the datatype in postgres is int8 but it is converted to int rather than a bigint when the table is created inside databricks. this is the log from sync,
    Copy code
    (type int8[19], nullable false) -> JsonSchemaType({type=number, airbyte_type=integer})
    k
    • 2
    • 2
  • r
    @kapa.ai Stream resets from MySQL-to-Snowflake connection keep failing with error 
    retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
    the run log only contains these three lines:
    Copy code
    2023-05-25 14:18:33 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):222 - Attempt 0 to get state
2023-05-25 14:18:33 INFO i.a.w.h.NormalizationInDestinationHelper(shouldNormalizeInDestination):52 - Requires Normalization: true Normalization Supported: false, Feature Flag Enabled: false
2023-05-25 14:18:33 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):222 - Attempt 0 to set attempt sync config
    how can i fix this? my pipeline is blocked until this is fixed
    k
    • 2
    • 2
  • m
    Hi, using the Facebook Marketing connector (Airbyte Cloud) I am trying to extract the 
    video_thruplay_watched_actions
    field. I created a custom insights report that contains this field, but the field does not show up in the field list of the stream and also not in the synced table. Any help is much appreciated
    k
    • 2
    • 2
  • i
    Hi! I have a question. I want to connect a Postgres database using CDC but my blocker today is that one table is to big to do a full refresh without impacting (around 100gb). Is there any way to address this synchronization without making a huge impact in the database? Or maybe, the full synchronization process is smart enough to do it using batch without making big impact. Is there any documentation you’d recommend me to start? Thanks !
    k
    • 2
    • 2
  • a
    Hi there - I have Airbyte deployed on an HVM EC2 t2.large and am trying to connect to snowflake - running into the following error 
    Could not find image: airbyte/destination-snowflake:1.0.2
    when trying to add snowflake as a destination. Does anyone have an idea why?
    k
    e
    • 3
    • 6
  • y
    postgres to postgres sync fails after 8 gb of data transfer giving error Failure Origin: replication, Message: Something went wrong during replication with no specific error in logs
    k
    • 2
    • 8
  • y
    what is airbyte message consumer ?
    k
    • 2
    • 2
  • y
    what is this error log means ? State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@c826b19[type=STATE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=io.airbyte.protocol.models.AirbyteStateMessage@714234c1[type=STREAM,stream=io.airbyte.protocol.models.AirbyteStreamStat
    k
    • 2
    • 8
  • j
    Another issue with the retry mechanism for the MySQL->Redshift connection. The second attempt is marked as successful but no data has been transferred nor did the transformation run at all. The first step failed during the (airbyte internal) normalization:
    Copy code
    0 Bytes|no records|no records|Job id: 289|11s
  • j
    Is it maybe possible to disable reruns alltogether as a workaround?
    k
    • 2
    • 2
  • s
    This message was deleted.
    k
    • 2
    • 2
  • y
    I am getting error as what is this error ? I am using cursor as postgres's 
    timestamp without timezone
    column but getting error as WARN i.a.w.i.FieldSelector(lambda$reportMetrics$1):124 - *Schema validation error*s found for stream abc. Error messages: [$.etl_updated_at: 2023-05-24T144608.923456 is an invalid date-time
    k
    • 2
    • 8
  • j
    Hi everyone! Does anyone know why sales_report is the only table that can be chosen to load from Appstore Connect? No option for subscription, subscription event or subscriber that is mentioned in the documentation here: https://airbyte.com/connectors/app-store
    f
    • 2
    • 2
  • y
    automatically this error comes
    Copy code
    emporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
    k
    • 2
    • 2
  • y
    durring sync this is called TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation... and sync stops
    k
    • 2
    • 2
  • s
    hi everyone, I want to import csv and xml into woocommerce. There is a woocommerce source connector but no destination connector. Is there still a way to get those products from csv/xml into woocommerce? Also, I'd like to generate xml exports from woocommerce orders. Can airbyte generate xml files? Thanks!
    k
    • 2
    • 2
  • c
    Hi Everyone, I am trying to configure QuickBooks source in Airbyte (open source) using OAuth. The first time I enter the access token, refresh token, realm ID it is getting connected. I configured a snowflake destination and it synced. When I try to modify the streams and resync after sometime, it is throwing 401 error. Even the source also threw error after sometime. Is this a known issue. If so when it will be fixed? Thanks
    k
    b
    • 3
    • 5
  • b
    [Airbyte's Facebook Marketing GA Connection -> Still Broken]
     I've reported that the Facebook Marketing integration still broken, and have already onboarded about 14 customers using Airbyte's API, but still can't fetch the data. The problem for what I've read on the logs is on fetching the AdAccount details, because of some missing fields on the GraphQL API. I need an urgent response if this is going to be fixed or should look for other platform like Meltano. I've already been in contact through Intercom for 2 weeks, and nothing seems to be fixed. I've already tested with my personal Facebook accounts, and also my customer's facebook accounts. Same problem.
    Copy code
    Additional Failure Information: Message: Call was not successful Method: GET Path: <https://graph.facebook.com/v16.0/act_XXXXXXXX/> Params: {'fields': 'account_id,account_status,age,amount_spent,balance,business,business_city,business_country_code,business_name,business_state,business_street,business_street2,business_zip,can_create_brand_lift_study,capabilities,created_time,currency,disable_reason,end_advertiser,end_advertiser_name,extended_credit_invoice_group,failed_delivery_checks,fb_entity,funding_source,funding_source_details,has_advertiser_opted_in_odax,has_migrated_permissions,id,io_number,is_attribution_spec_system_default,is_direct_deals_enabled,is_in_3ds_authorization_enabled_market,is_notifications_enabled,is_personal,is_prepay_account,is_tax_id_required,line_numbers,media_agency,min_campaign_group_spend_cap,min_daily_budget,name,offsite_pixels_tos_accepted,owner,partner,rf_spec,spend_cap,tax_id,tax_id_status,tax_id_type,timezone_id,timezone_name,timezone_offset_hours_utc,tos_accepted,user_tasks,user_tos_accepted'} Status: 400 Response: { "error": { "message": "Unsupported request - method type: get", "type": "GraphMethodException", "code": 100, "fbtrace_id": "AsoqBO60K0RuFsalpbUZtuB" } }
    k
    • 2
    • 2
  • o
    🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!
  • b
    I’m getting my syncs sometimes hanging infinitely, especially with connection resets. They almost always hang at:
    Copy code
    INFO i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$1):166 Executing finalization of tables.
    I’m running Airbyte on a Kubernetes cluster using the airbytehq helm chart. I’m currently seeing this issue on a connection with a salesforce source and redshift destination
    k
    • 2
    • 2
  • j
    hi all - For Databricks destination type setup with Azure Blog Storage. Is there any way I can specify a path within my azure container? Right now i can only specify Azure Storage account Name and Container Name. I would like to specify a specific path within the container
    k
    • 2
    • 2
  • y
    why TemporalAttemptExecution is attempting cancellation suddenly ?
    k
    • 2
    • 2
  • y
    what is this error WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation
    k
    u
    • 3
    • 2
  • c
    Hi, Every one Can i use locally minio s3 as destination? And how do i set S3 bucket region on airbyte?
    k
    • 2
    • 2
  • q
    I have existing connection on airbyte that sync data from Postgres to Bigquery at intervals of 30min using deduped/Incremental sync mode. Everything had been working for all these days, however I just noticed that few of synchronization today, is doing full refresh, in spite of the fact all tables are set for deduped /Incremental mode. Please, appreciate any insights on where to look for resolution. Thanks
    a
    • 2
    • 2
  • a
    Hi Guys, I'm having an issue when I connect any source to bigquery as destination Failure Origin: normalization, Message: Normalization failed during the dbt run. This may indicate a problem with the data itself. Pickling client objects is explicitly not supported. Clients have non-trivial state that is local and unpickleable.
    k
    • 2
    • 2
1...196197198...245Latest