After upgrading a docker compose setup to abctl and upgrading the BigQuery destination we are seeing this error breaking all our connections:

Caused by: io.temporal.failure.ApplicationFailure: message='Server error : 500 Internal Server Error {"message":"Internal Server Error","logref":null,"path":null,"_links":{"self":{"href":"/api/v1/workload/create","templated":false,"profile":null,"deprecation":null,"title":null,"hreflang":null,"type":null,"name":null}},"_embedded":{"errors":[{"message":"Internal Server Error: Error executing PERSIST: ERROR: could not read block 8 in file \"base/16385/17447\": read only 0 of 8192 bytes","logref":null,"path":null,"_links":{},"_embedded":{}}]}}', type='io.airbyte.workload.api.client.generated.infrastructure.ServerException', nonRetryable=false

Would be happy if anyone had any idea what is causing this or how to troubleshoot. I haven't found anyone with a similar problem on Github issues.

#44790 An error occurred: Unexpected status code 401 from token endpoint New discussion created by muhammadhrazzaq I am trying to create the source by using apis. I have hosted the airbyte on my azure vm using abctl which works fine. when I try to create source it give me 401 error , I am using the same token to get list of the airbyte sources it works fine for that endpoint but it is throwing me error when I try to create source .Can anyone please help me with this

An error occurred: Unexpected status code 401 from token endpoint

airbytehq/airbyte

Hi, I'm building a source connector for the Bill API using Airbyte's low-code CDK. I'm trying to implement pagination. This is what the payload of the request to the API looks like:

payload = {
    "data": "{"start":0,"max":999}",
    "devKey": "xx",
    "sessionId": "xx"
}

The "start" field represents the index of the first record that should be returned, and the "max" field is the maximum number of records that can be returned in the response. The highest value for this is 999. I need to implement pagination, and I need the "start" field to be incremented by 999 to request a new page of records for every subsequent request until there are no more records left to be returned by the API. MY ISSUE: How exactly can I increment the "start" field by 999, since it is nested within the payload? On top of this, the payload is sent to the API in form encoded format, and I'm not sure how to configure the YAML file in order to alter the nested "start" field for implementing the offset. Please help. Here's the link to the API docs and the endpoint in question: https://developer.bill.com/reference/ap-vendormgmt-listvendor BILL API documentation List vendors List vendor objects. See Search operations with lists to learn about pagination, sorting, and filtering. Vendor address details are required for paying the vendor by check or for inviting the vendor to join the BILL network. Keep this information current at all times. nameOnCheck address1 addressCit...

#44793 Issue with pagination New discussion created by KrishivGubba Hi, I'm building a source connector for the Bill API using Airbyte's low-code CDK. I'm trying to implement pagination. This is what the payload of the request to the API looks like: payload = { "data": "{"start":0,"max":999}", "devKey": "xx", "sessionId": "xx" } The "start" field represents the index of the first record that should be returned, and the "max" field is the maximum number of records that can be returned in the response. The highest value for this is 999. I need to implement pagination, and I need the "start" field to be incremented by 999 to request a new page of records for every subsequent request until there are no more records left to be returned by the API. MY ISSUE: How exactly can I increment the "start" field by 999, since it is nested within the payload? On top of this, the payload is sent to the API in form encoded format, and I'm not sure how to configure the YAML file in order to alter the nested "start" field for implementing the offset. Please help. Here's the link to the API docs and the endpoint in question: https://developer.bill.com/reference/ap-vendormgmt-listvendor airbytehq/airbyte

Hi team. I am building an application which uses facebook-marketing source. I am using custom insights for fetching ads_insights and campaign_insights. MY ISSUE: I am unable to retrieve complete info from the actions and action_values field in the json. action_values is null for all the records. Although those metrics are visible if I use the default ads_insights table of airbyte. This is crucial for my developement. I am stuck the final stage for my product. Can you please let me know the issue? Also is this the same issue as this one? - https://developers.facebook.com/community/threads/349898201041765/

Hello! I've recently setup a GCP VM and installed Airbyte using the quickstart quide. (Just testing things out for now) I'm having trouble understanding what needs to be done to add GCP Secret Manager as the external secret manager. I found this document but I'm not really sure what needs to be done here. Are the two examples two different files that need to be installed with the abctl commands? If anyone can explain this a bit more clearly or can point me to more documentation that would be appreciated.

Anyone else suddenly having issues with connection checks/syncs failing with Mailchimp? We noticed this started a couple days ago, and the errors vacillate between "Internal Server Error" (assuming HTTP 500) and "Too Many Requests" (assuming HTTP 429). Oddly we're seeing this happen across clients with Airbyte (which are authed by separate users, so don't share quota), but the same requests work fine through Postman (or even

curl

from the same IP).

Hey all.. has anyone tried triggering airbyte data sync from airflow. When I used the docker compose up method for installing airbyte then the sync worked because airflow was also set up using docker compose up. Now that I have started using abctl which is managed by kubernetes, my airflow container is unable to network with airbyte. I request for any guidance here, some directions that I can look into to finding a solution to this.

Hello 👋🏻 I'm trying to create my own connector but I don't get the stream idea. My YAML looks like this : ------------------------- version: '1.0' sync: streams: - name: test sync_mode: full_refresh destination_sync_mode: overwrite frequency: daily source: name: AS400 DB2 connector: airbyte/source-custom-db2 configuration: ssl: false schema: acm jdbc_url: jdbc:as400://111.22.333.44;librairies=acm password: MyUserPassword username: MyUser driver_path: /tmp/airbyte_local/jt400.jar driver_class: com.ibm.as400.access.AS400JDBCDriver ------------------------- But the error I have is : Error handling request: A valid manifest should have at least one stream defined. Got None I don't understand how to correctly create the stream

Hi Guys, I have setup Airbyte on Kubernetes. I am doing postgres to postgres data ingestion. I am using CDC for data ingestion. I am concerned with the amount of time it is taking. It runs every hour The time it took for replication is 30mins for below mentioned number of records 9776 record(s) extracted / 9776 record(s) loaded 5 MB extracted / 5 MB loaded

Is

airbyte-ci

supported on macos running on Intel? I have “successfully” installed it on my local machine…

michael@michaels-mbp-16 airbyte-master % make tools.airbyte-ci.install
Nothing to uninstall for pipelines :sleeping:
Uninstalled pipelines via pipx
Removing /Users/michael/.local/bin/airbyte-ci
Removed airbyte-ci
Removed airbyte-ci-internal
Removed airbyte-ci-dev
All airbyte-ci references removed from pyenv versions.
Cleanup of airbyte-ci install completed.
macOS based system detected.
Downloading from <https://connectors.airbyte.com/files/airbyte-ci/releases/macos/latest/airbyte-ci>

  ╔───────────────────────────────────────────────────────────────────────────────╗
  │                                        │
  │  AAA  IIIII RRRRRR BBBBB  YY  YY TTTTTTT EEEEEEE     CCCCC IIIII  │
  │  AAAAA  III RR  RR BB  B YY  YY  TTT  EE       CC    III  │
  │ AA  AA III RRRRRR BBBBBB  YYYYY  TTT  EEEEE  _____ CC    III  │
  │ AAAAAAA III RR RR BB  BB  YYY   TTT  EE       CC    III  │
  │ AA  AA IIIII RR  RR BBBBBB  YYY   TTT  EEEEEEE     CCCCC IIIII  │
  │                                        │
  │ === Installation complete. v(latest)===                   │
  │ /Users/michael/.local/bin/airbyte-ci                              │
  ╚───────────────────────────────────────────────────────────────────────────────╝
   
Checking if airbyte-ci is correctly installed...
airbyte-ci is correctly installed at /Users/michael/.local/bin/airbyte-ci4

However, when I come to run it…

michael@michaels-mbp-16 airbyte-master % airbyte-ci
zsh: bad CPU type in executable: airbyte-ci

Thanks

Hi Community, I need to change the airbyte authentication e-mail. Firstly, I make an update in user table, seting the new e-mail, after, I restarted all airbyte kubernetes pods, but nothing worked. I also have changed the profile user in airbyte ui. Someone could help me?

Hey community! I’m wondering if anyone has managed to get Google Ads events from Airbyte into BQ? Not only ‘installs’ for example, but custom events as well. I know that the API has some limitations in that you can’t split conversions by conversion type.. but what about custom events? Any help appreciated!

I have migrated my airbyte instance from dockercompose to abctl. the firsttime i ran abctl local credentials i got the credentials. But running the same command the following day gives me an error [ec2-user@ip-10-24-2-203 bin]$ abctl local credentials INFO Using Kubernetes provider: Provider: kind Kubeconfig: /home/ec2-user/.airbyte/abctl/abctl.kubeconfig Context: kind-airbyte-abctl ERROR Unable to determine organization email ERROR unable to determine organization email: failed to get organization: unable to fetch token: unable to decode token request: invalid character '<' looking for beginning of value The password still works on the browser but I am NOT able to retrieve from command line using the abctl local credentials Any idea how to fix this

Hi community, I'm facing an error syncing data from snowflake to BQ. I'm using the OSS helm chart to deploy airbyte. When I manually (from the UI) create the snowflake source, it passes the set up check, but then when I try to do a full sync to BQ, the job hangs and the logs of the source-snowflake-read pod only show this

Waiting on CHILD_PID 7
PARENT_PID: 1

the orchestrator pod shows a ton of these

2024-08-27 16:10:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 768 MB, allocated: 10 MB (10.0 MB), %% used: 0.013020833333333334 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-08-27 16:10:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0

and the destination-bigquery-write shows something identical to the source

Waiting on CHILD_PID 7
PARENT_PID: 1

Any ideas?

Hi everyone, I’m running into an issue with Airbyte where the data syncs from BigQuery to SQL Server, but the data seems to be stuck in the

airbyte_internal

schema. It doesn’t seem to be unpacking into the expected tables in the actual schema. I’m using the default setup without any custom transformations or dbt, so I’m not sure what might be causing this. Has anyone else experienced this or know how to ensure the data is correctly unpacked into the final destination schema? Thanks for any help you can provide!

Hello, I am hosting Airbyte with docker compose in an EC2 instance and no jobs are running anymore. From the temporal container I see this logs: > "level":"info","ts":"2024-08-27T192228.972Z","msg":"history client encountered error","service":"frontend","error":"service rate limit exceeded","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"metric_client.go:90"} And from the worker: > x2024-08-27 191333 WARN i.t.i.w.WorkflowWorker$TaskHandlerImpl(logExceptionDuringResultReporting):416 - Failure while reporting workflow progress to the server. If seen continuously the workflow might be stuck. WorkflowId=connection_manager_6b4ace68-3ee4-4c0f-bc35-f2b8ff9e6d80, RunId=b49c49fd-0e6a-4413-9ff1-1214c9d52403, startedEventId=0 > io.grpc.StatusRuntimeException: NOT_FOUND: query task not found, or already expired Does anyone know what they mean and how to recover from this state? Or where to look for more information?

Has anyone else gotten this error when using MSSQL as a source?

An unknown error occurred. (HTTP 504)

Hi - Having an issue here with the WooCommerce connector. I can make a successful connection and do receive data from WooCommerce to my MS SQL database. However, the tables are not populating, or have only populated with a few orders and new syncs, even with a complete refresh do not update. The raw data streams are updating with the latest data, however. I don’t see much on this connector for problems at this level, so curious if anyone else has experienced this issue and how you resolved. Thanks!

I am trying to pull campaign performance report at device level from google Ads but it does not have device Type segment. I tried to write custom GQAL but stream does not have option of incremental update. Any idea, how can I get campaign and ad_group permformance reports at device Type and date level ?

Hey community, I have been trying out version 0.63.19/helm 0.453.2 The airbyte metrics server is failing due to

PortUnreachableException

, Port 80 can't be reached, anyone raised this before? We are urging to use new version for Postgres Refresh & Retain feature

Hi team, I am observing a weird behavior for a postgres table. As this table is not having timestamp column I am using full refresh overwrite mode for syncing the data. But the problem is some times the table come as blank and because of full refresh the table in destination bigquery is also getting blank. I have verified the data is present in postgres table and it is not blank, why still airbyte is fetching it to be blank?

#44852 Workspace Notifications Configuration via API? New discussion created by dominon12 Hi everyone, I couldn't find specific details in the API documentation, so I wanted to ask here: Is there an option to configure notifications via the API? Specifically, I'm looking to disable certain types of email notifications and enable webhooks. Any guidance or pointers would be appreciated! airbytehq/airbyte

Hi all. I'm trying to set up Airbyte via Kubernetes with Helm Chart. I'm setting up with my own values.yaml, but I'm having trouble having Airbyte up and running due to not being able to add sslmode=true to the values.yaml. My values.yaml is as follows:

postgresql:
    enabled: false

# Trying with "externalDatabase" and without "externalDatabase" makes no difference
externalDatabase:
  host: "host-name"
  user: "user-name"
  database: "db-name"
  password: "password-value"
  port: 5432
  jdbcUrl: "<postgresql://host-name:5432/db-name?ssl=true&sslmode=require>"
  # trying "jdbc:<postgresql://host-name:5432/db-name?ssl=true&sslmode=require>" doesn't help either

global:
  database:
    host: "host-name"
    user: "user-name"
    database: "db-name"
    password: "password-value"
    port: 5432
    sslmode: "require"
    jdbcUrl: "<postgresql://host-name:25060/db-name?ssl=true&sslmode=require>"

Hi all, I am trying to use amazon ads source, but getting empty product and other reports with airbyte. Am I missing some configurations here?

Comment on #44852 Workspace Notifications Configuration via API? Discussion answered by nataliekwong There is not currently a way to configure the notification settings through the API. Unfortunately this can only be done through the UI today. I appreciate the suggestion and will add it to our feature roadmap! airbytehq/airbyte

Hello I am trying to connect connect to the database through abctl. I have done everything as in the video but get this error:

unable to install airbyte chart: unable to install helm: pre-upgrade hooks failed: 1 error occurred:
          	* pod airbyte-abctl-airbyte-bootloader failed

it is pointing to an old airbyte pg database

Hello Airbyte team 👋 We just update our airbyte instance to the 0.63.8 version and all our connectors seems to run correctly! We also update our Snapchat connector but we are now facing an issue (related to this conversation) The connector is working well but no data is loaded. Are you aware of this issue? Thanks a lot 🔥

I'm working with a postgres connector via supabase to bigquery and have someone experiencing errors regarding maximum slot reservations exceeding:

Caused by: org.postgresql.util.PSQLException: ERROR: cannot read from logical replication slot "airbyte_ratings_slot"
  Detail: This slot has been invalidated because it exceeded the maximum reserved size.

The configuration for the LSN commit behavior is currently set to "While Reading Data". Could switching to "After Loading Data in the Destination" resolve the issue? It's not my airbyte instance so I don't have control of changing it. I do want to make sure I can advise the best course of action.