#47335 SessionTokenAuthentication: make SessionToken available as a global secret variable New discussion created by Tom-DK I’m implementing an API using the no-code builder, but unfortunately I’m hitting the limits… The API requires a sessionToken to be injected in a specific way into the body payload. Currently, you can only inject the sessionToken as a Bearer token or a custom API_KEY header. It would be wonderful to have a third option, that just stores the session token in a global secret variable e.g.

config[‘sessionToken’]

. This way you can interpolate the tokenstring wherever in many ways. The no-code builder could be covering much more use cases by this. airbytehq/airbyte

#47337 [Connector-Request] Pinata New discussion created by aazam-gh Topic This is a suggestion to create a connector for Pinata using the UI Connector Builder. The website for the connector is https://pinata.cloud/ Relevant information https://docs.pinata.cloud/quickstart airbytehq/airbyte

Hi all - I am trying to build the workload-launcher image from the airbyte-platform repository but I am not sure how can I build it with amd64 architecture while running it on Mac M1 - Any hints on how can I accomplish this?

Hi all, Is there Airbyte API available which i can use to update the frequency ?

Hello, Re. Google Secret Manager Roles described in the documentation, it turned out it was not enough for me, I had to add

roles/secretmanager.admin

to be able to create a secret in Google Secret Manager.

Hey, it seems that the facebook marketing connector does not export data from facebook lead forms. Is this supposed to be the case or am i missing something? Is there another connector that will allow me to get this data?

Hey everyone i am running timscaledb/postgres image and the postgres has multiple dbs and i want to have cdc enabled for the all the table in all dbs evwntually i followed the documentation of the airbyte how to set up but in the air byte its not showing incrementa sync

Can any one lead me to the correct way that would be much help ful for me

in the developer guide its says enable logical replication but in the airbyte ui there is no logical replication selection airbyte version v1.1.0 can i know why is that

#47347 Setting up CDC(Change Data Capture) for Timescaldb PostgreSQL New discussion created by Nanagoud007 I am running a TimescaleDB/PostgreSQL image with multiple databases and want to enable CDC (Change Data Capture) for all tables across all databases. I followed the Airbyte documentation here, but the incremental sync option isn't appearing in Airbyte. The documentation also mentions that CDC doesn't work with TimescaleDB, so I created regular tables to sync instead, but the sync still isn't working. The developer guide suggests enabling logical replication, but there is no option for logical replication in the Airbyte UI. I'm using Airbyte version v1.1.0—can you clarify why this might be happening? Screenshot from 2024-10-24 13-43-29 Can any one lead me to the correct way that would be much helpful for me airbytehq/airbyte

trouble connecting to salesforce sandboxThe authentication to SalesForce has expired. Re-authenticate to restore access to SalesForce.

Hello, Does anyone know how can I reset a connection airbyte in version

0.61.0

? I can find any button anywhere to reset the connection. I need to reset the connection because of this error:

Failure in source: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future. See <https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs> for more details.

Why i am getting this every time whenever i open postures source its keep happening

Hello, I'm having this issue where Airbyte is not mounting secrets correctly into my pods. I'm assuming I screwed up with the configuration somewhere, but when I describe the connector pod that gets created after I input my api keys in the UI I get this event message "MountVolume.SetUp failed for volume "airbyte-secret" : secret "airbyte-config-secrets" not found" I have deployment secrets set up within my values.yaml file fine, but it seems like for some reason Airbyte isn't generating or automatically mounting the secrets I input into the UI? Any help would be very appreciated!

Trying to quickstart install on a Amazon 2023 Linux EC2 on t3.xlarge. Install hasn't failed but i'm getting this warning/error in a loop for nearly an hour

▄  Installing 'airbyte/airbyte' (version: 1.1.0) Helm Chart (this may take several minutes) (55m22s)I1024 18:43:35.123571   26825 request.go:697] Waited for 2.493354681s due to client-side throttling, not priority and fairness, request: GET:<https://127.0.0.1:46369/api/v1/namespaces/airbyte-abctl/pods?fieldSelector=metadata.name%3Dairbyte-abctl-airbyte-bootloader&resourceVersion=614>
▄  Installing 'airbyte/airbyte' (version: 1.1.0) Helm Chart (this may take several minutes) (58m15s)W1024 18:46:37.506962   26825 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *unstructured.Unstructured: Get "<https://127.0.0.1:46369/api/v1/namespaces/airbyte-abctl/pods?fieldSelector=metadata.name%3Dairbyte-abctl-airbyte-bootloader&resourceVersion=614>": net/http: TLS handshake timeout
▀  Installing 'airbyte/airbyte' (version: 1.1.0) Helm Chart (this may take several minutes) (58m21s)I1024 18:47:04.712537   26825 trace.go:236] Trace[517752767]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229 (24-Oct-2024 18:42:52.674) (total time: 234982ms):
Trace[517752767]: ---"Objects listed" error:Get "<https://127.0.0.1:46369/api/v1/namespaces/airbyte-abctl/pods?fieldSelector=metadata.name%3Dairbyte-abctl-airbyte-bootloader&resourceVersion=614>": net/http: TLS handshake timeout 222375ms (18:46:35.049)
Trace[517752767]: [3m54.982861097s] [3m54.982861097s] END

Anyone seen anything like this before?

hi every one i am trying to sync the one table at the start of the connection sync took 21 minutes for 6000 rows from postgres source to the postgres destination is there anything i have to looks up to speed up the process or did do anything wrong this is the logs i am getting 024-10-24 193417 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1012 2024-10-24 193427 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=111, polls=0 1013 2024-10-24 193427 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.024673926S after its previous call which was also logged. 1014 2024-10-24 193427 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1015 2024-10-24 193431 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 72.7 MB, allocated: 10 MB (10.0 MB), %% used: 0.1375515822040699 | Queue

mt_postgresdaily_connector_exported_uptime_data

, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 1016 2024-10-24 193431 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 1017 2024-10-24 193437 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=112, polls=0 1018 2024-10-24 193437 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.02155353S after its previous call which was also logged. 1019 2024-10-24 193437 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1020 2024-10-24 193448 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=113, polls=0 1021 2024-10-24 193448 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.522454308S after its previous call which was also logged. 1022 2024-10-24 193448 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1023 2024-10-24 193458 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=114, polls=0 1024 2024-10-24 193458 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.025536811S after its previous call which was also logged. 1025 2024-10-24 193458 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1026 2024-10-24 193508 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=115, polls=0 1027 2024-10-24 193508 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.036445464S after its previous call which was also logged. 1028 2024-10-24 193508 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat. 1029 2024-10-24 193518 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=116, polls=0 1030 2024-10-24 193518 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.024164285S after its previous call which was also logged. 1031 2024-10-24 193518 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.

hello #C021JANJ6TY, does anyone knows how to setup the webhook? I have a rest API server waiting for the POST from the webhook notification, but when I add the url, it says its invalid, but if I run it as a curl with the example payload from the docs, it works.

curl -X POST <http://localhost:8080/airbyte_internal/sync_webhook> \
-H "Content-Type: application/json" \
-d '{
    "data": {
        "workspace": {
            "id":"b510e39b-e9e2-4833-9a3a-963e51d35fb4",
            "name":"Workspace1",
            "url":"<https://link/to/ws>"
        },
        "connection": {
            "id":"64d901a1-2520-4d91-93c8-9df438668ff0",
            "name":"Connection",
            "url":"<https://link/to/connection>"
        },
        "source": {
            "id":"c0655b08-1511-4e72-b7da-24c5d54de532",
            "name":"Source",
            "url":"<https://link/to/source>"
        },
        "destination": {
            "id":"5621c38f-8048-4abb-85ca-b34ff8d9a298",
            "name":"Destination",
            "url":"<https://link/to/destination>"
        },
        "jobId":9988,
        "startedAt":"2024-01-01T00:00:00Z",
        "finishedAt":"2024-01-01T01:00:00Z",
        "bytesEmitted":1000,
        "bytesCommitted":1000,
        "recordsEmitted":89,
        "recordsCommitted":89,
        "bytesEmittedFormatted": "1000 B",
        "bytesCommittedFormatted":"90 B",
        "success":true,
        "durationInSeconds":3600,
        "durationFormatted":"1 hours 0 min"
    }
}'

Webhook test failed. Please verify that the webhook URL is valid.

trouble connecting to salesforce sandboxThe authentication to SalesForce has expired. Re-authenticate to restore access to SalesForce. any help on resolving this is really appreciated

I am using kubernetes setup & are trying to use external postgres db setup. I keep running into a PersistentVolumeClaims not being available issue.

helm install airbyte airbyte/airbyte --namespace airbyte --values ./values.yaml

on

StatefulSet/airbyte-minio

0/7 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.

I've checked &

kubectl get sts -n airbyte airbyte-minio -oyaml

I noticed my spec is missing a storageClassName

spec:
      storageClassName: gp2

I have updated the class manually & restarted it. This resolved this issue, however not a real solution. I've tried adding this to my values.yaml file as such

airbyte:
  minio:
    persistence:
      enabled: true
      storageClassName: gp2
      size: 500Mi

without luck either

kubectl get storageclass

NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    <http://kubernetes.io/aws-ebs|kubernetes.io/aws-ebs>   Delete          WaitForFirstConsumer   false                  28h

I’ve set up logical replication and enabled CDC in PostgreSQL, configuring the sync mode in Airbyte as incremental append+dedupe. However, during each sync, it’s writing entire rows from the table. Is this expected behavior, or is there additional configuration needed in Airbyte or the source? can any enlighten me please

Hi everyone! I am using the Salesforce Source connector and it has the option to "Force to use BULK API". However, the info warns that "this might cause empty fields for some streams". Does anyone know what that refers to? There is no mention of it in the documentation.

Am testing out mongo to couchbase connection However, i am getting an error from the replication platform

Hy Friends. My salesforce sync is regularly incomplete with the following error message, however I do not see "WorkThanks" as a stream I can disable. Any direction on what could be causing this?:

'GET' request to '<https://brownhavenhomes.my.salesforce.com/services/data/v57.0/sobjects/WorkThanks/describe>' failed with status code '404' and error message 'The requested resource does not exist'

#47374 Yuki New discussion created by Tom-DK Yuki is a Belgian/Dutch accounting software airbytehq/airbyte

Hi Everyone! I am trying to move data from one Redshift cluster to another Redshift cluster. When I create both the source and the destination targets everything seems to be ok, but when I try to create the connection I'm getting this error:

java.lang.NullPointerException: null value in entry: isNullable=null

https://github.com/airbytehq/airbyte/issues/47369

I'm trying to load a historical table with 400 million records and I receive the error below regarding lack of resources. How can I resolve this? How do I configure batch size for the Oracle source? Airbyte could not start the sync process. This may be due to insufficient system resources. ResourceRequirements(claims=[], limits={memory=30Gi, cpu=8}, requests={memory=30Gi, cpu=8}

Hello channel, I'm planning to write a custom docker image for airbyte due to security requirements within our infrastructure at as we are unable to use the community Docker images directly. I need to build a custom Docker image that complies with our artifactory security protocols. Could anyone please provide community Dockerfile or any related guidance on constructing a custom Docker image for Airbyte? Access to this would greatly assist us in ensuring compliance and maintaining security standards. Thank you for your support and looking forward to your guidance! thanku

Hi, I am getting this error in a EC2 deployment when enter to airbyte in localhost. My airbyte version is community 0.63.9 (status 500)`"message": "Internal Server Error: Error getting connection from data source HikariDataSource (HikariPool-1)",`

"exceptionClassName": "org.jooq.exception.DataAccessException",

"exceptionStack": [

"org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-1)",

"\tat org.jooq_3.19.7.POSTGRES.debug(Unknown Source)",

It doesn't happen all the time, but it's quite annoying and I have to restart the container at least once a day. Does anyone know why this happens?, thanks.

abctl install on a ec2 instance with external rds airbyte v.1.1:

ERROR   airbyte-bootloader: i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
  ERROR   airbyte-bootloader: org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-1)
  ERROR   airbyte-bootloader: Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=1)
  ERROR   airbyte-bootloader: Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
  ERROR   airbyte-bootloader: Caused by: java.net.UnknownHostException: database-host
  ERROR   airbyte-bootloader: i.m.m.h.i.HealthResult$Builder(exception):123 - Health indicator [database-host:5432/airbyte] reported exception: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=1)

I am working with the Pyairbyte library and I wanted to know if there's a way to just load the documents from the source without extracting data from it using Pyairbyte. This is because I am facing issues with loading and extracting pdf type documents