Hi how can we connect organization github repo as a source in Airbyte using personal access token. Personal git works for me but for organization it does not work. Is there a way to configure the github url for that? Thanks!

Hey Airbyte pros... Probably a simple one. We'll see if the AI can handle it. I have a json file that I am pulling in from a public API... there are about 94k rows, but Airbyte is only pulling down 1k. I'm using the File (CSV, JSON, Excel, Feather, Parquet) with a destination of MS SQL Server. Any thoughts? I checked but I didn't see any other settings. Maybe it's the wrong connector? Could there be a pagination thing going on in the source? Any way to over-come that with reader settings?

Hi All! , I am trying to deploy airbyte in EKS cluster using helm chart using external minio and external db I have disabled minio helm and enabled minio under external minio section in values.yaml file there is no option to provide access key and secret key under

global.logs.secretKey.password

after deploying when I am trying to create source connection I get the following error

Internal Server Error: Unable to execute HTTP request: Connect to minio-s3:9000 [minio-s3/x.x.x.x, minio-s3/x.x.x.x, minio-s3/x.x.x.x] failed: Connect timed out

Hi everyone, I am just getting started with Airbyte and I can't figure out how to install Octavia CLI on my machine This curl command resulted nothing:

curl -s -o- <https://raw.githubusercontent.com/airbytehq/airbyte/master/octavia-cli/install.sh> | bash

the same with docker run command

docker run --name octavia-cli -i --rm -v my_octavia_project_directory:/home/octavia-project --network host --user $(id -u):$(id -g) --env-file ~/.octavia airbyte/octavia-cli:0.44.1

This message was deleted.

Hello there, I am Rob. I need help. I am trying to delete a workspace through an API call in a python environment. However, my request was not executed.

Hi I am trying to get security approval for the product internally. Is there any documentation or person I can talk to in order to help me with this.

Hi here! After reading the core concepts glossary, I still don’t manage to grasp the difference between a connector and a connection. Could someone enlight me on the conceptual distinction? 🙏

Hi all, I am trying to deploy airbyte:0.44.1 using docker-compose on my Mac M1 follow this tutorial https://docs.airbyte.com/deploying-airbyte/local-deployment/#setup--launch-airbyte. All the service are up and running, but I could not connect to webserver with the error

Cannot reach server. The server may still be starting up.

Airbyte webserver returns the following error:

airbyte-webapp                    | 2023/04/25 13:45:53 [error] 30#30: *55 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.176.9, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.0", upstream: "<http://192.168.176.7:8001/api/v1/workspaces/list>", host: "localhost", referrer: "<http://localhost:8000/>"
airbyte-webapp                    | 192.168.176.9 - airbyte [25/Apr/2023:13:45:53 +0000] "POST /api/v1/workspaces/list HTTP/1.0" 502 497 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0" "192.168.176.1"

All the .env config is using default. In the airbyte_worker and server has the following warning log that just keep looping, which says waiting for temporal

2023-04-25 13:48:26 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):259 - Waiting for namespace default to be initialized in temporal...
2023-04-25 13:48:28 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=107, target=airbyte-temporal:7233}}
2023-04-25 13:48:38 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):269 - Ignoring exception while trying to request Temporal namespace:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions will be exceeded in 9.998615459s.

There is no error log in both worker and server Is someone run to this deployment issue? Many thanks for your help!

Example of the metrics, I can better

Hi there, I'm trying to deploy Airbyte via Helm (chart 0.45.12/App version 0.44.0) using an external database. The bootstrapper runs successfully, I can see the database has been initialized, and all the pods start up, except airbyte-webapp gets stuck in a crashloop with the following log output, so the helm deployment eventually fails and I'm unable to access the webserver.

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/04/24 20:15:56 [emerg] 1#1: unknown "airbyte_version" variable
nginx: [emerg] unknown "airbyte_version" variable

Is there a way in Airbyte to do an incremental sync for a table every [x] hours, and a full re-sync every [30] days?

Hi there, I arrived at the reply.io-to-snowflake landing page and now that I’ve created an account I can’t find reply.io in the list of sources.

I am deploying airbyte to EKS using terraform and the provided helm chart. If I am wanting to change the default basic auth username and password while deploying, would it be best to set the

env_vars

argument in either the

server

or

webapp

block, or is there not a way to change while deploying?

Hi there! I think I am missing something about Airbyte Workers scaling using docker swarm deployments. At the moment we have 3 servers (32G RAM, 8 cores) dedicated specifically for Airbyte Workers. Each server has it’s own containers: 1. airbyte-server 2. airbyte-temporal 3. airbyte-worker The issue is, when we run hundreds of syncs from time to time all three airbyte-workers stop spawning containers to process Airbyte jobs. As a result we have 500+ jobs in a running state, but none of them got actually processed for 5-60 minutes. After some time the processing usually starts, but it is not stable. That means one of the server might be fully loaded (100% CPU), and the other will have no jobs running at all. Also when the running jobs are processed, it might take a minute or two before the next batch starts processing. I’ve read all the docs about jobs/scaling/workers a dozen of times, I’ve check all configuration options and started reading actual code to check what’s going on there. Did you experience similar behaviour? Maybe you have any suggestions? Could you please point me to the place in the code where airbyte-worker spawns the containers to process the jobs, so I can proceed with my investigation?

I am trying to add a test to https://github.com/airbytehq/airbyte/pull/25386 in the file https://github.com/airbytehq/airbyte/blob/8c9ac5492cc91b5be8576818971903a1820540b5[…]e-db/db-lib/src/test/java/io/airbyte/db/jdbc/TestJdbcUtils.java How do I run it? I tried

./gradlew --max-workers 2 test --tests '*TestJdbcUtils*'

The test I want to add looks like this:

@Test
  void testParseJdbcParameters() {
    Map<String, String> parameters = JdbcUtils.parseJdbcParameters("sessionVariables=max_execution_time=10000", "$");
    assertEquals("max_execution_time=10000", (int) my_hashmap.get("sessionVariables"));
  }

@Marcos Marx (Airbyte) FYI

hi there, thank you for all the hard work on this tool, it is really great. I am trying to build a custom connector for an API which has a auth endpoint like

<https://api.us-st.synchtank.net/auth/login>

and it accepts a json body with username/password. The response provides a

access_token

refresh_token

and a

expiry

, the access token to be used as a bearer for future api calls. Is there any recommendation on how to set up the spec file to accommodate for this? I tried doing it via oauth but there is no client_id or client_secret so that did not seem appropriate. As an alternative I have just manipulated the sources file to get the the access_token as part of starting a stream but it seems like a better approach to be able to use the refresh token and monitor when a given access_token is close to expiry

👋 Hi All, Curious if anyone has seen this error before on airbyte connecting to the Databricks Lakehouse destination:

Received fatal alert: bad_record_mac

I am running on docker-desktop k8s cluster with airbyte deployed through helm for testing purposes on an M2 Mac. The data loads when resetting the sync, but does not succeed in the 3 following attempts.

I want to download ZIP files from a URL without extracting them. I don't think the current Files Source connector support this feature. Is there any other connector that supports this feature?

Hi guys how are you? Maybe its simple but could not solve here, I running a Sync between Postgres 2.0.25 and Bigquery 1.3.1 using CGS Staging, the process is freeze and only got this message?

```2023-04-26 05:35:52 destination > INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$3):131 Preparing airbyte_raw tables in destination completed.

Are there a configuration on GCP Bucket that not config?

According to https://docs.airbyte.com/cloud/managing-airbyte-cloud/edit-stream-configuration it is possible to select a set of columns to sync for each stream:

1. Toggle individual fields to include or exclude them in the sync, or use the toggle in the table header to select all fields at once.

It is not obvious how to open the dialog to select fields from the documentation. I tried this with a connection from MySQL to Redshift but didn't manage. And I remember reading a Github issue with a similar request, so my guess is that this feature is restricted to specific destinations. Or am I mistaken and this is that actually possible somehow? 🤞 :D

Hi, I'm using source-jira 0.3.7 to destination-postgres 0.3.27, on airbyte 0.44.0. The connection seems to be ok for a while - data is synced across, but after some time the connection hangs, and then 3 hours later times out. See logs in thread 🧵

Is it possible to set a timezone to different workspaces. I have several of them that contain very different timezones (asia/US) when I want to extract data for the previous day for Asia timezone. I fail to do so because it is still the current date. E.G. 2023-01-01 22:30 UTC -> 2023-01-02 00:30 EET. any thoughts about it?

Hi all, my team and I are looking at migrating from fivetran to a self-hosted airbyte instance. During our exploration, we've come to learn that in order to capture deletes in the source system, we'd need to use CDC. However we may not have control over whether the source databases have activated bin logs or not. Is there any other way to capture deletes from the source in airbyte?

Hey Team, facing some problem with destination elasticsearch with full refresh-overwrite mode getting duplicates into elasticsearch... Please help me on this

Hello, I'm currently replicating data from postgres to clickhouse using airbyte. The sync mode is Incremental( Deduped + history ) and the cursor field is

updated_at

which default value is current timestamp. There's an error message in data normalization as shown below:

normalization > 11:37:51.420721 [debug] [Thread-1 (]: dbt_clickhouse adapter: Error running SQL: /* {"app": "dbt", "dbt_version": "1.4.5", "profile_name": "normalize", "target_name": "prod", "node_id": "model.airbyte_utils.airbyte_test_my_table"} */
INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
normalization >         insert into postgres_airbyte.airbyte_test_my_table__dbt_tmp ("fwd", "f_date", "code", "updated_at", "iid", "trfd", "thrudate", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_airbyte_test_my_table_hashid")
normalization >         select "fwd", "f_date", "code", "updated_at", "iid", "trfd", "thrudate", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_airbyte_test_my_table_hashid"
normalization >         from postgres_airbyte.airbyte_test_my_table
normalization >           where (_airbyte_unique_key) not in (
normalization >             select _airbyte_unique_key
normalization >             from postgres_airbyte.airbyte_test_my_table__dbt_new_data
normalization >           )
...
normalization >    Code: 47. DB::Exception: Missing columns: '_airbyte_unique_key' while processing query: 'SELECT fwd, f_date, code, updated_at, iid, trfd, thrudate, _airbyte_ab_id, _airbyte_emitted_at, _airbyte_normalized_at, _airbyte_airbyte_test_s

It seems like airbyte is copying data from

airbyte_test_my_table

to

airbyte_test_my_table__dbt_tmp

without the column named

_airbyte_unique_key

, but

_airbyte_unique_key

needs to be retrieved in the query. Does it reproduce on recent release?

Hi I have a question about twilio connection. We get an error “Failure Origin: source, Message: Date range specified in query is partially or entirely outside of retention window of 7 days”, but we cannot constantly change the start date in the connector setting. How should we set the parameters so that we could always ingest the last 7 days of data?

I am using the connector builder ui and have a 3 connectors built . The startdate and endDate are injected into tye request body . The cursor field of date uses a different date format to the request paramter. Does anyone know of there is a way around this . The cursor fields date is Y%-m%-d% and the request body takes m%/d%/Y% . It seems that the saved point in the connection uses the max from the last cursor request ? Is there anyway around this

has renamed the channel from "airbyte-connections-issues" to "help-connections-issues"

has renamed the channel from "public-airbyte-help" to "public-airbyte-connections-issues"