Airbyte #ask-ai

Rubén Lucena

10/25/2024, 8:03 PM

Hi team, I'm involved in a project using Airbyte to migrate data from ElasticSearch to S3. I configured Elastic as source and I'm connected to it. The same having S3 as target. But when I sync the schema, Airbyte only returns the indexes. I also need the data streams. How can I retrieve them?

Avinash Vijayarangan

10/25/2024, 8:19 PM

Hello channel, I'm planning to write a custom docker image for airbyte due to security requirements within our infrastructure at as we are unable to use the community Docker images directly. I need to build a custom Docker image that complies with our artifactory security protocols. Could anyone please provide community Dockerfile or any related guidance on constructing a custom Docker image for Airbyte? Access to this would greatly assist us in ensuring compliance and maintaining security standards. Thank you for your support and looking forward to your guidance! thanku

Aasim ali

10/25/2024, 8:35 PM

Error: UPGRADE FAILED: YAML parse error on airbyte/charts/server/templates/deployment.yaml: error converting YAML to JSON: yaml: line 221: could not find expected ‘:’ @kapa.ai

Ethan Stuart

10/25/2024, 10:04 PM

Is there any need to do database maintenance on the airbyte databases? such as deleting old rows from any of the tables.

+14

Braeden Stamas

10/26/2024, 12:27 AM

How should I structure the requester portion of a connector's yaml file to allow for multiple headers? Is this correct? base_requester: type: HttpRequester url_base: https://www.gottman.com/wp-json/wc/v3 http_method: GET request_headers: User-Agent: Airbyte-WooCommerce-Connector/1.0 authenticator: type: ApiKeyAuthenticator header: Authorization api_token: <redacted>

Stan Kirdey

10/26/2024, 1:55 AM

Backoff before next attempt: 10 seconds 2024-10-25 224437 ERROR i.a.w.l.p.h.FailureHandler(apply):39 - Pipeline Error io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: CHECK pod failed to init within allotted timeout. It seems when deploying to kubernetes, I get tolerations on some of the pods, not on connectors - they start without tolerations. I am on 1.1.0 Airbyte and it was upgraded from 0.48.

Carolina Buckler

10/26/2024, 2:24 AM

trying to set up ec2 instance with abctl and an external rds, but getting this error on the install:

Copy code

ERROR   airbyte-bootloader: i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
  ERROR   airbyte-bootloader: org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-1)
  ERROR   airbyte-bootloader: Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms (total=0, active=0, idle=0, waiting=0)
  ERROR   airbyte-bootloader: Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
  ERROR   airbyte-bootloader: Caused by: java.net.UnknownHostException: database-host

Yannick Sacherer

10/26/2024, 7:41 AM

@kapa.ai do you have an example custom source that sends queries against an api?

Damien Querbes

10/26/2024, 7:42 AM

@kapa.ai Hello, I run an Airflow Kubernetes pod. I run Airbyte on a Kubernetes pod within the same VPC network/subnetwork. I want to trigger an airbyte connection through airflow DAG. However, I get this error: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.0.2', port=8080): Max retries exceeded with url: /api/v1/connections/sync (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7838d78526e0>, 'Connection to 172.16.0.2 timed out. (connect timeout=None)'))
172.16.0.2 is the internal endpoint of my Airbyte kb8 pod.

Yannick Sacherer

10/26/2024, 7:56 AM

@kapa.ai I dont get server side pagination, but I need to send a new api requests every 10.000 rows. I don't get any information like the offset or something else. it really just is. take the last state from the api call and run again api call. hjow to do this with the httpstream class

+12

Jake Duckworth

10/26/2024, 2:26 PM

I have a connection set up between 2 postgres databases. The source is configured to use CDC replication. The tables in the stream are configured to use

incremental_deduped_history

. When the sync runs, it does a full refresh every time instead of processing the incremental changes. Airbyte version: 1.1.0 Postgres Source Version: 3.6.22 Postgres Destination Version: 2.4.0 Poatgres config:

Copy code

CREATE ROLE source_1_airbyte_role;
ALTER ROLE source_1_airbyte_role REPLICATION LOGIN;
GRANT SELECT ON TABLE public.table_1 TO source_1_airbyte_role;
ALTER TABLE public.table_1 REPLICA IDENTITY DEFAULT;

Jake Duckworth

10/26/2024, 2:32 PM

I have a connection set up between 2 postgres databases. The source is configured to use CDC replication. The tables in the stream are configured to use the Incremental | Append + Deduped (
incremental_deduped_history
) sync mode_._ When the sync runs, it does a

full refresh

every time instead of processing the incremental changes. Do I have something misconfigured? Any idea why the incremental processing is not working? Here is my current setup: Airbyte version: 1.1.0 Postgres Source Version: 3.6.22 Postgres Destination Version: 2.4.0 Postgres config:

Copy code

CREATE ROLE airbyte_role;
ALTER ROLE airbyte_role REPLICATION LOGIN;

GRANT SELECT ON TABLE public.table_1 TO airbyte_role;
ALTER TABLE public.table_1 REPLICA IDENTITY DEFAULT;
 
CREATE PUBLICATION airbyte_publication FOR TABLES IN SCHEMA public;
SELECT pg_create_logical_replication_slot('airbyte_slot', 'pgoutput');

Atharva Gunjkar

10/26/2024, 3:28 PM

Hello Channel , I have airbyte oss(appVersion: 0.50.56) on gke running from last one year ,this is the only application running on the whole cluster Suddenly i am getting 504 and not able to open the webapp all the cpu and memory request are within limit , workder nodes are 4cpu and 16gb ram My Traefik logs get 200 for initial few request for the assets and the later are retuning 499 for all the other requests My ingressroute is pointed to the webapp on port 80 kubectl port-forwarding it to local i am able to access my app perfectly and my jobs are still running This setup is behind VPN Any leads on debugging this issue will be helpful Also tried adding below to traefik spec: forwardingTimeouts: dialTimeout: 120s responseHeaderTimeout: 300s idleConnTimeout: 300s

Alasdair Ellis

10/26/2024, 5:18 PM

@kapa.ai i am receiving this error can you give more information: Failed to start sync: An unexpected error occurred. Please report this if the issue persists. (HTTP 500)

Kay Burdeos

10/27/2024, 1:47 AM

Can I update an existing airbyte installation to use instanceProfiles for logging?

Dainius Salkauskas

10/27/2024, 6:19 AM

I am developing a new source connector. I want to add the connector icon. How can I do that using the UI connector builder?

Power School

10/27/2024, 7:07 AM

helo,

Power School

10/27/2024, 7:08 AM

I install airbyte on local sever and i am ab;e to access the webpage within the same sever, but not from other machine. using the http://10.127.6.82:8000/

Power School

10/27/2024, 7:10 AM

I install airbyte on local sever on docker compose and i am able to access the webpage within the same sever, but not from other machine. using the http://10.127.6.82:8000/

Power School

10/27/2024, 8:09 AM

how to get dataline-key-airbyte.pem

Power School

10/27/2024, 9:00 AM

how to create users for airbyte

Power School

10/27/2024, 9:02 AM

how to create users for login to web interface for airbyte

Power School

10/27/2024, 9:04 AM

how to create users for login to web interface for airbyte install on docker on local server

Power School

10/27/2024, 9:11 AM

how to create extra users for login to web interface for airbyte install on docker on local server

Braeden Stamas

10/27/2024, 11:04 PM

what should the yaml configuration look like the woocommerce api?

Jack Novorr

10/28/2024, 12:43 AM

@kapa.ai is there a common reason why this happens Check exceeded the timeout.

ABHISHEK TRIPATHI

10/28/2024, 1:53 AM

I am getting the following error with the Airbyte Sql Server source connector

user

10/28/2024, 1:53 AM

I apologize, but you haven't provided the specific error message you're encountering with the Airbyte SQL Server source connector. Without knowing the exact error, I can't provide a precise answer. However, I can offer some general guidance based on common issues with the SQL Server source connector in Airbyte: 1. TLS/SSL Errors: If you're experiencing TLS-related errors, it's often due to version incompatibility. Airbyte supports TLS 1.2 and 1.3, but not TLS 1.0. You may need to enable a higher TLS version on your SQL Server. [Github Issue](https://github.com/airbytehq/airbyte/issues/30287) 2. Connection Issues: Ensure that your SQL Server is accepting TCP/IP connections and that the port (usually 1433) is not blocked by a firewall. [Forum Post](https://discuss.airbyte.io/t/2892) 3. Named Instance Issues: If you're trying to connect to a named SQL Server instance, make sure you're using the correct format for the host name. [Forum Post](https://discuss.airbyte.io/t/2892) 4. Data Type Incompatibilities: Some data types in SQL Server may not be compatible with Airbyte's replication process, especially when using columnstore indexes. [Forum Post](https://discuss.airbyte.io/t/3646) 5. CDC-related Issues: If you're using Change Data Capture (CDC), be aware of retention period limitations. Airbyte may trigger a full refresh if the CDC logs are no longer available. [Github Issue](https://github.com/airbytehq/airbyte/issues/42435) To provide a more accurate solution, please share the specific error message you're receiving. This will help in identifying the exact issue and providing a targeted solution.

ABHISHEK TRIPATHI

10/28/2024, 1:53 AM

I am getting the following error with the Airbyte Sql Server source connector:

Copy code

"failureOrigin" : "source",
  "failureType" : "config_error",
  "internalMessage" : "io.airbyte.commons.exceptions.ConfigErrorException: Saved offset no longer present on the server. Please reset the connection, and then increase binlog retention and/or increase sync frequency.",
  "externalMessage" : "Saved offset no longer present on the server. Please reset the connection, and then increase binlog retention and/or increase sync frequency.",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 15078,
    "from_trace_message" : true,
    "connector_command" : "read"
  },
  "stacktrace" : "io.airbyte.commons.exceptions.ConfigErrorException: Saved offset no longer present on the server. Please reset the connection, and then increase binlog retention and/or increase sync frequency.\n\tat io.airbyte.integrations.source.mssql.initialsync.MssqlInitialReadUtil.isSavedOffsetStillPresentOnServer(MssqlInitialReadUtil.java:155)\n\tat io.airbyte.integrations.source.mssql.initialsync.MssqlInitialReadUtil.getMssqlInitialLoadGlobalStateManager(MssqlInitialReadUtil.java:168)\n\tat io.airbyte.integrations.source.mssql.MssqlSource.initializeForStateManager(MssqlSource.java:638)\n\tat io.airbyte.integrations.source.mssql.MssqlSource.initializeForStateManager(MssqlSource.java:79)\n\tat io.airbyte.cdk.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.kt:158)\n\tat io.airbyte.cdk.integrations.base.ssh.SshWrappedSource.read(SshWrappedSource.kt:69)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.kt:333)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:184)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:116)\n\tat io.airbyte.integrations.source.mssql.MssqlSource.main(MssqlSource.java:618)\n",
  "timestamp" : 1730027410603

Slackbot

10/28/2024, 4:46 AM

This message was deleted.