Airbyte #ask-community-for-troubleshooting

Ferre Janssen

05/12/2022, 2:08 PM

Hi, is it possible to skip the creation of indexes with the postgres destination connector?

Kavin Rajagopal

05/12/2022, 2:55 PM

Hi team! I have 2 GA sources going to one Postgres destination. How do I make sure the second source appends to the first source while writing into the Postgres database?

Vaibhav Kumar

05/13/2022, 5:19 AM

Hello, does

ab_cdc_log_file

ab_cdc_log_pos

attributes in airbyte means binlog file and binlog position from DB ? Can someone from the community please help??

Piyush Bajaj

05/13/2022, 6:25 AM

Hey i am new to airbyte . I am using Airbyte installed on the Compute engine in the GCP.As airbyte is writing a lot of log files daily which is filling the disk space very fast , so is there a way we can automate the deletion of the logs or does airbyte do it automatically . The logs have been written to this path :-/var/lib/docker/volumes/airbyte_workspace/_data Can we delete the enitre _data folder ? will it do some damage to airbyte ?

Piyush Bajaj

05/13/2022, 7:02 AM

what is the path for these

MAXIMUM_WORKSPACE_SIZE_MB

MAXIMUM_WORKSPACE_RETENTION_DAYS

and

MINIMUM_WORKSPACE_RETENTION_DAYS

variables in GCP compute engine?

Cédric Malet

05/13/2022, 9:28 AM

Hello, is it possible to use MariaDB as a source? I only see the MariaDB connector as a destination.

Satish Chinthanippu

05/13/2022, 9:52 AM

Hello Everyone I tried to rebuild airbyte platform code locally using SUB_BUILD=PLATFORM ./gradlew build. Facing below issue. Can anyone help

Peter Li

05/13/2022, 11:07 AM

According to the https://docs.airbyte.com/deploying-airbyte/local-deployment, it doesn't have the hardware requirements section. I think it should need this and let users prepare proper host machine to build their Airbyte locally 👍.

Mikhail

05/13/2022, 11:49 AM

Hi gyus! I am trying to sync salesforce to snowflake, but somehow not all the tables from source support

Incremental

sync mode (i.e. only

Full refresh

is available for some) What could be the reason?

✅ 1

Sam Wiltshire

05/13/2022, 12:25 PM

Hey all, I am trying to set up BigQuery as a destination and I am getting the following error which is stumping me:

Sam Wiltshire

05/13/2022, 12:25 PM

Any help would be greatly appreciated!

Alec Dou

05/13/2022, 1:46 PM

Hi all, I am considering contributing to AirByte and just starting to figure out the code logic. I opened the project in IntelliJ and set up the dependencies, however, it seems some classes are missing. Can anyone help me with this? Really appreciate it!!

Satish Chinthanippu

05/13/2022, 2:27 PM

Copy code

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-workers:test'.
> There were failing tests. See the report at: file:///var/airbyte/airbyte/airbyte-workers/build/reports/tests/test/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See <https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings>

Execution optimizations have been disabled for 8 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD FAILED in 11m 54s
237 actionable tasks: 193 executed, 26 from cache, 18 up-to-date
Gradle was unable to watch the file system for changes. The inotify watches limit is too low.

Satish Chinthanippu

05/13/2022, 2:27 PM

Getting this exception...is it because of gradle version mismatch?

Yoel Ferdman

05/13/2022, 2:47 PM

Hi I'm new here. Does Airbyte offer connectors with instantaneous updates for IOT solutions?

Emily Cogsdill

05/13/2022, 2:55 PM

Hi team! I have a question about a custom connector I am developing… would this be the right channel to post it in? I don’t see any existing channels dedicated to connector development, but maybe I missed it?

Sergey Kornev

05/13/2022, 3:55 PM

Hi there - we are trying to use Airbyte open source to pull data from Salesforce? We've run into issues with the very basics and I wonder if we're missing anything. To give you an example - we try to fetch the Account data and it does not allow us to use BULK API because there is a couple of unsupported fields. We can't exclude these fields since Airbyte seems to only allow all or nothing at this time and so the job falls back to REST API that ends up in a timeout after trying to load the entire table for the first time (400k+ records). Any suggestions on the workarounds?

Andrés O. Arredondo

05/13/2022, 4:02 PM

Hi, I've been having some problems with the update of the Redshift destination 0.3.32. As shown by many issues in github such as https://github.com/airbytehq/airbyte/issues/12843 , https://github.com/airbytehq/airbyte/issues/12460 and https://github.com/airbytehq/airbyte/issues/12265 . I decided to revert to 0.3.27, which was working previously. However, the same error is still showing:

function json_extract_path_text(super, "unknown", boolean) does not exist

Then

JSON schema validation failed.

Debug info: Airbyte Version: 0.35.51-alpha Source: Google Sheets (0.2.9) Destination: Redshift (0.3.27) I think the normalization (dbt file) is not being generated again when I downgrade or upgrade. I am not sure about what to do to completely rollback the configuration.

Garrett McClintock

05/13/2022, 6:45 PM

Is anyboyd else having issues where the normalization completes successfully, but the job itself is marked as failed? I know this isn’t the place for troubleshooting, but I have posted on the discourse forum, and not got a reply. The help from the airbyte team itself is seemingly non-existent. I’ve tried upgrading to the latest verison of airbyte itself and connectors. destination is snowflake - issue is happenign with github, jira and zendesk all the same.

Soufiane Odf

05/13/2022, 7:46 PM

hey, I want to consume a rest api, but can’t find a source for that ?

John Joo

05/13/2022, 10:49 PM

I saw that Airbyte API was not available for Airbyte Cloud. Does anyone know if the CLI (https://github.com/airbytehq/airbyte/tree/master/octavia-cli) works for Airbyte Cloud?

John Joo

05/13/2022, 11:33 PM

I have a bunch of Sentry projects that I’d like to use as sources in Airbyte. So, that’s going to be n Sentry projects. How should I think about the destination in Snowflake? Should I create a new schema for each source? If I have n Sentry projects, n Airbyte sources, then should I have n Airbyte destinations where each destination is a different schema in the Snowflake database?

Thomas Adams

05/14/2022, 2:14 AM

Hi all I just found this tool and want to try it out. I am trying to connect to Snapchat but seems like everything uses Oauth. My main confusion is w/ the redirect uri. I am running airbyte in docker locally. I am not too sure what the redirect uri should be. Or if I need to somehow create an application on the internet w/ a dns redirect or something. New to application development.

Satish Chinthanippu

05/14/2022, 7:18 AM

Hi All, While building Airbyte locally using sudo SUB_BUILD=PLATFORM ./gradlew build on unix, getting below issue. Can anyone help on this

Copy code

Task :airbyte-db:jooq:generateJobsDatabaseJooq
2022-05-14 07:13:50 INFO o.t.d.DockerMachineClientProviderStrategy(isApplicable):58 - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /snap/bin])
2022-05-14 07:13:50 ERROR o.t.d.DockerClientProviderStrategy(lambda$getFirstValidStrategy$4):150 - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
2022-05-14 07:13:50 ERROR o.t.d.DockerClientProviderStrategy(lambda$getFirstValidStrategy$4):152 -     UnixSocketClientProviderStrategy: failed with exception TimeoutException (org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException). Root cause TimeoutException (null)
2022-05-14 07:13:50 ERROR o.t.d.DockerClientProviderStrategy(lambda$getFirstValidStrategy$4):154 - As no valid configuration was found, execution cannot continue
2022-05-14 07:13:51 ERROR o.j.t.JooqLogger(error):454 - Error while closing database
java.lang.NullPointerException: Cannot invoke "org.jooq.DSLContext.close()" because "this.dslContext" is null
        at io.airbyte.db.instance.FlywayMigrationDatabase.close(FlywayMigrationDatabase.java:102) ~[io.airbyte.airbyte-db-lib-0.38.3-alpha.jar:?]
        at org.jooq.codegen.GenerationTool.run0(GenerationTool.java:857) [jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.run(GenerationTool.java:222) [jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.generate(GenerationTool.java:217) [jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.main(GenerationTool.java:189) [jooq-codegen-3.13.4.jar:?]
2022-05-14 07:13:53 ERROR o.j.t.JooqLogger(error):454 - Cannot read /home/satish/airbyte/source/airbyte/airbyte-db/jooq/build/tmp/generateJobsDatabaseJooq/config.xml. Error : Failed to launch postgres container and run migration
java.lang.RuntimeException: Failed to launch postgres container and run migration
        at io.airbyte.db.instance.FlywayMigrationDatabase.getInternalConnection(FlywayMigrationDatabase.java:68) ~[io.airbyte.airbyte-db-lib-0.38.3-alpha.jar:?]
        at io.airbyte.db.instance.FlywayMigrationDatabase.create0(FlywayMigrationDatabase.java:60) ~[io.airbyte.airbyte-db-lib-0.38.3-alpha.jar:?]
        at org.jooq.meta.AbstractDatabase.create(AbstractDatabase.java:274) ~[jooq-meta-3.13.4.jar:?]
        at org.jooq.meta.AbstractDatabase.create(AbstractDatabase.java:263) ~[jooq-meta-3.13.4.jar:?]
        at org.jooq.meta.AbstractDatabase.setConnection(AbstractDatabase.java:253) ~[jooq-meta-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.run0(GenerationTool.java:521) ~[jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.run(GenerationTool.java:222) ~[jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.generate(GenerationTool.java:217) ~[jooq-codegen-3.13.4.jar:?]
        at org.jooq.codegen.GenerationTool.main(GenerationTool.java:189) [jooq-codegen-3.13.4.jar:?]
Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
        at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$4(DockerClientProviderStrategy.java:157) ~[testcontainers-1.17.1.jar:?]
        at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
        at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:149) ~[testcontainers-1.17.1.jar:?]
        at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:147) ~[testcontainers-1.17.1.jar:?]
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:188) ~[testcontainers-1.17.1.jar:?]
        at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:102) ~[testcontainers-1.17.1.jar:?]
        at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:108) ~[docker-java-api-3.2.13.jar:?]
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:325) ~[testcontainers-1.17.1.jar:?]
        at io.airbyte.db.instance.FlywayMigrationDatabase.createInternalConnection(FlywayMigrationDatabase.java:84) ~[io.airbyte.airbyte-db-lib-0.38.3-alpha.jar:?]
        at io.airbyte.db.instance.FlywayMigrationDatabase.getInternalConnection(FlywayMigrationDatabase.java:66) ~[io.airbyte.airbyte-db-lib-0.38.3-alpha.jar:?]
        ... 8 more

> Task :airbyte-db:jooq:generateJobsDatabaseJooq FAILED

Ariyan Majidi

05/14/2022, 9:00 AM

Hi guys, I searched about security options that we have in airbyte orchestration( authentication and authorization), just find out this link https://shey.ca/2022/03/17/oauth2-proxy-with-airbyte.html did anyone try OAuth proxy with airbyte? is it the best option that is currently available? any suggestion? thanks #C02BZS48Q1Z

Thomas Adams

05/15/2022, 3:55 AM

Can we please get a design document for google ads? https://github.com/airbytehq/airbyte/issues/1981

Engineering Team

05/15/2022, 12:45 PM

Hi all,I wanted to know how to enable Basic Normalization ?

Vachagan Gevorgyan

05/16/2022, 2:21 AM

Hi all! Help me plz with this error at mysql source :

Copy code

The connection tests failed.
Could not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (The server time zone value 'MSK' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'serverTimezone' configuration property) to use a more specific time zone value if you want to utilize time zone support.)

Michael

05/16/2022, 2:32 AM

Hello everyone, may I know how to test if our json schema is correct?

Jerry Lee

05/16/2022, 6:13 AM

Hi, I'm using open source version, and I did some steps as below: 1. clone the github repo; 2. run command "docker-compose up"; 3. I've seen the web page through http://ip:8000/ 4. I tried to create a source with Exchange rates API; 5. It's failure when I set up; Please check the log file below, any suggestions?

logs-65818d54-b708-4834-8d78-a0d489bccae1.txt