Hi all, we’re experiencing an onboarding blocker. We’re trying to set up a MySQL db as a source, and BQ as a destination. However, our MySQL DATETIME fields are getting imported as strings into BigQuery. This is not ideal, because we require date fields for our BI tool. It appears that the MySQL connector designed to import dates as strings, looking at the docs: https://docs.airbyte.com/integrations/sources/mysql/#data-type-mapping However, Big Query appears to support importing a date field: https://docs.airbyte.com/integrations/destinations/bigquery/#data-type-map How do we import date types into BQ?

Hi All, I am trying to set up a data lake

I have some worries about to use Google Cloud storage instead of Google Big query, My idea is to store raw data in GCS and after make a transformation to Big query

I will aprecciate some help

This message was deleted.

New install on macOS, trying to connect to a MariaDB instance on the host and it is not working. Says it cannot connect. I tried using host.docker.internal as well as localhost and 127.0.0.1, nothing works. I have seen this asked previously but no clear answers. Could not connect with provided configuration. HikariPool-1 - Connection is not available, request timed out after 60002ms.

After trying to connect, I see it spawned a new container destination-mysql-check-dad4f928-8097-404e-a02c-a4b3e4414eee-0-wsrec. The name changes each time. But while it was up I was able to exec a bash in there, install ping, and ping host.docker.internal so it seems it should have network connectivity back to the host.

And I fired up another docker container that connects to MySQL and had no problems using the host.docker.internal.

It is even stranger perhaps, I can set up MySQL as a SOURCE using my local install via the Airbyte Docker setup, but I can not set it up as a DESTINATION. I get the timeout message.

We are experiencing source schema refresh for large schema (against Oracle DB). Its is running for more than 2 hours and its failing.. Is this known issue?

does anyone know how can i update source schema from octavia ?

Why this link is broken

Hi, I'm trying to use dbt custom transformation and I don't understand why dbt doesnt trigger "dbt deps" before trying to run my commands. I've tried to add 2 custom transformations : first with "deps" and second one with "run" but it looks like the second one is unaware of the first one

Hi… I am trying to deploy locally the latest version in macOs… but it gets stuck on “Waiting for namespace default to be initialized in temporal”

airbyte-worker      | 2022-08-08 16:11:10 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):264 - Waiting for namespace default to be initialized in temporal...
airbyte-server      | 2022-08-08 16:11:15 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
airbyte-server      | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998859829s.

Can someone please help me!

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hello, when i try to sync new streams of freshdesk, I get the exception - ValueError: Client does not know how to read stream

ticket_fields

airbyte version 0.39.42 - alpha

Hello, Please how do i set a cursor field for a MySQL connector? My sync's keep fetching all the data from the source every hour

do we need CREATE schema access for Snowflake destination?.. Incremental load failing but full load completed successfully

2022-08-08 22:34:20 normalization > 22:34:13.474910 [debug] [ThreadPool]: Using snowflake connection "create_SANDBOX__AIRBYTE_ORACLE_BASE_TABLES"

2022-08-08 22:34:20 normalization > 22:34:13.475389 [debug] [ThreadPool]: On create_SANDBOX__AIRBYTE_ORACLE_BASE_TABLES: /* {"app": "dbt", "dbt_version": "1.0.0", "profile_name": "normalize", "target_name": "prod", "connection_name": "create_SANDBOX__AIRBYTE_ORACLE_BASE_TABLES"} */

2022-08-08 22:34:20 normalization > create schema if not exists "SANDBOX"._AIRBYTE_ORACLE_BASE_TABLES

2022-08-08 22:34:20 normalization > 22:34:13.475609 [debug] [ThreadPool]: Opening a new connection, currently in state closed

2022-08-08 22:34:20 normalization > 22:34:14.739019 [debug] [ThreadPool]: Snowflake adapter: Snowflake query id: 01a628ca-0604-9a8e-0000-19c1191bfdd2

2022-08-08 22:34:20 normalization > 22:34:14.739374 [debug] [ThreadPool]: Snowflake adapter: Snowflake error: 003001 (42501): SQL access control error:

2022-08-08 22:34:20 normalization > Insufficient privileges to operate on database 'SANDBOX'

Hi guys, i have a custom connector that was deployed and working, we made an update to its spec file and now its not showing in the UI. Any ideas how to refresh the UI after upgrading a custom connector?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hey Airbyte Team, I am currently using the Airbyte API to setup a connection between a custom source connector and the S3 destination. The connection only works if I specify the JSON schema explicitly in the

syncCatalog

. When setting up the connection via the UI instead, Airbyte fetches the schemas from the source itself (no need to specify them again). Is there a way to replicate this behavior using the API so that I only need to specify the streams' names? Thanks a lot in advance for your support 🙂

Is there a recommended RDS instance size for

airbyte-db

?

is ther any way to encrpyt the secrets stored in the postgres db?

The

airbyte/normalization

image arm64 version might be bad. Attempting to run it on AL2 arm64 gives:

docker run --name norm -it --rm --entrypoint bash airbyte/normalization:0.2.16
standard_init_linux.go:228: exec user process caused: exec format error

I submitted an issue https://github.com/airbytehq/airbyte/issues/15498. Note that testing this on M1 mac will succeed because of emulation I guess.

I'm going thru the getting started document

and for new source, i'm not getting the new fields, it just wants source type

am i doing something wrong

Can I select only some tables in database in the Airbyte UI's connection setting page since it enabled every table in database by default. It would consume a lot of time to disable all of the tables that I don't want to sync.

Hi. I`m new to airbyte and I would like to create new logic for google source for custom query using GAQL to fetch data with out using the date segment for example:

SELECT geo_target_constant.canonical_name, geo_target_constant.country_code, geo_target_constant.id, geo_target_constant.name, geo_target_constant.parent_geo_target, geo_target_constant.resource_name, geo_target_constant.status, geo_target_constant.target_type FROM geo_target_constant

What is the best approach to do so? Is there a place I can look for an example on how to add data to an existing source component since creating a new ones we already know how to Thank you.

Unable to build AppStore connector

Downloading <https://services.gradle.org/distributions/gradle-7.4-bin.zip>
...........10%...........20%...........30%...........40%...........50%...........60%...........70%...........80%...........90%...........100%

Welcome to Gradle 7.4!

Here are the highlights of this release:
 - Aggregated test and JaCoCo reports
 - Marking additional test source directories as tests in IntelliJ
 - Support for Adoptium JDKs in Java toolchains

For more details see <https://docs.gradle.org/7.4/release-notes.html>

Starting a Gradle Daemon (subsequent builds will be faster)
Building all of Airbyte.
/home/ubuntu/.ab/airbyte/airbyte-integrations/connectors
Type-safe dependency accessors is an incubating feature.
> Task :airbyte-integrations:connectors:source-appstore-singer:checkPython FAILED
> Task :airbyte-integrations:bases:base-normalization:checkPython FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python (/home/ubuntu/.ab/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv) verion 3.8.10 does not match minimal required version: 3.9

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-appstore-singer:checkPython'.
> Python (/home/ubuntu/.ab/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv) verion 3.8.10 does not match minimal required version: 3.9

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See <https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 6m 32s
12 actionable tasks: 12 executed