Hello, is it correct commit ? I see hard-coded version 0.30.23 in Dockerfile (current version in master already 0.30.25)

commit b1f5c23a6ffb11572a0b141f6aa93cbebc6c8a4a
Author: Jared Rhizor <jared@dataline.io>
Date:   Tue Nov 2 09:45:54 2021 -0700

    use ADD to reduce docker image sizes (#7537)
    
    * use ADD to reduce docker image sizes
    
    * switch to full paths

Hi question on iterating on an existing source connector. I have added some new fields and a new stream to a source, I ran it locally with the python commands read etc. and it worked fine pulling in the new data - however when I run docker-compose up in the main airbyte directory the connector is still in its old state. How can I get the connector to represent the changes in the UI?

Hello, everybody! Is it correct statement after https://github.com/airbytehq/airbyte/pull/7501/files was merged we should bump connector version

airbyte-config/init/src/main/resources/seed/destination_specs.yaml

or here

airbyte-config/init/src/main/resources/seed/source_specs.yaml

as well? Thanks.

Hi, everyone, does anyone know with what this issue could be related?

Exception in thread "main" java.lang.RuntimeException: Failed to fetch valid spec file for docker image airbyte/source-drift:0.2.4 from GCS bucket io-airbyte-cloud-spec-cache
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.lambda$fetchSpecFromGCS$4(SeedConnectorSpecGenerator.java:120)
        at java.base/java.util.Optional.orElseThrow(Optional.java:401)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.fetchSpecFromGCS(SeedConnectorSpecGenerator.java:120)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.lambda$fetchUpdatedSeedSpecs$3(SeedConnectorSpecGenerator.java:113)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.fetchUpdatedSeedSpecs(SeedConnectorSpecGenerator.java:114)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.run(SeedConnectorSpecGenerator.java:82)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.main(SeedConnectorSpecGenerator.java:72)

1. do you need to perform a full-refresh in order to take advantage of the new incremental normalization? (apologies in advance if this is in the docs somewhere). i assume so given

_AIRBYTE_AB_ID does not exist. The table needs to be rebuilt in full_refresh

but just want to confirm there’s no workaround as its a pretty big task 2. are the dbt logs removed?

Why do we have a bunch of tagged versions without any releases?

Hi All, why are there no primary key constraints placed in Airbyte generated final tables? even when primary keys are specified in the connection setting? if they are not in the table, how is Airbyte deduplicating? I see it is indeed deduplicating.

Oh, only the data in final table is deleted. We would need a way to delete all data and tables, for a connection in a destination. Not a big deal, I can hard code it. But would be nice if Airbyte took care of it, since it knows the intermediate tables it created.

❓ Does Airbyte offer anything around Audit Logging? Thank you!

👋 I'm struggling running

SUB_BUILD=PLATFORM ./gradlew build

on the master branch. I'm on a Macbook M1 and I've set some specific env var as suggested in the doc (you can find it below). The build is failing on the

airbyte-webapp:npmInstall

task with the error below. I installed node 16 and logged to npm with

npm login

. Env vars:

export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export JDK_VERSION=17
export NODE_VERSION=16.13.0
export PATH="/opt/homebrew/opt/node@16/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/node@16/lib"
export CPPFLAGS="-I/opt/homebrew/opt/node@16/include"

Error:

npm ERR! code E401
npm ERR! Incorrect or missing password.
npm ERR! If you were trying to login, change your password, create an
npm ERR! authentication token or enable two-factor authentication then
npm ERR! that means you likely typed your password in incorrectly.
npm ERR! Please try again, or recover your password at:
npm ERR!     <https://www.npmjs.com/forgot>
npm ERR!
npm ERR! If you were doing some other operation then your saved credentials are
npm ERR! probably out of date. To correct this please try logging in again with:
npm ERR!     npm login

Hi All, I’m Welly from scifin.tech. I’m looking into singer and found out that airbyte is inspired on that. That’s a great one. However i see that the communication between source and sink still using json via I/O. Is there any plan on changing that into more performant model like arrow flight etc? Previously we are using Spark for ingestion and utilize the parallization there to put the data into parquet format. any thought on this?

@Harshith (Airbyte) I just came across your cli. Looks very cool! Is this up to date? What is the yaml that this accepts? Does something like this help us solve the yaml config part of this thread? https://airbytehq.slack.com/archives/C01A4CAP81L/p1615928383010200

Hey all! I've just started playing around with Airbyte and was curious about something. Is there a way to reset an incremental sync to a particular position/point in time? My only guess would be to manually update the record that is stored in the database that holds that offset, which feels wrong. Thank you!

Hey, has there recently been a change in airbyte, I am using a psql-psql pipeline and it has recently stopped giving me the logs of the replication progres, as in x rows have been read. Will post a formal issue for this as well

Hi all, If I wanted to use pydantic models in my source, how can I get it to work with `$ref`s

Keeping in line with the pydantic models, anyone knows how to declare properties of a BaseModel in such a way as to produce a

"oneOf": [{…}, {…}]

type structure? I’ve tried different approaches and the closest I got was an

"anyOf": [{…},{…}]

structure using

account_ids: Union[AccountConfigStrategy1, AccountConfigStrategy2] = Field(...)

Hi everyone, I was wondering if you could help me with a syncing issue. I am currently syncing data to my data warehouse but had to resync it twice. With resyncing the data twice, the data warehouse has kept the data from both syncs so I am dealing with duplicate entries. Do you know how I can solve this?

Hi, I want to create workflow postgresql CDC -> S3 -> Redshift raw schema -> Redshift processed schema. From Airbyte documentation ai will need to create 3 flows: 1. Postgresql -> S3 2. S3 -> Redshift raw schema 3. Redshift raw schema -> Redshift processed schema. Is it possible to create dependency on those 3 flows inside Airbyte?

Hello 🙂 I’m running

/test connector

command for google search console source in this PR and it fails: • https://github.com/airbytehq/airbyte/actions/runs/1460805371 • https://github.com/airbytehq/airbyte/actions/runs/1460954249 I found this answer on stackoverflow and it looks like there is a shortage of virtual memory. Is it something that could be fixed by updating github workflows or the only way is to reduce amount of records?

When we setup a connection, it asks for the frequency, and the first sync starts immediately after the connection is created. Is it possible to set specific timestamp in future from which the first sync would commence instead of immediately ?

Folks I have a question about new “advancedAuth” specification is it ok that ? on Java part we have camelCase

oauthUserInputFromConnectorConfigSpecification

https://github.com/airbytehq/airbyte/blob/cb9c5123b1789310ee313c2823868ecba65aa35f/airbyte-api/src/main/openapi/config.yaml#L3259 on Python part we have snake_case

oauth_user_input_from_connector_config_specification

https://github.com/airbytehq/airbyte/blob/cb9c5123b1789310ee313c2823868ecba65aa35f/airbyte-cdk/python/airbyte_cdk/models/airbyte_protocol.py#L122 I see that for old deprecated specification “authSpecification” we have camelCase for both python and java

1. Is it already possible to run Airbyte with a CloudSQL database instead of a database inside the cluster?

2. Is it possible to run Airbyte without persistent volume claims? Can it run stateless?

i’m currently on version

0.30.23

(pre- new incremental normalization) i’d like to upgrade to

v0.32.25

to try out the recent changes made there, so i need to first upgrade to

v0.32.0

to facilitate a safe migration. question: if i encounter issues with

v0.32.25

can i immediately revert back to

0.30.23

or will there be backward incompatible changes?

Hi! I am working on #5712. The main purpose of this task is to support all possible date/date-time formats in the destinations. Currently, some formats are not supported and we get errors during the normalization phase: "Date '1/1/2021' is not recognized". We agreed with the team to implement some converter that recursively goes through the

JsonNode

from

AirbyteMessage

, reads the date/date-time value, converts it to a valid ISO-8601 format, and replaces the value in the

JsonNode

. Can you suggest a place in the code where we can do this job? P.S. A possible place to use this solution is

io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer#acceptTracked

method.

How do I get the airbyte:normalization image to be pulled from local docker? I am trying to test some local changes I made to the normalization project but even after following these steps:

SUB_BUILD=CONNECTORS_BASE ./gradlew clean build 
VERSION=dev docker-compose u

I notice in the airbyte logs that it still pulls the airbyte:normalization image from remote:

2021-11-26 22:37:46 INFO () DefaultNormalizationRunner(runProcess):135 - Running with normalization version: airbyte/normalization:0.1.60
2021-11-26 22:37:46 INFO () LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.60 exists...
2021-11-26 22:37:47 INFO () LineGobbler(voidCall):82 - airbyte/normalization:0.1.60 not found locally. Attempting to pull the image...

Any ideas on this? Thanks!

Hi, I have a question. Is it possible to run custom dbt transformation independently(not right after sync)?

Hi, I'm planning to contribute by configuring the gcp bucket and credentials in the helm chart : https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/templates/env-configmap.yaml#L20 Does it seem ok to the <!subteam^S01M74AUQSE|@team> ? I don't know if I can create a PR or if you prefer being asked before. The exact change I want to make :

GCP_STORAGE_BUCKET: {{ default "" .Values.gcpLogs.bucket }}
  GOOGLE_APPLICATION_CREDENTIALS: {{ default "" .Values.gcpLogs.credentials }}

Hi. We have a data in Hive and we want add BigQuery to using Airbyte. Is it possible? I search but I am not certainly find answer.

4k commits ✨