Hi folks, coming back to my post from yesterday (link) I have reinstalled Docker and tried to install Airbyte in a new directory. I am still unable to install the newest version of Airbyte on my MacBook M1, 2020. It seems that there is an issue with java???

airbyte-bootloader                | 2023-01-24 14:12:26 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
airbyte-bootloader                | java.lang.IllegalArgumentException: Invalid version string: latest
airbyte-bootloader                | 	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[guava-31.1-jre.jar:?]
airbyte-bootloader                | 	at io.airbyte.commons.version.Version.<init>(Version.java:37) ~[io.airbyte-airbyte-commons-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.commons.version.AirbyteVersion.<init>(AirbyteVersion.java:15) ~[io.airbyte-airbyte-commons-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$getConnectorRepositoryToInfoMap$3(ActorDefinitionMigrator.java:145) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at java.util.HashMap.merge(HashMap.java:1391) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.Collectors.lambda$toMap$68(Collectors.java:1673) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:?]
airbyte-bootloader                | 	at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1003) ~[?:?]
airbyte-bootloader                | 	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
airbyte-bootloader                | 	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ActorDefinitionMigrator.getConnectorRepositoryToInfoMap(ActorDefinitionMigrator.java:110) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ActorDefinitionMigrator.updateConfigsFromSeed(ActorDefinitionMigrator.java:77) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$migrate$0(ActorDefinitionMigrator.java:59) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.Database.lambda$transaction$0(Database.java:27) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:564) ~[jooq-3.13.4.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.Tools$12$1.block(Tools.java:4926) ~[jooq-3.13.4.jar:?]
airbyte-bootloader                | 	at java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744) ~[?:?]
airbyte-bootloader                | 	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689) ~[?:?]
airbyte-bootloader                | 	at org.jooq.impl.Tools$12.get(Tools.java:4923) ~[jooq-3.13.4.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:616) ~[jooq-3.13.4.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:533) ~[jooq-3.13.4.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.Database.transaction(Database.java:27) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.ExceptionWrappingDatabase.transaction(ExceptionWrappingDatabase.java:31) ~[io.airbyte.airbyte-db-db-lib-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ActorDefinitionMigrator.migrate(ActorDefinitionMigrator.java:57) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ConfigRepository.seedActorDefinitions(ConfigRepository.java:1204) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:70) ~[io.airbyte.airbyte-config-init-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:44) ~[io.airbyte.airbyte-config-init-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.DefaultPostLoadExecutor.execute(DefaultPostLoadExecutor.java:46) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Bootloader.load(Bootloader.java:124) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.40.29.jar:?]
airbyte-bootloader                | 2023-01-24 14:12:26 INFO i.m.r.Micronaut(lambda$start$0):115 - Embedded Application shutting down
airbyte-bootloader exited with code 255
service "bootloader" didn't completed successfully: exit 255

I am really lost and would appreciate a bit of help. Thanks

Hi community, I am having a question: we have connection to 3rd party Snowflake instance which have many tables with a lot of columns. We are interested in only 10 tables and subset of these columns in there. Currently, we are having a separate python connector and running sql query to make joins and extract 1 final table. With Airbyte we would copy all 10 tables to destination and run sql there, which is not optimal, because then we would store unnecessary information + sensitive data. We are still willing to use airbyte for standardisation. Does it make sense to build a customer connector which would do the transformations before uploading everything into destination? I know that it is not ELT anymore, but still. Do you have any other recommendations on how to treat this differently? Thanks in advance!

Hello everyone, I'm using Airbyte to ingest SQL Server CDC events and ran into a strange issue. Some events were fired at the exact same moment, but with different values. As seen in the screenshots, I have two registers with the exact same

emmited_at

values, with different

ab_id

and different values in the column

dt_fim_transacao_preset

with one being

null

. I'm not even sure how to determine which one is the newest value. Does anyone have any insights on what might have happened?

Hey guys

PostgreSQL Database directory appears to contain a database; Skipping initialization 2023-01-24 185421.756 UTC [1] FATAL: could not write lock file "postmaster.pid": No space left on device

Anyone get this error on EKS ?

Hi Everyone, brand new Airbyte user here. I'm looking for help understanding incremental syncs. For context, my goal is to load the

usage_records

stream from the Twilio source. I'd like to do this on a daily basis and get the previous day's usage records. I originally set the

Replication Start Date

to

2023-01-01

, but that caused the first sync to get

usage_records

for the period

2023-01-01

to today. And manually re-running the sync did the same thing. Not sure what

lookback window

does either. Can anyone suggest any guide or documentation so I can try to figure this out and unblock myself?

Hi, I want to transform the data in between source and destination. I am having json in source and custom api connector in destination. But Its showing transformation is not allowed for this connection. But If I set from postgres to postgres its allowing me to do the transformation. Is there any conditions or scenario's where I can transform the data? Please suggest. Attached is the image.

Also I am need to send data to multiple destination. Is there a way to select two desination with single source on single connection? Or Can we add two destination in a way that Source sends data to 1st destination and that act as a source for the second destination. I might be using Kafka queues as a source and Custom API on destination. I need to hit multiple API's on the basis of that. If that is possible or is there any work around that would be perfect for my case. Thanks in advance.

Hi! I have been trying the connector builder UI a few times and now after an upgrade it seems my trials have left the builder in a state that it is unable to recover from. I get an error message when i visit http://localhost:8000/connector-builder. Can I reset the state in any way? This is a test instance so truncating stuff is not a problem. Just need help in how to do it. Thanks

what is the variable to increase the amount of attempt of the works and over component ?

Hi, guys! I'm new using airbyte. I'm trying to make a connection between Facebook Marketing and BigQuery but whenever I try to bring the AdCreative table I get the error of "Too many calls to this ad account". Is there any path I can follow to overcome this problem?

I see that there is a clickup connector - https://docs.airbyte.com/integrations/sources/click-up/ But i dont see it from the airbyte cloud. What do i need to do to get that added?

Is it possible to configure the timeout between retries?

Hey friends, I'm having problems with parquet destinations for some cases, getting the error:

org.apache.avro.AvroTypeException: Could not evaluate union

. It seems that the schema is not being handled properly. I was looking around the channel and it seems this problem was already addressed in this thread. Can you folks share the outcome of it? I do not have access to the ticket 2753

Hi Folks, I'm trying to sync data from Jira issue properties endpoint Is this your first time deploying Airbyte: No Memory / Disk: 8Gb / 16GB Deployment: EKS Airbyte Version: 0.40.23 Source name/version: JIRA Destination name/version: S3 CSV Step: Running Jira Connection Description: The connection is not extracting data from issue properties. According to the Airbyte Jira documentation the issue properties is one of the available endpoints to extract data. The documentation points to the Jira documentation page I checked the api using a python snippet before running the Airbyte Connection and I returned data to me: However after running the Airbyte connection, I got no rows or files from this endpoint... It seems to me, this endpoint requires parameters in order to return data, Am I right? Does Airbyte extract full data from this endpoint without issueid and propertykey parameters?

Hello everyone. I'm trying to deploy airbyte on EKS, but there is a lot of problem. Do you have any resources I can refer to? Thanks

Hello I have k8s cluster placed behind the proxy, If I need access to the internet I need to use

HTTP_PROXY

HTTPS_PROXY

NO_PROXY

while trying to deploy airbyte with helm values look like that:

webapp:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x
global:
  env_vars:
    DUMMY: "DUMMY"
server:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x
worker:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x

While testing the access to yaml file with curl inside airbyte-server

curl -v <https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/seed/source_definitions.yaml>

Im getting good output but while trying to do something with UI airbyte getting crashed [1] Do I need to add more env_vars with java options to use? https://stackoverflow.com/questions/61658908/java-ignores-http-proxy-settings

Hi All SQLState: HY000, Message: Query execution was interrupted, maximum statement execution time exceeded airbyte Stack Trace: java.lang.RuntimeException: java.sql.SQLException: Streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming@7551da2a is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries what is this error in MySQL connector, I'm facing this for the first time for one of the tables. anyone faced this, please help.

Hello, do you know when Google Analytics 4 (GA4) will be available for Airbyte OSS ?

Hey all - having a major issue with our Stripe connector and possible other sources affected Prior to this morning we used to receive fields in Stripe that now are no longer present e.g. Subscriptions -

start_date

Invoices -

status_transitions

Our models depended on these fields quite heavily - It seems we used to receive every available attribute from the Stripe API in our

_airbyte_data

columns and now we only receive what's explicitly specified in the repo schema files (e.g. status_transitions is not present in that file, so we don't receive it) Is this expected behaviour? I would have thought the

_airbyte_data

column should be full and raw, and the schemas should only impact the transformation side of Airbyte

Hi, I'm trying to understand Airbyte's API. I can access some endpoints but others I get

405 Not Allowed

. This is cURL example of what I'm doing

curl -d '{"sourceDefinitionId": "MY_SOURCE_DEFINITION_ID", "workspaceId": "MY_WORKSPACE_ID_HASH" }' -H 'Authorization: Basic YWlyYnl0ZTpwYXNzd29yZA==' -H "Content-Type: application/json" -H "Accept: application/json" -X POST <http://localhost:8000/v1/source_definition_specifications/get>

Basic authentication is done as described in the docs for OSS version I think

• When authenticating to the Configuration API, you must use Basic Authentication by setting the Authentication Header to Basic and base64 encoding the username and password (which are

airbyte

and

password

by default - so base64 encoding

airbyte:password

results in

YWlyYnl0ZTpwYXNzd29yZA==

). So the full header reads

'Authorization': "Basic YWlyYnl0ZTpwYXNzd29yZA=="

hi does airbyte supports SAP Hana connector ?

I suggest changing this channel name from “help” to “Unhelpful.” 🤣

Hi, guys! I'm new using airbyte. I'm trying to make a connection between Facebook Marketing and BigQuery but whenever I try to bring the AdCreative table I get the error of "Too many calls to this ad account". Is there any path I can follow to overcome this problem?

Who can I talk to about AirBytes licensing agreement? I'm building a custom solution for my clients.

Hi team airbyte, I would like to know for connecting S3 Is that possible to sync n tables in one connection? Or it has to be 1 connection to 1 table?

Hi, I don't actually know where to put that, but I thought this could be helpful here. I forked the airbyte repo and ran

SUB_BUILD=PLATFORM ./gradlew build

to build the platform but one of the tests failed (testResolveReferences from airbyte-json-validation). 🐛 The test actually failed because of my Java locale (FR and not EN), thus the assertion did not hold because in my runtime the expected error was in French and not in English

$.prop2: string found, boolean expected

✅ FIX: export the environment variable

export JAVA_TOOL_OPTIONS="-Duser.language=en"

to set the Java locale to EN Hope this helps!

Please let me know if this is not an appropriate question. I'd like to hire a freelancer to help me with Airbyte development (potentially both implementation and development of additional connectors/reports for existing connectors). I'd prefer to find someone who has commits in the main branch of Airbyte or with experience contributing to open source packages. Do you know of anyone? Thanks!

Folks, it’s possible to change the default user/password for the api when deploying via helm chart?