I have gotten a bit stuck regarding where my issue is and could use some pointers, I have managed to create an custom integration to an API that we are using, I have also managed to make it work for 2 of the streams, but when I try to add the third stream and I get the following error when I run read: Unexpected type for data_or_message: <class 'str'>: employers" , the schema is fully extended 3800 lines

{
    "type": "object",
    "properties": {
        "employers": {
            "type": "array",
            "items": {
                "$ref": "CompanyUserProfileEmployerModel"
            },
            "nullable": true
        }, ...

and in my reply I am getting

{"type": "DEBUG", "message": "Receiving response", "data": {"body": "{\"employers\":[{\"profileId\":122728,\"startDate\":\"2010-01-01T00:00:00\",\"endDate\":\"2012-09-30T00:00:00\", ...

(posted this message first in the wrong channel I believe so moving it here), So where do troubleshoot next, not really sure what is wrong here.

Which java version is required? https://docs.airbyte.com/contributing-to-airbyte/developing-locally/ says about 1.7 but I guess it's different

Hi All, I just wanted to check on the status of a PR (not sure if this is the right place to do that), but the issue is holding me back from finishing a pipeline and would love to get it checked off. Postgres Source : “Support JSONB datatype #21695” any update would be super appreciated!

Hello lovely people! We have facebook marketing data going into our database with airbyte. This may be a long shot but does anyone know if it’s possible to get the instagram post associated with a facebook ad? Either its permalink or its id from https://developers.facebook.com/docs/instagram-api/reference/ig-media ta

Hi, after deploying airbyte in GKE airbyte-server is stuck in ContainerCreating. And the error message is as follow:

FailedMount    kubelet     Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[airbyte-volume-configs gcs-log-creds-volume kube-api-access-c648m]: timed out waiting for the condition

I’ve tried changing the accessModes from ReadWriteOnce to ReadWriteMany in volume-configs.yaml as follow:

...
spec:
  accessModes:
    - ReadWriteMany
...

Any help would be appreciated.

Hi! Will the value of

_airbyte_emitted_at

be same for all the records loaded in the target for a specific load?

How do you get help here? i havent gotten any answers to my questions

Newb here, but trying to install airbyte and I get this from the docker Creating Container airbyte-connector-builder-server Creating Error response from daemon: invalid mount config for type “bind”: bind source path does not exist: /data/compose/6/flags.yml For context, I’m trying to install it from portainer inside a synology nas

Hi All, I am not able to make snowflake connection from Airbyte webapp UI, deployed using helm chart version 0.40.10, we have no S3 or Postgres in deployment or architecture, when we try creating the connection says connection tests failed SOME LOGS L2 �v�$���A�{Rֿ*�%M<��J��Type�V2Obj��ID��DDir���u��G�\ͱ�{æEcAlgo�EcM�EcN�EcBSize��EcIndex�EcDist��CSumAlgo�PartNums��PartETags��PartSizes�� ��PartASizes�� ��Size� ��MTime�A�{Rֿ*�MetaSys��x-minio-internal-inline-data�true�MetaUsr��content-type�application/octet-stream�etag� 99f242f34417a555b2ff8ed44da589d6�v�c�{��O%j���null� �\�B���g�!/�L��>�Z%�tq��$�&2023-02-08 182130 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.9 2023-02-08 182130 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation 2023-02-08 182130 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=55aced06-5d08-37fe-a540-4bfc55147a24, activityType=RunWithJobOutput, attempt=1 java.lang.NullPointerException: Cannot invoke "java.lang.Boolean.booleanValue()" because the return value of "io.airbyte.persistence.job.models.IntegrationLauncherConfig.getIsCustomConnector()" is null at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$getWorkerFactory$2(CheckConnectionActivityImpl.java:143) ~[io.airbyte-airbyte-workers-0.40.29.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:147) ~[io.airbyte-airbyte-workers-0.40.29.jar:?] at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:118) ~[io.airbyte-airbyte-workers-0.40.29.jar:?] at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Hi, Airbyte have any connector to create

Incoming Webhooks

?

could someone please review this PR? https://github.com/airbytehq/airbyte/pull/22527

Mixpanel to Redshift. Incremental stuff is not working for me. I ran the same job again with same parameters. Records doubled. In incremental, it should not insert the data that's already there right. Anyone knows how to change the cursor field which is reference column during incremental load?

When did we lose the ability to transform/normalize data in a Hubspot source connection? I have a Hubspot ---> Postgres connection in an environment that hasn’t been touched in awhile, and it seems like it runs normaliztion post sync, but attempting to setup the same connection as above shows this msg:

Normalization and Transformation operations are not supported for this connection.

Hi! I am running into an issue with the Zendesk connector (airbyte version 0.40.32, zendesk connector version 0.2.21). It won't reset and sync the tickets table (Reset only successful after turning the tickets table off). Everything else syncs without problems. Logs are in the comments.

it looks like the slack source is returning records that have fields that weren't part of the schema. For example the

channel_messages

stream sometimes returns

edited

and

files

property however it's not defined as part of the schema: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/schemas/channel_messages.json This is causing issues with the Weaviate destination, is this considered a bug with the Slack source? CC @EJ Oruche

Hello. The source documentation for the Hubspot connector provides incorrect scopes. Here is a PR with update: https://github.com/airbytehq/airbyte/pull/22618

how add .pem while creating source as MongDB? Plz help

This is my first day using AirByte and Docker. I’ve run into this issue https://github.com/airbytehq/airbyte/issues/20703 while testing it out. I am trying to setup source=SalesForce and Dest=MySQL and I hit that error. I found in the code where the error is coming from

File "/airbyte/integration_code/source_salesforce/source.py", line 95, in generate_streams

and would like to try and understand the problem better so I modified that file to output more debug information. How do I get the updated code to take effect so I can run the test again and get better debug information about the issue?

Hi Everyone, I have raised a PR for Instagram Source Connector. Please review this PR. https://github.com/airbytehq/airbyte/pull/22365 CC: @Jerri Comeau (Airbyte)

Hi All Can anyone please tell me know , how to add .pem file for authentication of source (MongoDB) in airbyte. if the .pem file to be added to any docker container or how it is to be done. Can someone please let me know.

Hello Everyone 👋 Beginner here ✋ I am running a stand-alone instance of airbyte using docker. We have some big tables that we need to refresh often all of them are from the same source (

mssql

). I have kept the env variable set to default. Should I run all the streams in the One job or should try to split them to take advantage of the job concurrency?

Hi, can anyone tell me where to find the channel called #dev or #general mentioned in https://docs.airbyte.com/contributing-to-airbyte/

does any one know why airbyte-db failed when I execute docker-compose up. Error msg is :

Do

dbt

transformations work with a custom source connector ( the connector will POST to REST APIs) ?

Suppose, I run a data migration, example mixpanel to redshift. I transferred 10 Million records. Data is loaded on to the target. Now will airbyte in its docker, store any cache or temporary data which we can clean off after the data load? How is memory management done?

Hi! Anyone has current experience with the Notion source? I was trying 1.0.1, and it pulls data fine, but if I enable basic normalisation, it fails, even on the stream reset stage (so my hunch is it’s configuration, rather than data issue?) The end is something like, but happy to get more logs out if useful.

2023-02-09 13:15:05 normalization >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/utils.py", line 121, in <genexpr>
2023-02-09 13:15:05 normalization >     data_type.WELL_KNOWN_TYPE_VAR_NAME in option[data_type.REF_TYPE_VAR_NAME] for option in properties[data_type.ONE_OF_VAR_NAME]
2023-02-09 13:15:05 normalization > KeyError: '$ref'

The

users

table normalises fine, it’s the

databases

or

pages

tables that breaks this way (I think both/either does). Any ideas?

Hi! Could someone review these two PRs, please? https://github.com/airbytehq/airbyte/pull/21524 https://github.com/airbytehq/airbyte/pull/19224

This message was deleted.

how can I clone airbyte repo I tried to clone however some file couldn't be cloned because the file name is too long

Hi everyone! First-time user here, so any help appreciated! I'm having issues with a MSSQL -> BigQuery connection that syncs tables with

Binary Data

type columns. The normalization step fails with this error:

Invalid cast of bytes to UTF8 string

Seems like dbt's trying to cast bytes into string and BigQuery doesn't like it. Anyone had the same issue? Any idea on how to solve this?