Hi All, getting below error while testing with docker container of my python custom destination , can someone help.

(.venv) sheshan@sheshan:~/Documents/Airbyte/airbyte/airbyte-integrations/connectors/destination-weav-destination$ cat messages.jsonl | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files peeyushweav/weav_destination-custom-python:c4prod write --config /secrets/config.json --catalog /sample_files/configured_catalog.json
{"type": "LOG", "log": {"level": "INFO", "message": "Begin writing to the destination..."}}
{"type": "LOG", "log": {"level": "FATAL", "message": "write() missing 1 required positional argument: 'logger'\nTraceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 11, in <module>\n    DestinationWeavDestination().run(sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/destinations/destination.py\", line 108, in run\n    for message in output_messages:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/destinations/destination.py\", line 103, in run_cmd\n    yield from self._run_write(config=config, configured_catalog_path=parsed_args.catalog, input_stream=wrapped_stdin)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/destinations/destination.py\", line 47, in _run_write\n    yield from self.write(config=config, configured_catalog=catalog, input_messages=input_messages)\nTypeError: write() missing 1 required positional argument: 'logger'"}}

Can we call another API call inside one stream? This is particularly needed on the jira connector. For

issues

stream, the API

/rest/api/3/search

has a limit of 100 rows of changelogs. To get the full history of changelogs, we have to call another API

/rest/api/3/issue/{issueIdOrKey}/changelog

when we know the issue ids with more than 100 changelogs in the history. I'm wondering if anyone has the experience, and how is this solved? Thanks

Hi Team, I am using external PSQL RDS for airbyte and got stucked with this error, Investigated a lot but could find resolutions . It looks like this

CONTAINER_ALREADY_STARTED_PLACEHOLDER

is not getting created within container, Looking at this bash script i feel container will through this error again again unless this command works

touch $CONTAINER_ALREADY_STARTED

. Please help with your assistance 🙏

Done init
PostgreSQL started.
touch: CONTAINER_ALREADY_STARTED_PLACEHOLDER: Permission denied
2023-01-08T22:26:34.633Z        ERROR   Unable to create SQL database.  {"error": "pq: database \"temporal\" already exists", "logging-call-at": "handler.go:97"}
2023-01-08T22:26:34.653Z        INFO    Starting schema setup   {"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:57"}

Does anyone know for the postgres to snowflake connectoin if it's supposed to deliver empty strings as null or

''

?

Hi, I am trying to set up a linkedin ads connection using airbyte open source, following the official instructions here: https://docs.airbyte.com/integrations/sources/linkedin-ads/

Hi, i’m using the JIRA source connector, and I see that there are many missing

custom fields

in table

issues

. According to my data, there are nearly 90% ingested data that lost its

custom fields

. Is there a bug? Airbyte version:

0.40.26

JIRA Source Version:

0.3.1

Hi, I'm trying to set up a connector to Linkedin Ads and I really don't understand what the redirect uri should be. I'm running Airbyte open source on GCP compute engine. There is absolutely no detail in the instructions for what the redirect url is supposed to be. Is it the url of the app i just made on linkedin developer portal? Or do i need to somehow get a url for my google cloud vm? I found this definition on google:

A redirect URI, or reply URL, is the location where the authorization server sends the user once the app has been successfully authorized and granted an authorization code or access token

how does that relate to the purpose of this airbyte connector fetching data into a warehouse? what would i be redirecting a user to?

Hi, I am trying to synchronize a table of my MySQL database that is in Google Cloud SQL and I get an error that I can not understand why because it is very generic. The connection to the database is done through a SSH tunnel and the error received is the following. Any help can be good, thank you very much.

Hi Everyone, I am facing a weird issue which I am literally unable to even figure out. I have two sources configured gitlab and jira and 1 destination Faros (hasura graphql) Sync of both sources is failing with the same exact error message which is below

2023-01-07 12:10:04 source > Syncing stream: issue_custom_field_contexts 
2023-01-07 12:10:08 source > {"errorMessages":["The custom field was not found."],"errors":{}}
2023-01-07 12:10:08 source > Encountered an exception while reading stream issue_custom_field_contexts

Now this should only be causing Jira source to fail, even that should not as I have not selected this stream for sync. Any idea how I can start debugging

Hi Everyone, I am facing a weird issue. I am adding

faros

as a destination in

airbyte

but literally nothing happens. 1. Added new

faros

connector 2. I am configuring Faros as destination as shown in the screenshot and nothing happens and every time I go to destinations I am asked to add a destination I am not sure how to proceed to next step as there is no button to either

save

or

next

hi Attaching the log file where i am getting error while deploying airbyte on Ec2 ubuntu machine TIA

hello everyone! do I understand correctly that column filtering was added in the latest Airbyte version? (https://github.com/airbytehq/airbyte/releases/tag/v0.40.26) is it available in Airbyte Open Source and do I need to do anything else aside from upgrading the Airbyte to get it activated?

UPDATE: I found the reason... Connection has two failed attempts and each attempt pulled some rows. Now I have very important question: Data will be duplicated or not? ORIGINAL MESSAGE: Guys I need help, Airbyte shows incorrect number of emitted and committed records. On screens you can see 515 on Airbyte side, which in fact gives 1034 rows on bigquery. Table was empty. anyway I checked my and Airbyte processing dates:

-- 2023-01-02 19:10:20.191 <- service call/processing start datetime
-- 2023-01-02 19:45:03.546000 UTC <- airbyte min emitted datetime
-- 2023-01-02 21:00:10.844000 UTC <- airbyte max emitted datetime
-- 2023-01-02 21:05:01.717 <- service call/processing end datetime

Hi everyone, I'm relatively new to Airbyte, I've successfully created a connection to MS SQL before, but there was a maintenance on the database, and after the database is working again, the connection kind of broke down and we encountered "Failed to fetch schema. Please try again Error: non-json response". I deleted the source and when I tried to create it again, it shows the non-json response again. If anyone could help that'd be great, thanks!

can we pay someone to make a helpscout.com source?

Hello friends, I am starting with airbyte and I have a problem with a local installation, I added a new postgres source, I ran the connection successfully, after a few minutes the server does not respond, the logs show the following:

2023-01-09 15:50:07 [1;31mERROR[m i.a.s.a.ApiHelper(execute):28 - Unexpected Exception

java.lang.IllegalStateException: Duplicate key 206e4781-61fd-4f0c-ba8f-1a2cd209f920 (attempted merging values io.airbyte.config.ActorCatalogFetchEvent@126b6c70[id=<null>,actorId=206e4781-61fd-4f0c-ba8f-1a2cd209f920,actorCatalogId=6345f8d9-8541-4f6f-9ab1-8706aea73482,configHash=<null>,connectorVersion=<null>,createdAt=1673273811] and io.airbyte.config.ActorCatalogFetchEvent@7ba6bea8[id=<null>,actorId=206e4781-61fd-4f0c-ba8f-1a2cd209f920,actorCatalogId=64c3d44c-0d0d-45c4-8b8e-f0e863bd047a,configHash=<null>,connectorVersion=<null>,createdAt=1673273811])

at java.util.stream.Collectors.duplicateKeyException(Collectors.java:135) ~[?:?]

please help me with this, I really don't know whats happening here

Hello Everyone, First timer here. I was using Airbyte for an ELT from MongoDB to Postgres DB and everything seems to work just fine except for refreshing source schema. For some reason it has 504 Gateway timeout for "/api/v1/web_backend/connections/get" API. Everything works from connection test to migrating the data works perfectly And I couldn't find a good solution on the internet.

Hello Guys, I need some help I am trying to connect my source "google-ads" to my destination "bigquery" • I started this 12:36PM 01/04 January and it is still running after 24hrs • Can any one help me with this? • Whats wrong here? I have added the Screenshots

Hello. When Airbyte fetch the schema of a bigquery table, it convert timestamp to string. Is this a bug or I miss something with the setup? Thanks. And happy new year 🍾

[kubernetes] Hello everyone. I'm having a strange failure with Airbyte deployed to Kubernetes. When trying to setup any connection or source, the

check_connection

calls are retuning a "upstream request timeout" error and I'm currently not able to understand where is it coming from because there are absolutely no errors in the logs, even with DEBUG enabled. I've logged an issue here, any help is appreciated: https://github.com/airbytehq/airbyte/issues/20963. Can anyone tell me how can I further debug this issue? I've exec'd into the server container and it can communicate and connect to all the other hosts (temporal and webapp, for example). I'm stuck and have no idea how to debug it further.

Hi! I have been having issues with airbyte ui, both hosted and cloud. Basically when setting up any new destination/source, any toggle on the setup page does not work, meaning when i click on a toggle it does not turn left or right. For example, I’m trying to setup a new clickhouse destination, there is SSL toggle which is turned off by default. When I attempt to turn on it it does not respond.

hey everyone, we've been using airbyte as a data integration tool for a while now, but recently we're facing some problems with data types in microsft sql server CDC. We have a legacy database with some custom data types that airbyte seems to completely ignore when syncing data. is there a way of capturing these fields?

Hey, I am using LinkedIn AD connector for several months, it was running fine. It started throwing error AttributeError("'NoneType' object has no attribute 'status_code'") since yesterday. What could be the reason for this? I've checked OAuth Client, Secret, Refresh Token are valid and working fine.

Hi guys, I'm getting the following error in the logs and it is causing the web UI to simply show:

Oops! Something went wrong… Unknown error occurred

2023-01-09 210048 ERROR i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception

airbyte-server | java.lang.IllegalStateException: Duplicate key 1ab0e414-c5d6-4920-a4dc-e5830965035a (attempted merging values io.airbyte.config.ActorCatalogFetchEvent@1bf15df7[id=<null>,actorId=1ab0e414-c5d6-4920-a4dc-e5830965035a,actorCatalogId=c1d6532c-068f-427c-a6f5-7e8dd6bdd037,configHash=<null>,connectorVersion=<null>,createdAt=1673286105] and io.airbyte.config.ActorCatalogFetchEvent@476a0ab2[id=<null>,actorId=1ab0e414-c5d6-4920-a4dc-e5830965035a,actorCatalogId=3deaf88f-f287-405c-aa3c-dee60d5ce8a3,configHash=<null>,connectorVersion=<null>,createdAt=1673286105])

I've no clue what to do to fix the issue - can anyone give me a clue please.

Hello Airbyte team, I’m following up on an old issue regarding enabling normalization for custom destinations: https://github.com/airbytehq/airbyte/issues/7229. It seems like this normalization code was either removed or refactored out in v0.40.26. Are we able to use normalization and dbt transforms with custom destinations now?

Wondering whether I'm doing something odd, or whether I need to raise a bug ticket, so any advice before I raise a ticket would be great 🙂 I'm trying to run run DBT transformations with: • Docker image: ghcr.io/dbt-labs/dbt-snowflake:1.3.latest • Entrypoint:

run --selector partner_staging

That then goes and: • Pulls down my github repo and docker image • Builds the image and dynamically creates the

profiles.yml

file The issue I'm facing is that when Airbyte auto builds the

profiles.yml

from the airbyte destination config it seems to auto create output of

prod

and I have no way of overriding it - it looks like this:

config:
  partial_parse: true
  printer_width: 120
  send_anonymous_usage_stats: false
  use_colors: true
normalize:
  outputs:
    prod:
      account: {obfuscated}
      client_session_keep_alive: false
      connect_retries: 3
      connect_timeout: 15
      database: DEV_PP_AIRBYTE_DB
      password: {obfuscated}
      query_tag: normalization
      retry_all: true
      retry_on_database_errors: true
      role: DEV_LOADER
      schema: PP
      threads: 5
      type: snowflake
      user: {obfuscated}
      warehouse: DEV_LOADING
  target: prod

This is problematic when defaulting to prod because I have a heap of jinja in dbt to handle prod/nonprod environments like this:

+database: |
            {%- if  "dev" in target.name -%} DEV_STUDYLINK_ANALYTICS
            {%- elif "prod" in target.name -%} STUDYLINK_ANALYTICS
            {%- else -%} invalid_staging_admit_database
            {%- endif -%}

This isn't an issue if I run DBT myself, because I can have my own profiles output called

dev

or

prod

and inject the

--target

on run. This allows the same code to run across prod/nonprod. But it's a problem for Airbyte as it seems to auto-create the fixed values of

normalise

and

prod

as the target. How can we configure airbyte so that when it does DBT transformations it doesn't auto create the profiles with target

prod

? FYI @Varun Khanna

Stack trace.txt

Hi, I am trying to build java-destination connector. I am unable to build the template. Getting the following error.

Hi I am trying to deploy Airbyte using helm on EKS. Can someone help me to connect an external DB using secrets manager.

repoURL: <https://airbytehq.github.io/helm-charts>

targetRevision: v0.43.18

chart: airbyte

Hi, I managed to deploy airbyte 0.40.27 using helm into AKS. I’m now trying to build the webapp docker image and deploy it (initially unmodified) Built successfully with:

./gradlew :airbyte-webapp:assemble

and pushed to my registry. Set the

webapp.image.repository

,

webapp.image.tag

accordingly Once deployed with helm, the webapp pod enters a crash loop backoff with this in the logs:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/10 12:57:19 [emerg] 1#1: host not found in upstream "$CONNECTOR_BUILDER_API_HOST" in /etc/nginx/conf.d/default.conf:6
nginx: [emerg] host not found in upstream "$CONNECTOR_BUILDER_API_HOST" in /etc/nginx/conf.d/default.conf:6

This seems very similar to https://github.com/airbytehq/airbyte/issues/19642 which I believe should already have been fixed in 0.40.22. Will appreciate some help!