Airbyte #ask-community-for-troubleshooting

11/04/2022, 1:49 PM

Hi Team, I need some guidance. Our env is locked down and I wanted to submit the firewall request but not getting clear information on which ports and destination to open so that airbyte is able to download the connectors images. I am getting error message that it is couldn't find image airbyte/postgres 😕

Henri Blancke

11/04/2022, 1:58 PM

👋 I've noticed since upgrading to Airbyte

v0.40.16

that when adding a new S3 connection, fetching the schema often results in

504 Gateway Timeout

. Trying again a few times often resolves the timeout. I wonder if the proxy's nginx default timeout should be increased to allow more time for dynamically fetching schemas like with the s3 connector? As always thanks for the help 🙂

Arthur Mbx

11/04/2022, 2:10 PM

Hey all , got a question is it possible to make a new connection and start it from x point in time as opposed to having to load the entire table from scratch ? I'm trying to replace a bash script that currently does the EL part and it works fine , but i dont want to have to reload 900Gb worth of data to do so with Airbyte

Arthur Mbx

11/04/2022, 2:10 PM

Basically setting the cursor to X

Kevin Peters

11/04/2022, 3:12 PM

Hi everyone, I'm in the process of updating the Mailchimp connector to bring in Automations, Segments, Members, Unsubscribes, and Sent-To streams. The first four are pretty easy but to allow for incremental sync on the Sent-To stream, I need the

cursor_field

to be based upon a field in the parent stream class. For example, the path is

"reports/{campaign_id}/sent-to"

and the cursor needs to be the

send_time

value in the Campaign. Does anyone know of an existing connector that has a similar situation? Or any other advise on how to proceed?

Venkat Dasari

11/04/2022, 3:36 PM

Hello Airbyte Professionals. I am setting up postgres, MSSQL server as my input sources on AWS RDS instance. How can i authenticate to them without giving the passwords on the Airbyte UI? Also can Airbyte run on 443? How to configure logs from Airbyte to write them to an S3 bucket?

Victor Bruno Castro

11/04/2022, 6:03 PM

I'm not able to connect do airbyte docs. When I search air airbyte docs on google this link is the first https://airbytehq.github.io/ not https://docs.airbyte.com/ Everything is ok with the site? 🤔

🚨 3

Christopher Brunton

11/04/2022, 7:04 PM

Hello Airbyte enthusiasts! I am working on setting up a connection that needs to decrypt (PGP) the data as it pulls it from PostGres. Does anyone have some documentation recommendations or git examples? I'm having trouble getting this one implemented. Any help would be much appreciated.

Adham Suliman

11/04/2022, 8:04 PM

I have a pull request open for CallRail. It’s going to be merged next week, which I’m perfectly fine with. My issue is that right now, I have GitHub actions running with errors continuously recurring. Any advice on how to turn off these actions without screwing with the pull request?

Nicolas

11/04/2022, 9:24 PM

Hi folks anyone seen this error before?

Copy code

java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!

i get this error every time i try to sync something bigger then 1-2 million rows and i cant really find what is going on . I am doing a Postgres -> MSSQL connection , its pretty simple , its 4 tables , and i want them to be dedup incremental . the 2 smaller table, works like a charm the bigger ones always get this error even if i try different modes or combinations of tables , and all tables get reset everytime it happens 😞

Zaza Javakhishvili

11/05/2022, 12:09 AM

Guys, using API I create source/connection, process data and delete source (logical connection will be deleted auto). But noticed that they still exist, when I request them by id. So, how to force delete logs/job and etc. I know I can delete volumes, or connect database and delete records manually. But I wish force it using API.

Marcelo Santoro

11/05/2022, 1:07 AM

Hey Guys... I am facing a problem trying to ingest Google Ads data to Bigquery...

Copy code

Database Error in model customer__data__campaigns (models/generated/airbyte_incremental/google_ads/customer__data__campaigns.sql)

Bad int64 value: 0.0

Does anyone knows how to resolve this issue ? 😕

Brian Castelli

11/05/2022, 2:55 AM

Is there a way to stop AirByte from emitting the AirByte-specific columns to CSV? I'm talking about, for example, _airbyte_ab_id
and

_airbyte_emitted_at

. I am having to strip them out before I do data processing. I would prefer not writing special-case code for when I'm using AirByte versus other things. Any ideas?

Faris

11/06/2022, 7:30 AM

I just wanted to check, I was under the impression that airbyte allows partial replication (like I can diselect certain columns from being replicated to my destination). Honestly I left the startup I was working for where I setup airbyte for them for sometime so I didn’t check airbyte UI in the latest version and now I have a new client where I recommended airbyte and wanted to confirm if I could select the columns that I want to replicate.

Bhupesh Varshney

11/06/2022, 7:50 AM

Hey folks, needed some help while setting up Oracle DB as a source, I have a ATP host on Oracle Cloud Free tier, but the connection is failing, i am guessing its due to certificate issue, but not sure how to proceed I have tried NNE, but it didn't work, i am guessing something needs to be changed on the host side?

Muideen Muibi

11/06/2022, 6:38 PM

Hi all, I am totally new to airbyte but can't seem to get going with the installation. I get a 401 error when I launch airbyte (http://localhost:8000/) for the first time after installation, I can't seem to find a way to log in with the fdefault credentails (airbyte and password) as described in the doc here. What could I be doing wrong?

Dusty Shapiro

11/06/2022, 7:29 PM

Airbyte deployed to K8s error, seems like it’s getting a container init error for the normalization pod. Log line before it times out:

Copy code

Log4j2Appender says: Attempting to start pod = normalization-normalize-541-0-pelpm for airbyte/normalization:0.2.23 with resources io.airbyte.config.ResourceRequirements@fa284d9[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]

When I

kubectl get pods

, I see that the above pod is getting an

Init:Error

The init container in describing that pod

Copy code

State:          Terminated
      Reason:       Error
      Exit Code:    1

Not sure where I should go from here as far as investigating the normalization pods that get stuck initializing 🤷 I rolled back the Helm chart to

0.40.35

, but still erroring out.

Mikhail Masyagin

11/06/2022, 8:20 PM

Hey friends! I'm trying to get streams schema for my connector via API:

Copy code

import requests
<http://requests.post|requests.post>('<http://localhost:8000/api/v1/scheduler/sources/discover_schema>', json={
    "workspaceId": '75bd5142-ed9f-45ad-bf05-8d663a74dc21',
    "sourceDefinitionId": '6e1eba1c-7849-4818-ab74-6d990329fedc',
    "connectionConfiguration": {
    'app_id': 'taxi.android.client',
    'languages': {'type': 'all'},
    'start_date': '2022-09-01',
    'timeout_milliseconds': 1000,
    'max_reviews_per_request': 100}
}).json()

and get response:

Copy code

{'message': 'Internal Server Error: Cannot invoke "java.util.UUID.toString()" because the return value of "io.airbyte.config.SourceConnection.getSourceId()" is null',
 'exceptionClassName': 'java.lang.NullPointerException',
 'exceptionStack': ['java.lang.NullPointerException: Cannot invoke "java.util.UUID.toString()" because the return value of "io.airbyte.config.SourceConnection.getSourceId()" is null',
  '\tat io.airbyte.server.scheduler.DefaultSynchronousSchedulerClient.createDiscoverSchemaJob(DefaultSynchronousSchedulerClient.java:128)',
  '\tat io.airbyte.server.handlers.SchedulerHandler.discoverSchemaForSourceFromSourceCreate(SchedulerHandler.java:264)',
  '\tat io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceDiscoverSchema$80(ConfigurationApi.java:755)',
  '\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:845)',
  '\tat io.airbyte.server.apis.ConfigurationApi.executeSourceDiscoverSchema(ConfigurationApi.java:755)',
  '\tat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)',
  '\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)',
  '\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)',
  '\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)',
  '\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)',
  '\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)',
  '\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)',
  '\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)',
  '\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)',
  '\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)',
  '\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)',
  '\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)',
  '\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)',
  '\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)',
  '\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)',
  '\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)',
  '\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)',
  '\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)',
  '\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)',
  '\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)',
  '\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)',
  '\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)',
  '\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)',
  '\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)',
  '\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)',
  '\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)',
  '\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)',
  '\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)',
  '\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)',
  '\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)',
  '\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)',
  '\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)',
  '\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)',
  '\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)',
  '\tat org.eclipse.jetty.server.Server.handle(Server.java:501)',
  '\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)',
  '\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)',
  '\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)',
  '\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)',
  '\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)',
  '\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)',
  '\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)',
  '\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)',
  '\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)',
  '\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)',
  '\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)',
  '\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)',
  '\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)',
  '\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)',
  '\tat java.base/java.lang.Thread.run(Thread.java:1589)']}

It looks like API is broken for this link Airbyte version is 40.0.17

Lachlan Wells

11/07/2022, 4:41 AM

Hi there! Is there an unauthenticated health check endpoint that can be used with Airbyte Open Source? I have Airbyte deployed on an EC2, fronted by an ELB, however the Target Group is currently evaluating the health of the Airbyte instance and receiving a 401 from

/v1/health

Manish Tomar

11/07/2022, 7:39 AM

How much load airbyte instance T3. Medium EKS cluster can handle ? Any guide on that?

Srinidhi krishnamurthy

11/07/2022, 7:51 AM

Hello Team, we are currently using airbtyte setup as docker compose in AWS single EC2 prod setup ? would like to upgrade the setup to HA. we see EKS could be a possible solution for Airbyte HA , would like to understand following 1. is there a constraint on airbyte in EKS to be deployed on single eks worker ? or can we deploy in multi worker cluster of eks ? 2. can we have multiple replicas of temporal , worker , server and webapp ?

Avi Sagal

11/07/2022, 8:10 AM

Hi, I’m new to Airbyte and i’m trying to create connections between a GA source and a postgres destination via API. is there a way to enable the Normalized tabular data transformer via API? in the UI it looks like this: Thanks!

Rahul Borse

11/07/2022, 8:51 AM

Hi Team, is there any way we can secure airbyte UI using keyclock? I am following below document but I am stuck at keyclock gatekeeper where we need to write config.yaml file but I am not able to understand where we need to create that config.yaml file. Did somebudy tried securing airbyte UI using keyclock and what are steps? https://www.beyondbroadcast.com/post/securing-websites-and-apis-with-keycloak-and-keycloak-gatekeeper/

iAastha S

11/07/2022, 9:04 AM

hello all, We are setting up a new instance of airbyte…. and its throwing error on docker-compose up

Copy code

docker-compose up
WARNING: The DEPLOYMENT_MODE variable is not set. Defaulting to a blank string.
WARNING: The REMOTE_CONNECTOR_CATALOG_URL variable is not set. Defaulting to a blank string.
WARNING: The TEMPORAL_HISTORY_RETENTION_IN_DAYS variable is not set. Defaulting to a blank string.
WARNING: The UPDATE_DEFINITIONS_CRON_ENABLED variable is not set. Defaulting to a blank string.
ERROR: Invalid interpolation format for "environment" option in service "worker": "CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}"

✅ 1

Dave Tomkinson

11/07/2022, 11:47 AM

Hi, I'm struggling importing into Redshift Serverless. I used Airbyte to Export as Parquet to S3 then did a Redshift COPY to load the data (some of the data is coming cross account and it was easier to set up cross account S3 bucket than DB access). But I can't seem to find a valid data type to load `_airbyte_additional_properties`; I've tried SUPER, VARBYTE, VARCHAR(Max). The VARCHAR(MAX) gives a sensible error message saying incompatible parquet schema for column. The other two just fail with seemingly random .cpp errors. Any other suggestions for column data type? I'm starting to think Redshift Serverless is more like Redshift Spectrum than actual Redshift (so data types which say 'this doesn't work with Spectrum' are out?) Any ideas, suggestions or workarounds much appreciated. Parquet for

_airbyte_additional_properties

comes out as

_airbyte_additional_properties: map<string, string ('_airbyte_additional_properties')>

child 0, _airbyte_additional_properties: struct<key: string not null, value: string not null> not null

child 0, key: string not null

child 1, value: string not null

(This also seems to be affecting a timestamp field I'm using, which comes through as:

struct<member0: timestamp[us, tz=UTC], member1: string>

child 0, member0: timestamp[us, tz=UTC]

child 1, member1: string

)

Cesar Santos

11/07/2022, 12:16 PM

Hey folks! Do you have a list of companies that are currently using Airbyte? I searched for it in https://airbyte.com/, but I could not find it.

Alexandre Voyer

11/07/2022, 12:31 PM

Hi there! I'm currently using airbyte open source and I'm wondering if there's a way to estimate the number of gig I use per sync to approximate the cost of airbyte cloud

Rafael Silva

11/07/2022, 12:52 PM

Hello guys, my name is Rafael Silva and I'm a data engineer. I've been using Airbyte for a few months and today I came across an error that unfortunately I can't fix. In the case below we are trying to synchronize the Pipedrive connector with the PostgreSQL destination and it returns this normalization error. Following is a print of the error.

João Larrosa

11/07/2022, 1:12 PM

Hi, mates! Anything about this?

Gabriel Martelloti

11/07/2022, 1:55 PM

Hey guys, how is it going? We are currently using Airbyte inside a K8s stack and we are having some issues with a specific connection. Our source has approximately 150 tables and the whole ELT process takes 11 minutes, but that time is high considering the business needs and we are trying to reduce it. We tried some parallelizing parameters inside the docs, like setting a cpu limit for the airbyte worker, increasing the replica value, setting the max workers variable, but that didn't make any change. My question is: Is it possible to currently parallelize just a single ELT task or does the parallelize parameters only work for a bunch of different tasks?