Johannes Müller
02/10/2023, 1:25 PM.env
file's WEBAPP_URL
which I did, but the webserver is still on the original port 8000:Dusty Shapiro
02/10/2023, 1:35 PMAlexander Ettingshausen
02/10/2023, 1:36 PMJusten Walker
02/10/2023, 2:16 PMDany Chepenko
02/10/2023, 3:50 PMLuiz Aléssio Cesa
02/10/2023, 5:03 PMAdrian Bakula
02/10/2023, 5:21 PMSam Stoelinga
02/10/2023, 5:21 PMJohn Munson
02/10/2023, 6:43 PMpublic
.
This isn't specifically addressed in the docs, but using the .env file I was able to use the currentSchema parameter in the database connection string to accomplish this. Something like...
DATABASE_URL=jdbc:<postgresql://db:5432/airbyte?currentSchema=airbyte_metadata>
Airbyte successfully connects to the airbyte_metadata
schema and bootstraps the necessary tables. The problem though seems to be that as Airbyte continues initializing/bootstrapping, there are things further downstream that still expect to find these tables in the public schema.
Has anyone else come across this issue? Or does it seem that I have something configured wrong and this should totally be possible? Thanks!
(by the way, if it wasn't clear, I am using airbyte open source)Sharath Chandra
02/10/2023, 6:53 PMCamilo Atencio
02/10/2023, 6:55 PMAdditional Failure Information: message='io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.', type='java.lang.RuntimeException', nonRetryable=false
. Its not resources related, its the only job i run. Any other run without normalization will be succesful. If you look at the logs, it seems like the normalization runs fine. Actually I see data in the table. So im not sure why is the normalization being marked as failed. Anyone has any ideas?Wesley Gormley
02/10/2023, 7:04 PMSFTP Bulk
is not included in the default connectors?mohd shaikh
02/10/2023, 7:09 PMMarco de Nijs
02/10/2023, 7:55 PMNag R
02/10/2023, 8:08 PMFrancisco Viera
02/10/2023, 9:04 PMJake Vernon
02/10/2023, 10:36 PMJake Vernon
02/10/2023, 10:37 PMwebapp_url
in the .env and I did configure that to use my domain but that doesnt do anythingRytis Zolubas
02/11/2023, 9:26 AM2023-02-11 09:22:27 [1;31mERROR[m i.a.c.t.ConnectionManagerUtils(getWorkflowState):222 - Exception thrown while checking workflow state for connection id 5cbf7ac0-8a79-4394-95cb-d5f435c42668
io.temporal.client.WorkflowServiceException: workflowId='connection_manager_5cbf7ac0-8a79-4394-95cb-d5f435c42668', runId='', workflowType='ConnectionManagerWorkflow'}
at io.temporal.internal.sync.WorkflowStubImpl.throwAsWorkflowFailureExceptionForQuery(WorkflowStubImpl.java:379) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.query(WorkflowStubImpl.java:274) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.queryWorkflow(WorkflowInvocationHandler.java:309) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:272) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.17.0.jar:?]
at jdk.proxy2.$Proxy91.getState(Unknown Source) ~[?:?]
at io.airbyte.commons.temporal.ConnectionManagerUtils.getWorkflowState(ConnectionManagerUtils.java:220) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
at io.airbyte.commons.temporal.ConnectionManagerUtils.isWorkflowStateRunning(ConnectionManagerUtils.java:228) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
at io.airbyte.commons.temporal.TemporalClient.startNewManualSync(TemporalClient.java:226) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
at io.airbyte.server.scheduler.TemporalEventRunner.startNewManualSync(TemporalEventRunner.java:27) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at io.airbyte.server.handlers.SchedulerHandler.submitManualSyncToWorker(SchedulerHandler.java:417) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at io.airbyte.server.handlers.SchedulerHandler.syncConnection(SchedulerHandler.java:347) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at io.airbyte.server.apis.ConnectionApiController.lambda$syncConnection$7(ConnectionApiController.java:77) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:17) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at io.airbyte.server.apis.ConnectionApiController.syncConnection(ConnectionApiController.java:77) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.999985556s. [closed=[], open=[[remote_addr=airbyte-temporal/172.26.0.8:7233]]]
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.queryWorkflow(WorkflowServiceGrpc.java:4099) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.client.external.GenericWorkflowClientImpl.lambda$query$9(GenericWorkflowClientImpl.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.client.external.GenericWorkflowClientImpl.query(GenericWorkflowClientImpl.java:201) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.client.RootWorkflowClientInvoker.query(RootWorkflowClientInvoker.java:140) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.query(WorkflowStubImpl.java:270) ~[temporal-sdk-1.17.0.jar:?]
... 63 more
Praveenraaj K S
02/11/2023, 10:04 AMAnkit Kumar
02/11/2023, 1:23 PMLior Chen
02/11/2023, 1:46 PM2023-02-11 13:20:11 normalization > WARN: Unknown type for column source_url at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column updated_at at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column location_id at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column source_name at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column total_price at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column completed_at at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column landing_site at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column total_weight at abandoned_checkouts
2023-02-11 13:20:11 normalization > WARN: Unknown type for column referring_site at abandoned_checkouts
.....
which eventually causing errors during the dbt execution:
2023-02-11 13:20:45 normalization > 001790 (42601): SQL compilation error:
2023-02-11 13:20:45 normalization > inconsistent data type for result columns for set operator input branches, expected VARCHAR(16777216), got VARIANT for expression [{2}] branch {3}
2023-02-11 13:20:45 normalization > Database Error in model ABANDONED_CHECKOUTS_SCD (models/generated/airbyte_incremental/scd/ORG_6950_01GRBK7DJ1QN50C5F3GXAVWAWQ/ABANDONED_CHECKOUTS_SCD.sql)
this this is basically for all streams that synced successfully until now and to all similar connectors. so it doesnt seem like a local data type collision, but rather a separate problem. any idea? or suggestions on how to debug that?..Chris
02/11/2023, 6:23 PMdocker compose up -d
, the Container airbyte-bootloader doesn’t seem to get created properly. I followed steps in the following documentation:
https://docs.airbyte.com/operator-guides/configuring-airbyte-db/
https://cloud.google.com/sql/docs/postgres/connect-instance-compute-engine
I am using Private IP method. I have set up same network for my Compute Engine and Cloud SQL.
I also changed .env
:
DATABASE_USER=username
DATABASE_PASSWORD=password
DATABASE_HOST=project-id:us-central1:airbyte-sql
DATABASE_PORT:5432
DATABASE_DB=airbyte
DATABASE_URL=jdbc:<postgresql://project-id>:us-central1:airbyte-sql:5432/airbyte
Does anyone know why it is not working, and know how to set it up properly?Martin Larsson
02/11/2023, 7:09 PMINFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $: null found, string expected
When I run the sync with BigQuery as destination I get the same messages but the sync succeeds but with the result that about half of the rows are empty.Dallin Bentley
02/12/2023, 12:40 AM2023-02-11 22:00:04 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Trying to get all of the streams available for Mixpanel... I know that there are api rate limits and such, so wanted to double check and see if others have had issues. This is just a free Mixpanel plan as well.Kaiming Wan
02/12/2023, 9:08 AMKaiming Wan
02/12/2023, 9:08 AMPaulo José Ianes Bernardo Filho
02/12/2023, 1:15 PMCarolina Buckler
02/12/2023, 2:25 PMFacebookAPIException('Error: 2635, (#2635) Your app has been upgraded to version v16.0, please use this version or newer. This can be verified in the settings tab on the App Dashboard.')
Dusty Shapiro
02/12/2023, 3:25 PMpython:3.9.9-slim-bullseye
to python:3.9.16-slim-bullseye
?
Our vulnerability detection software is throwing quite a few warnings for the former, and recommends bumping to the later.
We’re on K8s via Helm Deploy. Thanks!