Anton Svetlov
11/29/2021, 11:23 AMERROR[m i.a.i.d.b.BufferedStreamConsumer(close):219 - {} - Close failed.
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:473) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:393) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279) ~[postgresql-42.2.18.jar:42.2.18]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194) ~[commons-dbcp2-2.7.0.jar:2.7.0]
[35mdestination[0m - 2021-11-29 08:30:33 INFO DefaultAirbyteStreamFactory(lambda$create$0):61 - at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194) ~[commons-dbcp2-2.7.0.jar:2.7.0]
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - Exception in thread "main" org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:473)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:393)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at io.airbyte.db.jdbc.JdbcDatabase.lambda$execute$0(JdbcDatabase.java:40)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at io.airbyte.db.jdbc.DefaultJdbcDatabase.execute(DefaultJdbcDatabase.java:52)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at io.airbyte.db.jdbc.JdbcDatabase.execute(JdbcDatabase.java:40)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at io.airbyte.integrations.destination.jdbc.JdbcSqlOperations.executeTransaction(JdbcSqlOperations.java:95)
[35mdestination[0m - 2021-11-29 08:30:33 ERROR LineGobbler(voidCall):82 - at io.airbyte.integrations.destination.jdbc.JdbcBufferedConsumerFactory.lambda$onCloseFunction$3(JdbcBufferedConsumerFactory.java:179)
Steve West
11/29/2021, 5:48 PMKamil Baldyga
11/29/2021, 7:24 PMMake sure your cluster is active and accessible from the machine running AirbyteDoes this mean Airbyte Cloud doesn't support Redshift? If this is not correct, and the documentation needs updating, does it support SSH tunnel or the instance needs to have a public ip address?
Guus van Heijningen
11/30/2021, 8:25 AMNiclas Grahm
11/30/2021, 10:23 AMapache-airflow-providers-airbyte
and setting up a connection in airflow pointing at '<airbyte://host.docker.internal:8000>'
However, when trying to start my DAG in airflow, I get the following error:
requests.exceptions.HTTPError: 404 Client error: Not found for url: <http://host.docker.internal:8000/api/v1/connections/sync>
Hos OS is Ubuntu 18.04.
Does anyone have any ideas on what I should try?
Thanks in advance!Hein Hofman
11/30/2021, 12:11 PMBean L
11/30/2021, 12:19 PMEmin Can Oğuz (Hepsiburada)
11/30/2021, 12:36 PMEmma-Louise Upton
11/30/2021, 1:30 PMEmmanuel Orrego
11/30/2021, 1:58 PMDave Lyons
12/01/2021, 3:07 PMScott Pedersen
12/02/2021, 12:35 AMhan so
12/02/2021, 10:47 PMOscar Gonzalez
12/03/2021, 2:25 PMTom Griffin
12/04/2021, 2:14 AMM Andriansyah Putra
12/06/2021, 5:11 AMStefan Otte
12/06/2021, 9:41 AMdocker-compose
instances of airbyte in parallel without sharing any data/config?
Locally I have to run multiple airbyte instances for completely separate projects and I want to use docker-compose
for that. No data or configuration between the instances should be shared.
The docker-compose.yaml
contains a few volumes (workspace
, data
, db
) and I can change the mount point, but the .env
file contains many more paths that would be the same between instances.
Is there maybe a global prefix to change all these variables? Do I have to change the variable one by one? Or is there a better way?Yuriy Markiv
12/06/2021, 3:45 PMDave Lyons
12/06/2021, 5:40 PMdocker-compose.yaml
and build it into the container, or do I need to bash in and copy it somewhere? I can’t find any instructions.Danny Duncan
12/06/2021, 6:43 PMBoggdan Barrientos
12/06/2021, 7:26 PMgit checkout v0.32.0-alpha
I got error: pathspec 'v0.32.0-alpha' did not match any file(s) known to git.
Oscar Gonzalez
12/06/2021, 9:36 PMMax Krog
12/07/2021, 9:35 AMcreate or replace table `project-name.airbyte_things._airbyte_raw_things` as (
with base as (
select
*,
row_number() over(partition by sha512(_airbyte_data) order by _airbyte_emitted_at) as hashed_data_rn -- hash the data and find the row_number
from
`project-name.airbyte_things._airbyte_raw_things`
)
select * except(hashed_data_rn) from base where hashed_data_rn = 1
)
Basically it hashes the raw data in the source table, orders the entries with the same hash, and deletes every record except the first for the hash.
What is this process of "condensing"/"cleaning" a source table called? 🙂Parthasarathy Balasubramanian
12/07/2021, 2:40 PMMaurice
12/07/2021, 7:50 PMBERKIN
12/08/2021, 4:56 AMDivya (Proximity)
12/08/2021, 5:21 AMxiaxp
12/08/2021, 6:47 AMxiaxp
12/08/2021, 6:48 AMMatt Wright
12/08/2021, 7:39 PM