everytime it would say that it s successful on airbyte but n Airbyte #ask-community-for-troubleshooting

Join Slack

everytime, it would say that it's successful on ai...

# ask-community-for-troubleshooting

John Gaspar

05/22/2021, 6:11 AM

everytime, it would say that it's successful on airbyte, but nothing's in postgres.

✅ 1

Chris (deprecated profile)

05/22/2021, 6:21 AM

How do you check there s nothing in postgres? What about:

Copy code

select ° from hello._airbyte_raw_doot

Chris (deprecated profile)

05/22/2021, 6:22 AM

Summary says it wrote two records there and that you didnt enable normalisation to create

hello.doot

tables

John Gaspar

05/22/2021, 6:28 AM

So,

hello.doot

should be the source, i.e. mysql. The destination is a postgres table

goodbye

John Gaspar

05/22/2021, 6:28 AM

normalisation is enabled

John Gaspar

05/22/2021, 6:30 AM

I'll run another test again

John Gaspar

05/22/2021, 6:31 AM

Copy code

2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) DefaultReplicationWorker(run):77 - start sync worker. job id: 312 attempt id: 0
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) DefaultReplicationWorker(run):86 - configured sync modes: {hello.doot=full_refresh - overwrite}
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) DefaultAirbyteDestination(start):67 - Running destination...
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Checking if airbyte/destination-postgres:0.3.3 exists...
2021-05-22 06:30:39 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - airbyte/destination-postgres:0.3.3 was found locally.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.p.PostgresDestination(main):83 - {} - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: WRITE
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(startTracked):119 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):143 - {} - Preparing tmp tables in destination started for 1 streams
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 - {} - Preparing tmp table in destination started for stream doot. schema: hello, tmp table name: _airbyte_1621665040357__airbyte_raw_doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):153 - {} - Preparing tables in destination completed.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):77 - {} - hasFailed: false.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(close):189 - {} - executing on success close procedure.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - {} - Finalizing tables in destination started for 1 streams
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):185 - {} - Finalizing stream doot. schema hello, tmp table _airbyte_1621665040357__airbyte_raw_doot, final table _airbyte_raw_doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):198 - {} - Executing finalization of tables.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):200 - {} - Finalizing tables in destination completed.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):203 - {} - Cleaning tmp tables in destination started for 1 streams
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):207 - {} - Cleaning tmp table in destination started for stream doot. schema hello, tmp table name: _airbyte_1621665040357__airbyte_raw_doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):212 - {} - Cleaning tmp tables in destination completed.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 2021-05-22 06:30:40 [32mINFO[m i.a.i.d.p.PostgresDestination(main):85 - {} - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) DefaultReplicationWorker(run):116 - sync summary: io.airbyte.config.StandardSyncSummary@5c481db7[status=completed,recordsSynced=0,bytesSynced=0,startTime=1621665039659,endTime=1621665040558]
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) DefaultNormalizationWorker(run):61 - Running normalization.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Checking if airbyte/normalization:0.1.28 exists...
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - airbyte/normalization:0.1.28 was found locally.
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/312/0/normalize')
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - transform_postgres
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Processing destination_catalog.json...
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab1_64d.sql from doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab2_64d.sql from doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab3_64d.sql from doot
2021-05-22 06:30:40 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Generating airbyte_tables/hello/doot_185.sql from doot
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Running with dbt=0.19.1
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Found 4 models, 0 tests, 0 snapshots, 0 analyses, 364 macros, 0 operations, 0 seed files, 1 source, 0 exposures
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 -
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 | Concurrency: 32 threads (target='prod')
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 |
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 | 1 of 1 START table model hello.doot.......................................................................... [RUN]
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 | 1 of 1 OK created table model hello.doot..................................................................... [[32mSELECT 0[0m in 0.07s]
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 |
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - 06:30:42 | Finished running 1 table model in 0.19s.
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 -
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - [32mCompleted successfully[0m
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 -
2021-05-22 06:30:42 INFO (/tmp/workspace/312/0) LineGobbler(voidCall):69 - Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2021-05-22 06:30:43 INFO (/tmp/workspace/312/0) DefaultNormalizationWorker(run):76 - Normalization executed in 0.
2021-05-22 06:30:43 INFO (/tmp/workspace/312/0) TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...

John Gaspar

05/22/2021, 6:31 AM

Here, this one should have normalisation 🙂 I may have given you the logs on one trial I did without it earlier.

Chris (deprecated profile)

05/22/2021, 6:32 AM

You cant control where data is written in destination yet... It is replicating source schema too

Chris (deprecated profile)

05/22/2021, 6:32 AM

Yes that log wrote into

hello.doot

in postgres

John Gaspar

05/22/2021, 6:33 AM

ahhh, you mean the destination db name has to be exactly the same too?

John Gaspar

05/22/2021, 6:34 AM

It also doesn't auto create the db if it doesn't exist, I suppose? even with root perms to do so?

Chris (deprecated profile)

05/22/2021, 6:34 AM

A db from mysql is translated to schema in postgres and it does create it

Chris (deprecated profile)

05/22/2021, 6:36 AM

https://github.com/airbytehq/airbyte/issues/3481

John Gaspar

05/22/2021, 6:41 AM

Ok, so I just made this run now: Run #1 source: mysql, db_name: hello dest: postgres, db_name: goodbye Run #2 source: mysql, db_name: hello dest: postgres, db_name: hello

Copy code

2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) DefaultReplicationWorker(run):77 - start sync worker. job id: 313 attempt id: 0
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) DefaultReplicationWorker(run):86 - configured sync modes: {hello.doot=full_refresh - overwrite}
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) DefaultAirbyteDestination(start):67 - Running destination...
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Checking if airbyte/destination-postgres:0.3.3 exists...
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - airbyte/destination-postgres:0.3.3 was found locally.
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Checking if airbyte/source-mysql:0.3.0 exists...
2021-05-22 06:34:44 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - airbyte/source-mysql:0.3.0 was found locally.
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.d.p.PostgresDestination(main):83 - {} - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.s.m.MySqlSource(main):80 - {} - starting source: class io.airbyte.integrations.source.mysql.MySqlSource
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.source.mysql.MySqlSource
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {read=null, catalog=source_catalog.json, config=source_config.json}
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: READ
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: WRITE
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='null'}
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.s.j.JdbcCdcStateManager(<init>):46 - {} - Initialized CDC state with: null
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.s.j.JdbcStateManager(createCursorInfoForStream):138 - {} - No cursor field set in catalog but not present in state. Stream: AirbyteStreamNameNamespacePair{name='doot', namespace='hello'}, New Cursor Field: null. Resetting cursor value
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(startTracked):119 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):143 - {} - Preparing tmp tables in destination started for 1 streams
2021-05-22 06:34:45 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:45 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 - {} - Preparing tmp table in destination started for stream doot. schema: hello, tmp table name: _airbyte_1621665285905__airbyte_raw_doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):153 - {} - Preparing tables in destination completed.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.j.AbstractJdbcSource(queryTableFullRefresh):518 - {} - Queueing query for table: doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.j.AbstractJdbcSource(lambda$queryTableFullRefresh$29):523 - {} - Preparing query for table: doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.j.AbstractJdbcSource(lambda$queryTableFullRefresh$29):528 - {} - Executing query for table: doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.j.AbstractJdbcSource(lambda$read$4):191 - {} - Closing database connection pool.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.j.AbstractJdbcSource(lambda$read$4):193 - {} - Closed database connection pool.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.b.IntegrationRunner(run):120 - {} - Completed integration: io.airbyte.integrations.source.mysql.MySqlSource
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.s.m.MySqlSource(main):82 - {} - completed source: class io.airbyte.integrations.source.mysql.MySqlSource
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):77 - {} - hasFailed: false.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.b.BufferedStreamConsumer(close):189 - {} - executing on success close procedure.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - {} - Finalizing tables in destination started for 1 streams
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):185 - {} - Finalizing stream doot. schema hello, tmp table _airbyte_1621665285905__airbyte_raw_doot, final table _airbyte_raw_doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):198 - {} - Executing finalization of tables.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):200 - {} - Finalizing tables in destination completed.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):203 - {} - Cleaning tmp tables in destination started for 1 streams
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):207 - {} - Cleaning tmp table in destination started for stream doot. schema hello, tmp table name: _airbyte_1621665285905__airbyte_raw_doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):212 - {} - Cleaning tmp tables in destination completed.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 2021-05-22 06:34:46 [32mINFO[m i.a.i.d.p.PostgresDestination(main):85 - {} - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultReplicationWorker(run):116 - sync summary: io.airbyte.config.StandardSyncSummary@104be532[status=completed,recordsSynced=2,bytesSynced=64,startTime=1621665284874,endTime=1621665286542]
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.22.3-alpha
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) DefaultNormalizationWorker(run):61 - Running normalization.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Checking if airbyte/normalization:0.1.28 exists...
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - airbyte/normalization:0.1.28 was found locally.
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/313/0/normalize')
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - transform_postgres
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Processing destination_catalog.json...
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab1_64d.sql from doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab2_64d.sql from doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Generating airbyte_views/hello/doot_ab3_64d.sql from doot
2021-05-22 06:34:46 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Generating airbyte_tables/hello/doot_185.sql from doot
2021-05-22 06:34:48 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Running with dbt=0.19.1
2021-05-22 06:34:48 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Found 4 models, 0 tests, 0 snapshots, 0 analyses, 364 macros, 0 operations, 0 seed files, 1 source, 0 exposures
2021-05-22 06:34:48 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 -
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 | Concurrency: 32 threads (target='prod')
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 |
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 | 1 of 1 START table model hello.doot.......................................................................... [RUN]
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 | 1 of 1 OK created table model hello.doot..................................................................... [[32mSELECT 2[0m in 0.07s]
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 |
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - 06:34:49 | Finished running 1 table model in 0.20s.
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 -
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - [32mCompleted successfully[0m
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 -
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) LineGobbler(voidCall):69 - Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) DefaultNormalizationWorker(run):76 - Normalization executed in 0.
2021-05-22 06:34:49 INFO (/tmp/workspace/313/0) TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...

In both situations, here's the results on dest postgres:

John Gaspar

05/22/2021, 6:42 AM

What am I missing here? 🤔

Chris (deprecated profile)

05/22/2021, 6:44 AM

Please share your config of the destination

Chris (deprecated profile)

05/22/2021, 6:45 AM

A screenshot of the destination settings (without password credentiald) would do

John Gaspar

05/22/2021, 6:45 AM

sure

John Gaspar

05/22/2021, 6:46 AM

message has been deleted

John Gaspar

05/22/2021, 6:47 AM

I swapped the db name for the dest for both the runs above. Also clicked on "reset data" on the UI before each run

Chris (deprecated profile)

05/22/2021, 6:47 AM

So it s writing in the hello database in the hello schema and doot table

John Gaspar

05/22/2021, 6:48 AM

so it seems, according to airbyte, but it's not showing in the pg cli

John Gaspar

05/22/2021, 6:48 AM

am using postgres:13.0 official docker image for context

Chris (deprecated profile)

05/22/2021, 6:48 AM

Once connected to your destination database you need to run this query: https://airbytehq.slack.com/archives/C021JANJ6TY/p1621664479047000?thread_ts=1621663907.044700&cid=C021JANJ6TY

Chris (deprecated profile)

05/22/2021, 6:49 AM

select * from hello.doot

if you have normalisation on

John Gaspar

05/22/2021, 6:50 AM

empty response

John Gaspar

05/22/2021, 6:50 AM

ahh sorry

John Gaspar

05/22/2021, 6:50 AM

semicolon on sql

John Gaspar

05/22/2021, 6:51 AM

hrmm

John Gaspar

05/22/2021, 6:51 AM

yes, it's showing now

John Gaspar

05/22/2021, 6:51 AM

data is there!

John Gaspar

05/22/2021, 6:51 AM

wondering why it doesn't show with \dt?

John Gaspar

05/22/2021, 6:51 AM

it doesn't update the schema?

John Gaspar

05/22/2021, 6:55 AM

also, it only writes to hello.doot as dest, even if I set the dest connector to go to db_name goodbye

Chris (deprecated profile)

05/22/2021, 6:57 AM

It d go in goodbye database but in hello schema and doot table

Chris (deprecated profile)

05/22/2021, 6:58 AM

Until the issue i linked is resolved

John Gaspar

05/22/2021, 7:01 AM

Okay! I see it now, even in goodbye db, doing a select * from hello.doot; still shows the data. Now i understand the issue.

John Gaspar

05/22/2021, 7:01 AM

what can I do to make the schema appear with \dt?

Chris (deprecated profile)

05/22/2021, 7:03 AM

Sorry I am not familiar with

\dt

command

John Gaspar

05/22/2021, 7:04 AM

No problem Chris. You've been wonderful 🙂 Thanks for the help. Happy to support airbyte in whatever capacity I can !

John Gaspar

05/22/2021, 7:21 AM

@Chris (deprecated profile) , what sql command would you use to show what hidden tables are available?

John Gaspar

05/22/2021, 7:22 AM

\dt

is usually used to do just that.

SHOW TABLES;

equivalent in mysql

👍 1

5 Views

Open in Slack

Previous Next