Niranjan Reddy Dumpa
06/15/2023, 4:57 AMChidambara Ganapathy
06/15/2023, 6:04 AMJoel
06/15/2023, 9:51 AMJoel
06/15/2023, 1:02 PMOctavia Squidington III
06/15/2023, 1:45 PMNico Gallinal
06/15/2023, 2:25 PMSharat Visweswara
06/15/2023, 3:48 PMChristian Lam
06/15/2023, 4:28 PM0.0.3
. I am wondering if it is a known issue with this version (or all other versions) where when setting the sync mode to Incremental Deduped + history
, the state will always increment up a day which causes all subsequent syncs to pull in 0 records.
For example, if I run the sync multiple times within a day, each run will increment the state by a day which causes the sync to skip over data that may not come in until later on in the day.
Here is a sample log from today which shows the state being set to dates in the future while bringing in 0 records:
2023-06-15 16:08:29 [44msource[0m > Starting syncing SourceGoogleAnalyticsDataApi
2023-06-15 16:08:29 [44msource[0m > Syncing stream: daily_active_users
2023-06-15 16:08:29 [44msource[0m > Setting state of daily_active_users stream to {'date': '2023-06-20'}
2023-06-15 16:08:29 [44msource[0m > Read 0 records from daily_active_users stream
2023-06-15 16:08:29 [44msource[0m > Finished syncing daily_active_users
2023-06-15 16:08:29 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
2023-06-15 16:08:29 [44msource[0m > Syncing stream: devices
2023-06-15 16:08:29 [44msource[0m > Setting state of devices stream to {'date': '2023-06-19'}
2023-06-15 16:08:29 [44msource[0m > Read 0 records from devices stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing devices
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
2023-06-15 16:08:30 [44msource[0m > Syncing stream: four_weekly_active_users
2023-06-15 16:08:30 [44msource[0m > Setting state of four_weekly_active_users stream to {'date': '2023-06-20'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from four_weekly_active_users stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing four_weekly_active_users
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
Syncing stream four_weekly_active_users 0:00:00.005786
2023-06-15 16:08:30 [44msource[0m > Syncing stream: locations
2023-06-15 16:08:30 [44msource[0m > Setting state of locations stream to {'date': '2023-06-19'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from locations stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing locations
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
Syncing stream four_weekly_active_users 0:00:00.005786
Syncing stream locations 0:00:00.001644
2023-06-15 16:08:30 [44msource[0m > Syncing stream: pages
2023-06-15 16:08:30 [44msource[0m > Setting state of pages stream to {'date': '2023-06-20'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from pages stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing pages
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Waliur Rahman
06/15/2023, 4:29 PMConfiguration check failed
Message: HikariPool-1 - Connection is not available, request timed out after 10002ms.
Stacktrace:
io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:84)
at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.createDatabase(AbstractJdbcSource.java:434)
at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:277)
at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:99)
at io.airbyte.integrations.source.relationaldb.AbstractDbSource.check(AbstractDbSource.java:85)
at io.airbyte.integrations.source.postgres.PostgresSource.check(PostgresSource.java:557)
at io.airbyte.integrations.base.ssh.SshTunnel.sshWrap(SshTunnel.java:270)
at io.airbyte.integrations.base.ssh.SshWrappedSource.check(SshWrappedSource.java:53)
at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:124)
at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:538)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:78)
... 10 more
Omar Mayar
06/15/2023, 4:36 PMSharat Visweswara
06/15/2023, 5:13 PMYoan Yahemdi
06/15/2023, 6:15 PMBa Thien Tran
06/15/2023, 8:43 PMJatin Morar
06/15/2023, 8:56 PMSlackbot
06/15/2023, 9:45 PMAshley Pace
06/15/2023, 9:51 PMMadhav
06/16/2023, 4:12 AMjava.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10001ms.
Ayushman Singh
06/16/2023, 6:01 AMcron
that can be specified in the sync frequency ?
⢠If there is a limit on the cron
why is it there for open source, can it be modifed?archna singh
06/16/2023, 7:01 AMJoel
06/16/2023, 7:49 AMRishav Sinha
06/16/2023, 8:29 AMJackson exception
while integrating Elasticsearch. can someone please help me with this ?
@kapa.aiMagnus Berg Sletfjerding
06/16/2023, 8:43 AMCJ Johanson
06/16/2023, 1:47 PMConfiguration check failed
State code: 08001; Message: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.i have been scouring the interwebs for a solution but im now plain ol' lost. i tried to update some setting on the postgres side. these changes include: 1. updating
listen_addresses = '*'
in postgresql.conf
2. uncommenting port=5432
in postgresql.conf
if anyone can provide any insight at all, i would very much appreciate it!Octavia Squidington III
06/16/2023, 7:45 PMBiondi Septian S
06/17/2023, 8:50 AMBiondi Septian S
06/17/2023, 8:51 AMBiondi Septian S
06/17/2023, 8:51 AMRoy Ben Dov
06/19/2023, 8:50 AMAmmar Tello
06/19/2023, 1:31 PMio.airbyte.workers.exception.WorkerException: Could not find image: airbyte/destination-snowflake:1.0.5
any idea regarding this?
Thanks in advance šCameron Smith
06/19/2023, 2:51 PM