Hi I am trying to move mixpanel export stream data to a postgres database, but I am the errors. Please check the below log. It would be really helpful if anyone could help me with the reason/solution for the error. Thanks.

Do we have AWS Cost Explorer source in Airbyte?

Hello everyone! We're going to start using airbyte to analyze our marketing performance which means we are going to connect Facebook ads, Apple search ads, Tiktok, Google, Snapchat... After looking at things it seems like a huge task to enable the needed api permissions from these services, and I'm not even sure I will be able to do it if I don't get help from our customer success managers because it seems like they are a bit reluctant handing out these api permissions. Have people in here concluded the same thing or was it easy for you to get the api permissions needed? If so, how did you do it?? 😁

In the docs I can't find any advice on which permissions to request from Tiktok to be able to analyze the marketing performance on the platform. Does anyone here know?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

A bunch of temp tables are being created, is there a way to prevent those?

Hello, one of my connections always gets stuck on sync or reset jobs at "getTaskQueue" e.g.

Hello, regarding GA4 connector

0.0.3

. I am wondering if it is a known issue with this version (or all other versions) where when setting the sync mode to

Incremental Deduped + history

, the state will always increment up a day which causes all subsequent syncs to pull in 0 records. For example, if I run the sync multiple times within a day, each run will increment the state by a day which causes the sync to skip over data that may not come in until later on in the day. Here is a sample log from today which shows the state being set to dates in the future while bringing in 0 records:

2023-06-15 16:08:29 [44msource[0m > Starting syncing SourceGoogleAnalyticsDataApi
2023-06-15 16:08:29 [44msource[0m > Syncing stream: daily_active_users 
2023-06-15 16:08:29 [44msource[0m > Setting state of daily_active_users stream to {'date': '2023-06-20'}
2023-06-15 16:08:29 [44msource[0m > Read 0 records from daily_active_users stream
2023-06-15 16:08:29 [44msource[0m > Finished syncing daily_active_users
2023-06-15 16:08:29 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
2023-06-15 16:08:29 [44msource[0m > Syncing stream: devices 
2023-06-15 16:08:29 [44msource[0m > Setting state of devices stream to {'date': '2023-06-19'}
2023-06-15 16:08:29 [44msource[0m > Read 0 records from devices stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing devices
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
2023-06-15 16:08:30 [44msource[0m > Syncing stream: four_weekly_active_users 
2023-06-15 16:08:30 [44msource[0m > Setting state of four_weekly_active_users stream to {'date': '2023-06-20'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from four_weekly_active_users stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing four_weekly_active_users
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
Syncing stream four_weekly_active_users 0:00:00.005786
2023-06-15 16:08:30 [44msource[0m > Syncing stream: locations 
2023-06-15 16:08:30 [44msource[0m > Setting state of locations stream to {'date': '2023-06-19'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from locations stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing locations
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
Syncing stream daily_active_users 0:00:00.015667
Syncing stream devices 0:00:00.001815
Syncing stream four_weekly_active_users 0:00:00.005786
Syncing stream locations 0:00:00.001644
2023-06-15 16:08:30 [44msource[0m > Syncing stream: pages 
2023-06-15 16:08:30 [44msource[0m > Setting state of pages stream to {'date': '2023-06-20'}
2023-06-15 16:08:30 [44msource[0m > Read 0 records from pages stream
2023-06-15 16:08:30 [44msource[0m > Finished syncing pages
2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:

Hi Folks, I am using Plural hosted version of Airbyte (0.44.8). When I am trying to connect to a Postgres database I am getting the following error. Wondering if anyone has any idea what this is about or how to resolve it. Thanks in advance.

Configuration check failed
Message: HikariPool-1 - Connection is not available, request timed out after 10002ms.

Stacktrace:
io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
	at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:84)
	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.createDatabase(AbstractJdbcSource.java:434)
	at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:277)
	at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:99)
	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.check(AbstractDbSource.java:85)
	at io.airbyte.integrations.source.postgres.PostgresSource.check(PostgresSource.java:557)
	at io.airbyte.integrations.base.ssh.SshTunnel.sshWrap(SshTunnel.java:270)
	at io.airbyte.integrations.base.ssh.SshWrappedSource.check(SshWrappedSource.java:53)
	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:124)
	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
	at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:538)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:78)
	... 10 more

does anyone know what the “unexpected behavior” described here might be? would ignoring this reset impact the cursor for upserts? not seeing much on airbyte docs so want to see how risky of a move it is to disregard

Hello, if I have multiple sources (e.g. facebook marketing) writing to the same set of tables, what is the impact of resetting one of the connections? Will it delete all the the data, including data for other sources?

Hello, I’m trying to create the connexion with BigQuery using Bucket as recommended. However I am facing exactly the same issue. Haven’t been able to resolve it. Could someone has some ideas or guidances? Thank you very much

Hi I’m trying to set up Amazon Seller Partner on Cloud but cannot find the connector in the connector overview

Hi, how can the s3 destination not override a file. Just add a new file per day?

This message was deleted.

Hi everyone - I'm gathering information on data replication tools in order to evaluate a few and ultimately make a recommendation to my boss. I'm seriously looking at Airbyte open source and CData Sync. Both self-hosted. CData is not free, the plan we're considering has a limit of 1 billion rows a month. Sources: MySQL, Azure SQL, Hubspot, Google Analytics, and a couple of APIs We don't have a ton of new data daily, maybe north of 1 million daily inserted or updated. The MySQL sources is our largest and it has many tables that have no identifiable incremental column AND we can't turn on CDC as we don't own the database. We'd have to replicate some of the tables once a day full and others incrementally which is fine. Anyway, all that said, I'd love to hear others' experiences with evaluating the various options out there and how you landed on Airbyte open source, aside from cost. I've pulled info on CData Sync, Airbyte, Fivetran and Stitch. Thanks!!!!

Hi, getting error while running my postgres connector , how can I increase timeout value, Does any one have idea?

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10001ms.

I have a question about the sync limits on connections. • For airbyte open source is there a limit on the

cron

that can be specified in the sync frequency ? • If there is a limit on the

cron

why is it there for open source, can it be modifed?

i want to add my ssl certificate in gitlab source of airbyte so that i can make connection, is there any solution to do it.? for now i am getting error

When asking for API permissions from Facebook should we create a new app that is solely used by airbyte or should we use the existing Facebook app we have (which we use for social sign in and profile pictures in our app store app)?

Hi Team, i am facing

Jackson exception

while integrating Elasticsearch. can someone please help me with this ? @kapa.ai

Hi Airbyte! I’m working with a CockroachDB source that I want to incrementally load according to this guide and these instructions. Our database has tables with ~10-100M rows, with ~100-500K added every day. We’ve been able to test (sandboxed) delta loads of up to 500K rows without problems, but we’re wondering if it’s possible to set up a connector to not do an “initial” load? I.E can we pre-set the “cursor field” when we create the connector the first time, and manually load our existing/past data from a dump? Related: We’ve experienced problems with the CockroachDB connector loading more than ~1M rows. I think that is related to this bug , since we get the same OOM message.

hi everyone! i am very new to airbyte and am trying to play around with it on my laptop and have run into some issues when trying to connect airbyte to postgres. my goal is to host a postgres server and database on my local while also hosting airbyte on my local (running it via docker). so far, i have used pgadmin4 to, i believe, set up a postgres server and database. I have put some fake data into said database. when i go to airbyte to try and connect the postgres db as a source, i keep getting the following error:

Configuration check failed

State code: 08001; Message: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

i have been scouring the interwebs for a solution but im now plain ol' lost. i tried to update some setting on the postgres side. these changes include: 1. updating

listen_addresses = '*'

in

postgresql.conf

2. uncommenting

port=5432

in

postgresql.conf

if anyone can provide any insight at all, i would very much appreciate it!

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

Hello Airbyte team, please take a look at this serious issue: https://github.com/airbytehq/airbyte/issues/24097

this is a serious timeout issue on typesense destination connector

I think this is the solution, by merging this pull request below: https://github.com/airbytehq/airbyte/pull/18806/commits

Hii! I am new to airbyte I created my first connection of data replication from a postgres db(version 15) to postgress db (version 14) all connection tests passed successfully, also sync passed but I didn't see the new data replicated. Sync mode: Incremental | Deduped + history and I cant choose Cursor Field

Hi Everybody, I am new to Airbyte, and I am trying to make a connection between Postgresql and Snowflake, When I try to make the Snowflake a destination, I face this error:

io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/destination-snowflake:1.0.5

any idea regarding this? Thanks in advance 😊

Hi, I'm trying to follow the airbyte tutorial to connect Google Sheets to AWS Postgres, and everything is great until I try to add Postgres as a source/destination. When I try with SSL disabled I get an error that unsecured connections ar enot allowed, and to try using require, verify-ca, verify-full. However when I try using require I get HikariPool-1 - Connection is not available. Any idea why? I can connect to the database unsecured through psql and vscode, and I've followed every step of the tutorial, but there's no mention of the ssl setting on there. Airbyte Tutorial