https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • n

    Niranjan Reddy Dumpa

    06/15/2023, 4:57 AM
    Hi I am trying to move mixpanel export stream data to a postgres database, but I am the errors. Please check the below log. It would be really helpful if anyone could help me with the reason/solution for the error. Thanks.
    95f47f36_959f_415b_892c_f8f6c9011159_logs_10_txt.txt
  • c

    Chidambara Ganapathy

    06/15/2023, 6:04 AM
    Do we have AWS Cost Explorer source in Airbyte?
    k
    l
    • 3
    • 3
  • j

    Joel

    06/15/2023, 9:51 AM
    Hello everyone! We're going to start using airbyte to analyze our marketing performance which means we are going to connect Facebook ads, Apple search ads, Tiktok, Google, Snapchat... After looking at things it seems like a huge task to enable the needed api permissions from these services, and I'm not even sure I will be able to do it if I don't get help from our customer success managers because it seems like they are a bit reluctant handing out these api permissions. Have people in here concluded the same thing or was it easy for you to get the api permissions needed? If so, how did you do it?? 😁
    k
    • 2
    • 2
  • j

    Joel

    06/15/2023, 1:02 PM
    In the docs I can't find any advice on which permissions to request from Tiktok to be able to analyze the marketing performance on the platform. Does anyone here know?
    k
    • 2
    • 5
  • o

    Octavia Squidington III

    06/15/2023, 1:45 PM
    šŸ”„ Office Hours starts in 15 minutes šŸ”„ Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves
  • n

    Nico Gallinal

    06/15/2023, 2:25 PM
    A bunch of temp tables are being created, is there a way to prevent those?
    k
    • 2
    • 2
  • s

    Sharat Visweswara

    06/15/2023, 3:48 PM
    Hello, one of my connections always gets stuck on sync or reset jobs at "getTaskQueue" e.g.
    k
    • 2
    • 3
  • c

    Christian Lam

    06/15/2023, 4:28 PM
    Hello, regarding GA4 connector
    0.0.3
    . I am wondering if it is a known issue with this version (or all other versions) where when setting the sync mode to
    Incremental Deduped + history
    , the state will always increment up a day which causes all subsequent syncs to pull in 0 records. For example, if I run the sync multiple times within a day, each run will increment the state by a day which causes the sync to skip over data that may not come in until later on in the day. Here is a sample log from today which shows the state being set to dates in the future while bringing in 0 records:
    Copy code
    2023-06-15 16:08:29 [44msource[0m > Starting syncing SourceGoogleAnalyticsDataApi
    2023-06-15 16:08:29 [44msource[0m > Syncing stream: daily_active_users 
    2023-06-15 16:08:29 [44msource[0m > Setting state of daily_active_users stream to {'date': '2023-06-20'}
    2023-06-15 16:08:29 [44msource[0m > Read 0 records from daily_active_users stream
    2023-06-15 16:08:29 [44msource[0m > Finished syncing daily_active_users
    2023-06-15 16:08:29 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
    Syncing stream daily_active_users 0:00:00.015667
    2023-06-15 16:08:29 [44msource[0m > Syncing stream: devices 
    2023-06-15 16:08:29 [44msource[0m > Setting state of devices stream to {'date': '2023-06-19'}
    2023-06-15 16:08:29 [44msource[0m > Read 0 records from devices stream
    2023-06-15 16:08:30 [44msource[0m > Finished syncing devices
    2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
    Syncing stream daily_active_users 0:00:00.015667
    Syncing stream devices 0:00:00.001815
    2023-06-15 16:08:30 [44msource[0m > Syncing stream: four_weekly_active_users 
    2023-06-15 16:08:30 [44msource[0m > Setting state of four_weekly_active_users stream to {'date': '2023-06-20'}
    2023-06-15 16:08:30 [44msource[0m > Read 0 records from four_weekly_active_users stream
    2023-06-15 16:08:30 [44msource[0m > Finished syncing four_weekly_active_users
    2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
    Syncing stream daily_active_users 0:00:00.015667
    Syncing stream devices 0:00:00.001815
    Syncing stream four_weekly_active_users 0:00:00.005786
    2023-06-15 16:08:30 [44msource[0m > Syncing stream: locations 
    2023-06-15 16:08:30 [44msource[0m > Setting state of locations stream to {'date': '2023-06-19'}
    2023-06-15 16:08:30 [44msource[0m > Read 0 records from locations stream
    2023-06-15 16:08:30 [44msource[0m > Finished syncing locations
    2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
    Syncing stream daily_active_users 0:00:00.015667
    Syncing stream devices 0:00:00.001815
    Syncing stream four_weekly_active_users 0:00:00.005786
    Syncing stream locations 0:00:00.001644
    2023-06-15 16:08:30 [44msource[0m > Syncing stream: pages 
    2023-06-15 16:08:30 [44msource[0m > Setting state of pages stream to {'date': '2023-06-20'}
    2023-06-15 16:08:30 [44msource[0m > Read 0 records from pages stream
    2023-06-15 16:08:30 [44msource[0m > Finished syncing pages
    2023-06-15 16:08:30 [44msource[0m > SourceGoogleAnalyticsDataApi runtimes:
    k
    • 2
    • 2
  • w

    Waliur Rahman

    06/15/2023, 4:29 PM
    Hi Folks, I am using Plural hosted version of Airbyte (0.44.8). When I am trying to connect to a Postgres database I am getting the following error. Wondering if anyone has any idea what this is about or how to resolve it. Thanks in advance.
    Copy code
    Configuration check failed
    Message: HikariPool-1 - Connection is not available, request timed out after 10002ms.
    
    Stacktrace:
    io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
    	at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:84)
    	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.createDatabase(AbstractJdbcSource.java:434)
    	at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:277)
    	at io.airbyte.integrations.source.postgres.PostgresSource.createDatabase(PostgresSource.java:99)
    	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.check(AbstractDbSource.java:85)
    	at io.airbyte.integrations.source.postgres.PostgresSource.check(PostgresSource.java:557)
    	at io.airbyte.integrations.base.ssh.SshTunnel.sshWrap(SshTunnel.java:270)
    	at io.airbyte.integrations.base.ssh.SshWrappedSource.check(SshWrappedSource.java:53)
    	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:124)
    	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
    	at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:538)
    Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10002ms.
    	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
    	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
    	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
    	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
    	at io.airbyte.db.jdbc.DefaultJdbcDatabase.getMetaData(DefaultJdbcDatabase.java:78)
    	... 10 more
    k
    u
    • 3
    • 2
  • o

    Omar Mayar

    06/15/2023, 4:36 PM
    does anyone know what the ā€œunexpected behaviorā€ described here might be? would ignoring this reset impact the cursor for upserts? not seeing much on airbyte docs so want to see how risky of a move it is to disregard
  • s

    Sharat Visweswara

    06/15/2023, 5:13 PM
    Hello, if I have multiple sources (e.g. facebook marketing) writing to the same set of tables, what is the impact of resetting one of the connections? Will it delete all the the data, including data for other sources?
    k
    • 2
    • 5
  • y

    Yoan Yahemdi

    06/15/2023, 6:15 PM
    Hello, I’m trying to create the connexion with BigQuery using Bucket as recommended. However I am facing exactly the same issue. Haven’t been able to resolve it. Could someone has some ideas or guidances? Thank you very much
    k
    • 2
    • 2
  • b

    Ba Thien Tran

    06/15/2023, 8:43 PM
    Hi I’m trying to set up Amazon Seller Partner on Cloud but cannot find the connector in the connector overview
    k
    • 2
    • 2
  • j

    Jatin Morar

    06/15/2023, 8:56 PM
    Hi, how can the s3 destination not override a file. Just add a new file per day?
    k
    • 2
    • 2
  • s

    Slackbot

    06/15/2023, 9:45 PM
    This message was deleted.
    k
    • 2
    • 2
  • a

    Ashley Pace

    06/15/2023, 9:51 PM
    Hi everyone - I'm gathering information on data replication tools in order to evaluate a few and ultimately make a recommendation to my boss. I'm seriously looking at Airbyte open source and CData Sync. Both self-hosted. CData is not free, the plan we're considering has a limit of 1 billion rows a month. Sources: MySQL, Azure SQL, Hubspot, Google Analytics, and a couple of APIs We don't have a ton of new data daily, maybe north of 1 million daily inserted or updated. The MySQL sources is our largest and it has many tables that have no identifiable incremental column AND we can't turn on CDC as we don't own the database. We'd have to replicate some of the tables once a day full and others incrementally which is fine. Anyway, all that said, I'd love to hear others' experiences with evaluating the various options out there and how you landed on Airbyte open source, aside from cost. I've pulled info on CData Sync, Airbyte, Fivetran and Stitch. Thanks!!!!
    k
    • 2
    • 2
  • m

    Madhav

    06/16/2023, 4:12 AM
    Hi, getting error while running my postgres connector , how can I increase timeout value, Does any one have idea?
    Copy code
    java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 10001ms.
    k
    g
    • 3
    • 3
  • a

    Ayushman Singh

    06/16/2023, 6:01 AM
    I have a question about the sync limits on connections. • For airbyte open source is there a limit on the
    cron
    that can be specified in the sync frequency ? • If there is a limit on the
    cron
    why is it there for open source, can it be modifed?
    k
    u
    • 3
    • 2
  • a

    archna singh

    06/16/2023, 7:01 AM
    i want to add my ssl certificate in gitlab source of airbyte so that i can make connection, is there any solution to do it.? for now i am getting error
    • 1
    • 1
  • j

    Joel

    06/16/2023, 7:49 AM
    When asking for API permissions from Facebook should we create a new app that is solely used by airbyte or should we use the existing Facebook app we have (which we use for social sign in and profile pictures in our app store app)?
    k
    • 2
    • 5
  • r

    Rishav Sinha

    06/16/2023, 8:29 AM
    Hi Team, i am facing
    Jackson exception
    while integrating Elasticsearch. can someone please help me with this ? @kapa.ai
    k
    • 2
    • 2
  • m

    Magnus Berg Sletfjerding

    06/16/2023, 8:43 AM
    Hi Airbyte! I’m working with a CockroachDB source that I want to incrementally load according to this guide and these instructions. Our database has tables with ~10-100M rows, with ~100-500K added every day. We’ve been able to test (sandboxed) delta loads of up to 500K rows without problems, but we’re wondering if it’s possible to set up a connector to not do an ā€œinitialā€ load? I.E can we pre-set the ā€œcursor fieldā€ when we create the connector the first time, and manually load our existing/past data from a dump? Related: We’ve experienced problems with the CockroachDB connector loading more than ~1M rows. I think that is related to this bug , since we get the same OOM message.
    k
    • 2
    • 5
  • c

    CJ Johanson

    06/16/2023, 1:47 PM
    hi everyone! i am very new to airbyte and am trying to play around with it on my laptop and have run into some issues when trying to connect airbyte to postgres. my goal is to host a postgres server and database on my local while also hosting airbyte on my local (running it via docker). so far, i have used pgadmin4 to, i believe, set up a postgres server and database. I have put some fake data into said database. when i go to airbyte to try and connect the postgres db as a source, i keep getting the following error:
    Configuration check failed
    State code: 08001; Message: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    i have been scouring the interwebs for a solution but im now plain ol' lost. i tried to update some setting on the postgres side. these changes include: 1. updating
    listen_addresses = '*'
    in
    postgresql.conf
    2. uncommenting
    port=5432
    in
    postgresql.conf
    if anyone can provide any insight at all, i would very much appreciate it!
    k
    • 2
    • 5
  • o

    Octavia Squidington III

    06/16/2023, 7:45 PM
    šŸ”„ Community Office Hours starts in 15 minutes šŸ”„ At 1pm PDT click here to join us on Zoom!
  • b

    Biondi Septian S

    06/17/2023, 8:50 AM
    Hello Airbyte team, please take a look at this serious issue: https://github.com/airbytehq/airbyte/issues/24097
    k
    • 2
    • 2
  • b

    Biondi Septian S

    06/17/2023, 8:51 AM
    this is a serious timeout issue on typesense destination connector
    k
    • 2
    • 2
  • b

    Biondi Septian S

    06/17/2023, 8:51 AM
    I think this is the solution, by merging this pull request below: https://github.com/airbytehq/airbyte/pull/18806/commits
    k
    • 2
    • 2
  • r

    Roy Ben Dov

    06/19/2023, 8:50 AM
    Hii! I am new to airbyte I created my first connection of data replication from a postgres db(version 15) to postgress db (version 14) all connection tests passed successfully, also sync passed but I didn't see the new data replicated. Sync mode: Incremental | Deduped + history and I cant choose Cursor Field
    k
    • 2
    • 2
  • a

    Ammar Tello

    06/19/2023, 1:31 PM
    Hi Everybody, I am new to Airbyte, and I am trying to make a connection between Postgresql and Snowflake, When I try to make the Snowflake a destination, I face this error:
    Copy code
    io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/destination-snowflake:1.0.5
    any idea regarding this? Thanks in advance 😊
    k
    • 2
    • 2
  • c

    Cameron Smith

    06/19/2023, 2:51 PM
    Hi, I'm trying to follow the airbyte tutorial to connect Google Sheets to AWS Postgres, and everything is great until I try to add Postgres as a source/destination. When I try with SSL disabled I get an error that unsecured connections ar enot allowed, and to try using require, verify-ca, verify-full. However when I try using require I get HikariPool-1 - Connection is not available. Any idea why? I can connect to the database unsecured through psql and vscode, and I've followed every step of the tutorial, but there's no mention of the ssl setting on there. Airbyte Tutorial
    k
    • 2
    • 2
1...203204205...245Latest