https://linen.dev logo
Join Slack
Powered by
# contributing-to-airbyte
  • c
    Hello, is the actual GoogleAnalytics V4 connector compatible with older google analytics version ( UA ) ? Thanks!
  • j
    Hey there, I just made a PR for mongodb v2. I was wondering how I could build the image without waiting for the next release?
    • 1
    • 1
  • p
    Wow, dbt 1.0.0 was just released! 🎉 Do we need to wait for Airbyte to do a new release that switches to dbt 1.0.0 before we can use it?
  • v
    Hi there 👋 Super happy to open my first Airbyte PR octavia partying I just wanted to share some minor fixes we did in our instance on the Airbyte's Helm Chart, and also tried to propose an easy way to manage logs on S3. I'm not really used to helm template functions, maybe I did some stuff too verbose or missed something, don't hesitate to comment ✍️ Thanks for the great product and doc octavia loves
    • 1
    • 1
  • h
  • m
    Hey team! we’re seeing almost all of our syncs fail with the following error: 
    io.airbyte.workers.WorkerException: Error while getting checking connection
    - has anyone run into this before?
    • 1
    • 5
  • m
    Hello Airbytians, is the right channel to RFR https://github.com/airbytehq/airbyte/pull/8675?
  • d
    Hey team, we’ve been seeing build failures for builds extending longer than 15 mins due to degraded performance for self hosted runners. This started yesterday, and was reported by the general community about 10 hours ago. Github just became aware of this about 30 mins ago. Follow here for the action: https://github.com/actions/runner/issues/1546 Status page: https://www.githubstatus.com/ I too lost a couple of hours today scratching my head until I found the issue. Sorry for the trouble!
    j
    • 2
    • 1
  • m
  • j
    Head's up! We're starting to move our Java development onto Java 17 today. If you're using a Java version manager like jabba/jenv it should be easy to change your configuration. Otherwise on Mac you should just run 
    brew install openjdk@17
     to get to the most recent version. If you see a version error when running Gradle commands, try running 
    ./gradlew clean
    or removing 
    .gradle
    from the root directory or Airbyte. If you're switching between 
    master
    and a current branch you may have to do this multiple times (merging latest 
    master
    into your branch after the Java 17 update should prevent the need for running into this multiple times). To configure IntelliJ, check to make sure that the in 
    Project Structure
    the 
    Project SDK
    is Java 17 and the language level is set to 17. This may happen automatically, but it likely depends on your configuration. Please let me know if you see anything surprising locally or on CI. We're first releasing this for our internal cloud project and then will soon release it for the OSS project.
    • 1
    • 1
  • r
    Hi Dev Team, I am new to Airbyte, I have a question for the team, does Airbyte support RBAC with SAML / SSO integration?
    r
    • 2
    • 4
  • o
    Hi, guys! Faced some issues during migration on Java 17 on Mac OS:
    Copy code
    airbyte-worker     | Exception in thread "main" io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998952187s. 
airbyte-worker     |    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
airbyte-worker     |    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
airbyte-worker     |    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
airbyte-worker     |    at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282)
airbyte-worker     |    at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
airbyte-worker     |    at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:182)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51)
airbyte-worker     |    at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41)
airbyte-worker     |    at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:40)
airbyte-worker     |    at io.airbyte.workers.WorkerApp.main(WorkerApp.java:189)
    What was done by me: • Switched global env to JDK 17 • Rebuild docker images • Switched IntelliJ to JDK 17 • run 
    VERSION=dev docker-compose up
    Did somebody faced some similar issue?
    • 1
    • 3
  • b
    Hi everyone! We are developing a tool that downloads Airbyte's data (sources, connections and destinations mainly), to store it inside a Git repo for future use This tool allows then to use this information (in json format) to load and restore (or configure a new Airbyte instance) as it was at "extraction time". It's quite like the Export-Import stuff inside Settings in the UI We acknowledge certain fields are hidden behind ****** in the API response (we use Git Secrets to load them when needed), but some others (like S3) are exposing critical information, i.e 
    aws_secret_access_key
    Is there anywhere in the docs, or Airbyte's repos, where we could find a list of the hidden (and non-hidden) fields for the different connectors? Finding them out by trial and error, taking into account the vast number of connectors available, is extremely difficult.
    m
    d
    • 3
    • 16
  • m
    I am on 0.33.11 now and getting this
    Copy code
    021-12-13 22:57:58 INFO i.a.s.RequestLogger(filter):95 - {cloud_workspace_app_root=/workspace/server/logs} - REQ 10.30.33.154 GET 200 /api/v1/health
======= service endpoint: <http://airbyte-minio>:%!s(int=9000)
    • 1
    • 5
  • r
    maybe the better question is how to set up this parameter of .env under kube/overlays/stable-with-resource-limit
    Copy code
    # Worker pod tolerations and node selectors
JOB_POD_TOLERATIONS=
JOB_POD_NODE_SELECTORS=
    • 1
    • 8
  • e
    Hi. How can I find the dbt model which was used for default normalization? I need to make a minor tweaks but don't know much about DBT so I though if I look at the model that what actually used, it would be easier for me to just make those minor tweaks without writing everything from scratch
    c
    • 2
    • 1
  • s
    Hi, Could someone point me to the source or a quick brief about how the data is transferred between Source and Destination and once the pipeline is triggered, i.e If I use 
    Salesforce
    as the source and 
    GCP GCS
    as a destination, do airbyte pull all the data from the source first then stage it on temporary storage and then start streaming the temporarily stored data to the destination i.e 
    GCP GCS
    ?? I see 
    streams
    usage but would like to understand how this is handled with large datasets like 20 GB or 30 GB data. Disclaimer : Just started using Airbyte.
    • 1
    • 1
  • c
    Hi, is there any special reason why there is no connector with Twitter? Thanks!
    • 1
    • 1
  • g
    i have CDC connector. i would like to add a table without performing a full reset. i know that you cant do that via the UI and implications from the pointer on the binlogs. but what are the steps to doing this manually. i was thinking: 1. create new connector with specific table to perform a full sync (connector B) 2. add table to existing connector without resetting (connector A) 3. remove connector B 4. if i have incremental dedupe on all tables, i am assuming this should be fine following subsequent syncs?
    • 1
    • 1
  • m
    Does anyone have experience writing logs for EC2 deploys to S3? Our workers seem to be generating a lot of data, and EBS storage is filling up quickly
    m
    • 2
    • 5
  • j
    Hi, I found an issue with the Postgres source connector 0.3.14+, would the best thing to do be create an issue/bug report on github? Specifically, the issue seems related to this change that excludes the tables where the DB user doesn’t have select privileges on the table. In my case, I have some materialized views, which the DB user does have select privileges on, but the materialized views don’t show up in the catalog in Airbyte.
    • 1
    • 3
  • a
    Parametrized source/destination connector , Is it supported ?
    u
    • 2
    • 4
  • e
    Not sure but looks like deduped history doesn't work as expected when using custom normalizations. My undestanding is that the deduplications gets overwritten by my custom dbt? source - MongoDB, destination - BQ
    u
    • 2
    • 11
  • e
    I've exported Airbyte normalization dbt models, edited them a little, pushed them to github repository and am trying to make them run during sync. getting this error 
    Database Error in model messages (models/airbyte_incremental/whatsapp/messages.sql)
    Syntax error: Expected "(" or keyword SELECT or keyword WITH but got keyword CREATE at [9:5]
    The model is trying create an intermediary table like it did in the original models
    u
    r
    +2
    • 5
    • 27
  • r
    Hi, not sure where to post this but there is a bug in the Google Ads connector where it is not able to normalize the keyword_report table (log error: 
    100038 (22018): Numeric value 'KEYWORD' is not recognized
    ) because of a wrong type definition here: https://github.com/airbytehq/airbyte/blob/13ac480a8b0024360d35c20fd6d640296b57f137[…]source-google-ads/source_google_ads/schemas/keyword_report.json
    u
    d
    • 3
    • 2
  • d
    Just wondering if theres a security channel or anyone looking at the security side of airbyte, Ive done a scan from the public repo using snyk free tier (Disclaimer: i do work for them) and it looks like theres some possible known open source vulns and some container config issues to be looked at. Possible code issues too, but those need exploring some more to see if they are false positives. continued in thread...
    u
    c
    • 3
    • 9
  • s
    Hey everyone! Wasn’t entirely sure where to post this as it’s related to configuration and integration. I’m looking to use DBT for normalization into Redshift. I’d love for all downstream models of my raw data to be ran anytime the airbyte job runs. This is possible with the 
    dbt run --select <model>+
    however to get the raw model to my dbt project, I don’t love the suggestion of hopping into the airbyte container, determining the normalization directory, and grabbing the generated dbt model, syncing that back to a dbt repo, and then integrating the dbt repo on the airbyte job configuration. Has anyone found a better way of exporting dbt models? I’m also planning on running on dbt cloud and this configuration means that I cannot “deploy” models via dbt cloud when the source dataset changes. Alternatively, I could trigger the dbt cloud api but that isn’t possible directly via airbyte, which means I would then have to use Airflow to schedule the airbyte job and then kick of the dbt cloud job. This means that Airflow code must be written for new data sources and I’m looking to keep data integration and normalization self-service to speed up development time for new datasets. If anyone has suggestions I’m all ears.
    z
    • 2
    • 8
  • z
    Hi Everyone, I have a problem that, when selecting Raw data - no normalization, data get inserted as json blob but with basic normalization I get error. ERROR: Database Error Normalization Fails. I am new to airbyte can some one help. As json blob data get inserted in form of rows like: eg row 1: {key:value} row 2: {key:value} row 3: {key:value} I would like to have keys as column names. I am using postgres.
    h
    u
    • 3
    • 2
  • w
    Hi All, I am creating a connector for a system that can also list the deleted rows from a certain point in time. Is there a way to mark rows as deleted in an incremental stream? Would it work if I add ab_cdc_deleted_at and ab_cdc_updated_at columns to my output?
    u
    b
    +3
    • 6
    • 10
  • f
    Looking into using cloudsql postgres source and wondering if pgoutput is more performant/stable than wal2json ?
    u
    • 2
    • 1
1...2021222324Latest