I’m using the latest 0.4.8

source-postgres

source and I have a problem where I’m not able to list any tables in my source even though my user is able to select from the tables. I see there are a couple relevant tickets. * https://github.com/airbytehq/airbyte/pull/9875 * https://github.com/airbytehq/airbyte/pull/10173 I debugged and found that in

AbstractJdbcSource.discoverInternal()

where it filters tables with

excludeNotAccessibleTables

protected Predicate<JsonNode> excludeNotAccessibleTables(final Set<String> internalSchemas,
                                                           final Set<JdbcPrivilegeDto> tablesWithSelectGrantPrivilege) {

the

tablesWithSelectGrantPrivilege

variable was empty. So because of that it entered this branch:

if (tablesWithSelectGrantPrivilege.isEmpty()) {
        return isNotInternalSchema(jsonNode, internalSchemas);
      }

and the

PostgresSource

has commented out the

isNotInternalSchema()

method to always filter out the

@Override
  protected boolean isNotInternalSchema(JsonNode jsonNode, Set<String> internalSchemas) {
    return false;
  }

If the

isNotInternalSchema

wasn’t overriden by postgres then my tables show up properly. I’m trying to understand the logic behind this? Whether the problem is with

getPrivilegesTableForCurrentUser

or that the

isNotInternalSchema

should not be overridden? Can someone shed some light on the philosophy behind how the

AbstractJdbcSource.excludeNotAccessibleTables

is supposed to work and the reasoning behind the

PostgresSource.isNotInternalSchema

overriding?

Hi, I notice when the airbyte scheduler and/or worker pods restart, the sync jobs show up as failed on the Airbyte UI, despite still running. Is this behaviour expected?

Hi everyone, i’m currently working on adding an

activities

endpoint for the

source-facebook-marketing

connector. The documentation for this is here: https://developers.facebook.com/docs/marketing-api/reference/ad-activity/v13.0 One thing I noticed is that the latest documentation seems to be missing parameters which can be seen here on an old version of the docs: https://web.archive.org/web/20210118065429/https://developers.facebook.com/docs/marketing-api/reference/ad-account/activities/ I can see the relevant activities pull is supported by the

facebook-python-business-sdk

that airbyte is using for this connector. I’m curious what the teams thoughts are on adding a new stream for

activities

which uses these parameters given this documentation issue?

Hello, i need help with normalization.. i dont get how this can be okey... is there a setting or why the normalization will not link referenced objects together?? For example: In the invoice api: i have the parent object "`invoice`" with ID which has an object named "`contact`" with an ID In the contact api: i have the object "`contact`" with an ID. as result i get... a table "`invoice`" with all invoice informations a table "`invoice_contact`" with just the object ID and some Airbyte fields a table "`contact`" with all contact informations but only the

invoice

and

invoice_contacts

are linked together with

airbyte_hashid

.... that makes no sense for me.. if the main "`contact`" table witrh all its informations about contacts isn't linked together with

invoice

the whole normalization is useless ... i mean i have everywhere the IDs in the api collected objects.. this should be easy to link together for normalization... whats wrong here? isnt this feature build in? i use BigQuery normalization! Thanks alot!

Hi everyone, I need help with destinations/postgres. I don't know how I can insert or update records with exists primary key? Currently, all data I crawed from HTTP request, are appending to destination with duplicated unique field. I just want to update new data if this primary key is exists and insert if none.

Anyone has successfully built a custom HTTP API connector ?? and willing to connect for sometime ??

Why do some sources have stream specific schemas if airbyte also requires a configured catalog? What is the point for both?

Hey! My company is window shopping for ELT tools. We were wondering if airbyte would work with a product feed we have as the source. It's a live product link feed in XML format

Hello, i have one question reg <*Incremental Sync - Deduped History>* Are the records(rows) deleted from the source table deleted also from the destination table? The documentation only covers updates and new records. I am pulling from Oracle and loading to Bigquery Thanks! PS. I guess not, as this basically makes a select using the PK and cursor

Hey! I see that the Airbyte Normalization stage ingests these fields into the end table to debug downstream issues in the DWH.

_airbyte_ab_id,_airbyte_emitted_at,_airbyte_normalized_at,

_airbyte_survey_questions_hashid

I will like to know if it is possible to add another field

_airbyte_connector_version

? I think it is super useful API schemas change over time, this field will help downstream modeling when API schemas changed, without a need of having two different tables or el jobs

I am running into an error adding the source I created to Airbyte running locally via

docker-compose up

. All of the python and docker commands in this guide succeed, but when I try to add the source from the UI at

localhost:8000

, it errors out. It seems like it's unable to pull the spec based on the error message:

airbyte-worker      | Caused by: io.temporal.failure.ApplicationFailure: message='Error while getting spec from image airbyte/source-kyriba:dev', type='io.airbyte.workers.WorkerException', nonRetryable=false

Any ideas how to get this source running locally?

Hi Team, How to provide path of the text file stored on my windows machine eg; C:\test.txt to read the data using File connector ? I have airbyte running in docker container. can you please suggest?

apart from shopify, are there any other connectors that use substreams? i'm still not quite getting how they work.

Hi Guys, I want to use the kafka connector but with SASL as authentication Method and a custom CA Certificate. Is anybody already working on providing custom certificates? If not can anybody point me in a direction how to provide the certificate to the kafka connector. Is there another connector which has such an option?

Hello All, I am just curious to see if anyone has connected to onfleet from airbyte?

Hi @Edward Gorkh, This is Arvi from Australia.

Hi! So how should I set that up If I don't let users fill in the tables they want to sync in the configuration. This is how the set up connection screen looks for me right now. It is one endpoint (GetConnections) which has dynamic sub endpoints under it which are different for every user. An example, /connections/foo and /connections/bar for use A, but it can be /connections/salaries or /connections/employees for user B. So my plan was to let them fetch the connections in the configuration part, and let them select the connections they want to user, in this example of user B ["salaries", "employees"]

Hi Guys, Thanks for your help. It appears that it was an issue with the Docker image that I built on my M1 Mac. I instead built the Docker image on my GCP cloud instance instead. I’ll add the steps below in case anyone in the future has the same

Internal Server Error: Get Spec job failed

when loading their Docker Image.

@Sherif Nada let me know if that all makes sense!

Thanks @Marcos Marx (Airbyte) so you mean I could write an intermediate adaptor that transforms Airbyte raw (Json) output tables (and possibly the Basic Normalized tables), to produce tables which are then compatible with the FiveTran dbt transformers?

Hi, Super quick question: does Airbyte expect sequential and/or numerical tags on Docker images for connectors? e.g. can I use commit hashes as my docker image tags or do I need to use 0.0.1, 0.0.2, 0.1.0 etc?

Hi everyone, I just started working with Airbyte and bit confused with the development flow so just wanted to ask whether everyone has to go through gradle build first and then up the containers in order to reflect every change or I am doing something wrong?

Hi ...I am trying to create a custom connector and everytime I try to deploy Custom connector it gives me an error "Internal Server Error: Get Spec job failed". I have tried with docker image on docker hub repository and azure container repository both.....but still no success.CAn somebody help me resolving this issue?

[Salesloft Connector] I’m looking forward to ETL Salesloft data (i’m stucked to get a refresh token 😂). Based on at I read on docs and github about this integration, we can only get data from the following endpoints right ? • CadenceMemberships • Cadences • People • Users What do you recommend if we need to get data from other endpoints like

activities/email.json

Hello, we were checking the Mongo DB connector and see that it is not leveraging the "change stream" service from Mongo DB. Are we mistaken or is it an upcoming feature ?

are there any examples other than mailchimp that uses incremental streams for non http and non singer taps? Or is how mailchimp built their incremental the only way? I'm trying to avoid inheriting

HttpStream

and

AbstractSource

. All the docs seem to just point to Http examples.

Hello all, I’m not sure what the appropriate etiquette is for asking this, so my apologies if I’m doing this wrong 😬 … I’m new to airbyte (and APIs for that matter) but I could have our dev team build connectors if needed. As a first step though, I was wondering if anyone has already built a source connector for LinkedIn Company Pages (not LinkedIn Ads)?

question: anyone know how to access the credit card payment table through airbyte?

Does anyone here have Simple API connector which can fetch JSON data, it don't even need Authentication

It is an very simple API request