Jonathan Alvarado
02/24/2022, 5:58 PMsource-postgres
source and I have a problem where I’m not able to list any tables in my source even though my user is able to select from the tables.
I see there are a couple relevant tickets.
* https://github.com/airbytehq/airbyte/pull/9875
* https://github.com/airbytehq/airbyte/pull/10173
I debugged and found that in AbstractJdbcSource.discoverInternal()
where it filters tables with excludeNotAccessibleTables
protected Predicate<JsonNode> excludeNotAccessibleTables(final Set<String> internalSchemas,
final Set<JdbcPrivilegeDto> tablesWithSelectGrantPrivilege) {
the tablesWithSelectGrantPrivilege
variable was empty.
So because of that it entered this branch:
if (tablesWithSelectGrantPrivilege.isEmpty()) {
return isNotInternalSchema(jsonNode, internalSchemas);
}
and the PostgresSource
has commented out the isNotInternalSchema()
method to always filter out the
@Override
protected boolean isNotInternalSchema(JsonNode jsonNode, Set<String> internalSchemas) {
return false;
}
If the isNotInternalSchema
wasn’t overriden by postgres then my tables show up properly.
I’m trying to understand the logic behind this?
Whether the problem is with getPrivilegesTableForCurrentUser
or that the isNotInternalSchema
should not be overridden?
Can someone shed some light on the philosophy behind how the AbstractJdbcSource.excludeNotAccessibleTables
is supposed to work and the reasoning behind the
PostgresSource.isNotInternalSchema
overriding?Jordan Velich
02/24/2022, 11:36 PMPhilip Corr
02/25/2022, 1:59 PMactivities
endpoint for the source-facebook-marketing
connector. The documentation for this is here: https://developers.facebook.com/docs/marketing-api/reference/ad-activity/v13.0
One thing I noticed is that the latest documentation seems to be missing parameters which can be seen here on an old version of the docs:
https://web.archive.org/web/20210118065429/https://developers.facebook.com/docs/marketing-api/reference/ad-account/activities/
I can see the relevant activities pull is supported by the facebook-python-business-sdk
that airbyte is using for this connector. I’m curious what the teams thoughts are on adding a new stream for activities
which uses these parameters given this documentation issue?flow
02/25/2022, 7:27 PMinvoice
and invoice_contacts
are linked together with airbyte_hashid
....
that makes no sense for me.. if the main "`contact`" table witrh all its informations about contacts isn't linked together with invoice
the whole normalization is useless ...
i mean i have everywhere the IDs in the api collected objects.. this should be easy to link together for normalization...
whats wrong here? isnt this feature build in?
i use BigQuery normalization!
Thanks alot!Hưng Lê
02/26/2022, 1:10 AMSai Sriram
02/28/2022, 12:33 PMNathan Gille
02/28/2022, 4:23 PMThomas Fox
02/28/2022, 7:58 PMAlexandru Manolache
03/01/2022, 9:30 AMJavier Llorente Mañas
03/07/2022, 11:12 AM_airbyte_ab_id,_airbyte_emitted_at,_airbyte_normalized_at,
_airbyte_survey_questions_hashid
I will like to know if it is possible to add another field _airbyte_connector_version
? I think it is super useful API schemas change over time, this field will help downstream modeling when API schemas changed, without a need of having two different tables or el jobsWill Watkinson
03/09/2022, 4:00 PMdocker-compose up
. All of the python and docker commands in this guide succeed, but when I try to add the source from the UI at localhost:8000
, it errors out. It seems like it's unable to pull the spec based on the error message:
airbyte-worker | Caused by: io.temporal.failure.ApplicationFailure: message='Error while getting spec from image airbyte/source-kyriba:dev', type='io.airbyte.workers.WorkerException', nonRetryable=false
Any ideas how to get this source running locally?Narender Kumar
03/14/2022, 11:02 AMEnrico Tuvera Jr.
03/14/2022, 12:57 PMJan Erik Herrmann
03/14/2022, 4:24 PMArvi
03/15/2022, 2:34 AMArvi
03/15/2022, 3:03 AMRamon Vermeulen
03/15/2022, 8:04 AMWill Massey
03/15/2022, 10:36 AMInternal Server Error: Get Spec job failed
when loading their Docker Image.Ethan Veres
03/15/2022, 2:36 PMPrasad Chalasani
03/15/2022, 3:33 PMWill Massey
03/15/2022, 4:07 PMAhmed Buksh
03/16/2022, 10:40 AMAlpana Shukla
03/16/2022, 12:42 PMYanni Iyeze - Toucan Toco
03/16/2022, 1:05 PMactivities/email.json
Charles VERLEYEN
03/16/2022, 2:34 PMNathan Gille
03/16/2022, 2:48 PMHttpStream
and AbstractSource
. All the docs seem to just point to Http examples.Jordan Scott
03/16/2022, 4:42 PMEthan Lu
03/16/2022, 4:48 PMManish Tomar
03/17/2022, 7:42 AMManish Tomar
03/17/2022, 7:42 AM