@Jared Rhizor (Airbyte) is https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connector-templates/generator/build.gradle#L20 1. part of the input files Gradle uses to determine incremental caching 2. looks like there is a slight bug since we aren't actually referring to the right files yeah?

I frequently see this transient error in builds

* What went wrong:
Execution failed for task ':airbyte-cdk:python:checkPython'.
> Python call failed: python3 -m virtualenv .venv

anyone see this also?

What’s the status of https://github.com/airbytehq/airbyte/pull/3289 @charles?

https://github.com/airbytehq/airbyte/pull/3343/files can someone look this please 😬

I am not sure if this is the “right” channel for questions, but here it goes. My understanding from the docs on namespaces is that if a source has a namespace then that can be used to define the target schema. Is that correct and also, is there no current way to specify a namespace for a source that does not have one? I would like to define one target database, but have each source land in a different schema. I believe the only way to achieve this today is to create one destination per source and specify the schema there.

I don't suppose anyone knows how to override a Docker container's standard behavior and make it boot into Bash rather than the process it's designed to startup as? (I have a malfunctioning container that starts up, errors out, and shuts down again too quickly for me to connect to it in Bash and poke around to figure out the problem.)

Hi Everyone. What is the latest version of AirByte?

Hi Team, Would it be possible to try out the CDC for MySql. I understand it's not stable right now, but would love to test it.

Hey Everyone. We are having about 100 connections is there any plan to get pagination on all connections screen.

Every time I go to run tests, IntelliJ wants to rebuild the entire Docker ecosystem. Gradle is intelligent enough to know that the Java projects are "UP-TO-DATE", but it completely fails at doing so on the Python side of things, wasting several minutes rebuilding stuff that I haven't even touched, over and over again. The task

:airbyte-integrations:bases:airbyte-protocol:airbyteDocker

seems to be a particularly egregious offender, spending a ton of time rebuilding Docker containers and downloading stuff from Pip. Anyone know of any way to fix this?

I think the answer to this is no, but want to confirm. Is there no way to trigger an action after a load is complete? e.g. triggering dbt or some other job?

Is it possible to access to standard output logs from tests ran on github? Let’s say for example, if i run the tests locally, everything is passing successfully but not in the CI workflow: For example, to see more details on the exceptions or test failing here: https://github.com/airbytehq/airbyte/runs/2583822491?check_suite_focus=true How do I debug to see what is different on github vs my local computer?

Hi, we noticed that Airbyte creates separate tables for all arrays and object data types for any stream. Is there a way to use the native support of record and repeatable for Bigquery? If not, any plans of implementing this?

When is "Support of data lakes, starting with Delta Lake" coming? https://github.com/airbytehq/airbyte/blob/master/docs/project-overview/roadmap.md#coming-within-a-few-weeks--months

Hey folks. Just curious, what is the rationale for having source as independent entity in the API ? I assume you can't reuse same source (because it has specific credentials/settings) in multiple connections, but you can share destinations.

@charles you recently got rid of the on disk queue right? so in theory we should no longer have disk space issues?

If I want to use the API to get the sources / destinations / connections, I need the workspace ID

Also, is the schedule field in https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create the same as the sync frequency in the UI settings?

Anyone have any idea why I'm seeing this error?

141223.148 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - ERROR StatusLogger Error parsing jarfile/airbyte/lib/io.airbyte-airbyte-commons-0.22.3-alpha.jar!/log4j2.xml

141223.155 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - javax.xml.parsers.ParserConfigurationException

141223.155 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at oracle.xml.jaxp.JXDocumentBuilderFactory.setFeature(JXDocumentBuilderFactory.java:374)

141223.156 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:213)

141223.156 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:205)

141223.156 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:194)

141223.156 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92)

141223.157 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46)

141223.157 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:453)

141223.157 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:385)

141223.157 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:260)

141223.158 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:615)

141223.158 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:636)

141223.158 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)

141223.158 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)

141223.159 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)

141223.159 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)

141223.159 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:121)

141223.159 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)

141223.169 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)

141223.169 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)

141223.169 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:355)

141223.170 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:380)

141223.170 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - at io.airbyte.integrations.destination.jdbc.AbstractJdbcDestination.<clinit>(AbstractJdbcDestination.java:46)

141223.208 [pool-4-thread-1] ERROR io.airbyte.workers.DefaultCheckConnectionWorker - ERROR StatusLogger No logging configuration

Hey guys, I have this MongoDB collection with thousands of docs, • connected airbyte to it • It tries to find the schema. • returns me some fields It does not contain fields that were seldom used in the collection. I checked the code https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mongodb/lib/mongodb_types_explorer.rb Apparently the approach it takes is just check the first 1000 records. So, even this is not sampling of any sort. Correct? Is there a way to edit the fields(add to it) in UI or backed API for the mongo source connector? For now I am trying to read https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html but would appreciate any help

If I'm seeing

java.lang.NoClassDefFoundError: com/google/common/collect/Streams

when running tests, what's the most likely culprit? Googling suggests that there's likely something missing from the classpath, but this error is being raised from core Airbyte code that I'm pulling in as a dependency, so it doesn't feel like my project should need something added to its classpath. On the other hand, I haven't touched those modules, and it doesn't seem as if other tests that depend on them have any trouble running, so the problem has to be in my project somewhere... right?

Has anybody faced an issue with airbyte-temporal not starting and writing next errors

airbyte-temporal    | + nc -z db 5432
airbyte-temporal    | nc: bad address 'db'
airbyte-temporal    | + '[' 1 -eq 0 ']'
airbyte-temporal    | + echo 'waiting for postgresql to start up'
airbyte-temporal    | + sleep 1
airbyte-temporal    | waiting for postgresql to start up

@charles : @Liren Tu ran into an issue running migrations where the version on 0.22.x wasn’t able to be validated on 0.23.0 when running the migration.

I just found that

JdbcBufferedConsumerFactory.toWriteConfig

has access to a

NamingConventionTransformer

instance, and

NamingConventionTransformer

has methods for generating table names, but it's not making use of them. (See lines 109-110.) Instead, it's unilaterally imposing a specific naming convention that doesn't work with all databases. Is there any rationale for this?

Does anyone else have trouble getting the build to reliably create new images on their local machine? My workflow:

* make a change in some core module, e.g. add a print statement in the server
* ./gradlew :composeBuild
* VERSION=dev docker-compose up -V
* expect print statement to fire

I would expect the print statement to run, but this isn't reliably happening in this workflow. It seems like there's some series of steps I need to run to get it to pick up changes. Still trying to figure out what the exact steps are (maybe includes

./gradlew clean

). Anyone else experiencing similar?

Any Oracle gurus in the house? https://dba.stackexchange.com/questions/291965/unable-to-connect-to-an-oracle-12c-docker-container

@Jared Rhizor (Airbyte) I was reviewing our truncation / hashing logic in normalization and I had a question come up. We have these two conditions:

# if everything fits except for the parent, just truncate the parent
    elif (len(norm_child) + len(json_path_hash) + len(norm_suffix)) < (max_length - min_parent_length):
        max_parent_length = max_length - len(norm_child) - len(json_path_hash) - len(norm_suffix)
        return f"{norm_parent[:max_parent_length]}_{json_path_hash}_{norm_child}{norm_suffix}"
    # otherwise first truncate parent to the minimum length and middle truncate the child
    else:
        norm_child_max_length = max_length - min_parent_length - len(json_path_hash) - len(norm_suffix)
        trunc_norm_child = name_transformer.truncate_identifier_name(norm_child, norm_child_max_length)
        return f"{norm_parent[:min_parent_length]}_{json_path_hash}_{trunc_norm_child}{norm_suffix}"

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-n[…]rmalization/normalization/transform_catalog/stream_processor.py Could you help me understand why we would ever want to prefer this elif case over the else case? I think anything that the elif case hands can be handled by the else case and it has the added benefit of including at least some part of the parent.

Hi, sorry i'm not sure where to post this but i'm in the office hours zoom call with another person also looking to talk to someone from airbyte. But we dont see anyone from airbyte joining the call

And Oracle just keeps getting weirder. Anyone know what to do about this one? https://dba.stackexchange.com/questions/292033/

I just noticed that test containers has a docker compose module. I wonder if that would be a good choice for our acceptance tests (instead of managing the lifecycle of the airbyte separately outside of the tests). Anyone ever use this before?