Is there a reason we have

checkOperation

as a separate endpoint instead of performing validation of the

OperatorConfiguration

internally in `createOperation`/`updateOperation`?

Hey folks. What’s the rule for language choice on the server and connectors? From a quick glance here’s what I gathered: • For database connectors -> Java • For REST API connectors -> Python • For airbyte backend server -> Java What’s the justification(s)?

I need help I want to know the db/ directory where the configuration is stored when new sources are added Please help me

To just continue a bit on

justifications

- am wondering of airbyte team could share a sentence or two on what’s like building a production service using temporal(cadence)? what are the pros/cons vs other alternatives you’ve considered? Thanks a lot!

Hello, Airbyte Team! I have problem. Can you help me?

Airbyte team - I’m running on a brand new clean branch.  I run ./gradlew format and there are whitespace changes in files.  I didn’t expect that to be possible with the build-enforced whitespace formatter.  It affects 4 different connectors, and hit java, json, and py files. Is this a side effect of not automatically running the connector build when building platform? Perhaps we need to see if we can still force the whitespace format, even without doing a full build?

When I try to a brand new Airbyte server from the latest commit from the master branch, the server will break at this line: https://github.com/airbytehq/airbyte/blob/master/airbyte-analytics/src/main/java/io/airbyte/analytics/SegmentTrackingClient.java#L93 because the

customerId

is

null

. I am digging into it, but does anyone know top of their head about the potential root cause?

This is not urgent, but something to consider for long-term: Do we have a way to reuse a chunk of spec.json across a bunch of connectors? My use case is ssh tunnel configuration, which is quite verbose and would be identical across many, many connectors. I’d rather DRY that section of json than have it in a ton of places. That would also facilitate internationalization/localization someday if necessary, rather than spreading explanatory text around in a ton of places. The connection configuration feels like the right place to expose it to user configuration, since it includes host and port information per database instance. But the json itself could use an import/template/abstraction instead of existing in tons of connectors independently.

hello, is there actually any way of filtering the data from a source database to a destination? Let’s suppose I have a table X on my source and I want to filter something ( for example: where X.id in (1,2,3) ), is there some “Airbyte way” of doing this kind of filters or the better way to to hat may be my just building a view (filtered) in the source database and import the entire view?

Hey Guys, facing an issue. When I import data (postgres to postgres) and select basic normalization my destination has both data tables and raw airbyte tables. Anyway to not have the raw tables.

Airbyte team: Do we have anybody who has extensive experience in ssh key formats (especially in java) and/or TLS protocol negotiation familiarity? My ssh tunnel work using open source libraries is hitting some very technical obstacles, for which the “just google it” solution doesn’t work.

I'm running

./gradlew airbyte-integrations:bases:base-normalization:build

and it's failing on Flake step trying to validate the venv libs?

Task :airbyte-integrations:bases:base-normalization:flakeCheck FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:flakeCheck'.
> Python call failed: .venv/bin/python -m flake8 . --config /home/marcos/code/airbyte/tools/python/.flake8

./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:4991:5: E731 do not assign a lambda expression, use a def
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5026:21: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5074:15: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5216:15: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5569:28: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5570:12: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5578:27: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5585:26: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5586:12: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5655:1: E302 expected 2 blank lines, found 1
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5870:23: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5894:23: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py:5916:26: E741 ambiguous variable name 'l'
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/appdirs.py:572:9: F401 'win32com.shell' imported but unused
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/appdirs.py:577:13: F401 'ctypes.windll' imported but unused
         ./venv/lib/python3.7/site-packages/pkg_resources/_vendor/appdirs.py:582:17: F401 'com.sun.jna' imported but unused

Hey Team! I wonder how easy is for you to use Pycharm with the mono-repo. I’m having hard time to control it since in case I work on multiple connectors I need to open each in a different window and such. Any better way to handle multiple interperters in Pycharm in the same window?

We finished the implementation of using DynamoDB as the destination. We are curious how to build the docker on the local machine, which has the GUI for the new added DynamoDB that can use to test

Hi there. Could someone take a look at the Python CDK deploying pipeline? Tried to run

/publish-cdk --dry-run=true

and got

Error: Unexpected inputs provided: ["--dry-run"]

And for the

/publish-cdk

got

Uploading distributions to <https://test.pypi.org/legacy/>
Uploading airbyte_cdk-0.1.9-py3-none-any.whl

  0%|          | 0.00/58.2k [00:00<?, ?B/s]
100%|██████████| 58.2k/58.2k [00:00<00:00, 90.5kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from <https://test.pypi.org/legacy/>
File already exists. See <https://test.pypi.org/help/#file-name-reuse> for more information.
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Publish Python Package

The latest version in the pypi is 0.1.8 and I'm uploading 0.1.9. Here's a PR https://github.com/airbytehq/airbyte/pull/5425 , extracted from https://github.com/airbytehq/airbyte/pull/5223

Idea for discussion - would people find it valuable to know which docker container holds the logs for errors that pop up? I know I am sometimes doing a lot of guesswork as to where to track down a stack trace when the UI shows a short error message and no details. I discovered that from inside a running docker container, we can obtain the hash form of that container id, with the following:

cat /proc/self/cgroup | head -1 | cut -d'/' -f3

So theoretically it would be possible to tell the running app which container it’s in (via an env var or something) so that it could report that information with any exceptions, and we could pass that up to the UI for a More Details display or somesuch. This might have some value as horizontal scale-out happens, too — sometimes a particular node gets corrupted and it’s helpful to know which node in a cluster is spitting out the errors. My primary goal, though, is just to make it easier on ops and developers to discover the underlying stack trace when something breaks anywhere. Thoughts?

@[DEPRECATED] Marcos Marx re: temporal issues. Do you have any examples of what you’ve been debugging or what you need to look at? From what I’ve seen, 99% of temporal errors we’ve seen have involved the underlying instance running out of space or other VM-level issues. Are you seeing something different where temporal itself is running into issues?

This

airbytebot

account occasionally publishes some docker images. Do we know who / what is that?

@Liren Tu just to confirm, we seed connectors directly from the yaml file today. When adding a new connector or updating a version, we no longer need to generate the json files?

I'm running

./gradlew format

on master and getting a failure. Anyone else is having the same issue? head ~8c1a5d57c795

> Task :spotlessJava FAILED
Step 'licenseHeader' found problem in 'airbyte-integrations/connectors/destination-keen/src/test/java/KeenTimestampServiceTest.java':
Unable to find delimiter regex ^package 
java.lang.IllegalArgumentException: Unable to find delimiter regex ^package 
        at com.diffplug.spotless.generic.LicenseHeaderStep$Runtime.format(LicenseHeaderStep.java:199)
        at com.diffplug.spotless.generic.LicenseHeaderStep$Runtime.access$100(LicenseHeaderStep.java:145)
        at com.diffplug.spotless.generic.LicenseHeaderStep.lambda$build$7(LicenseHeaderStep.java:116)
        at com.diffplug.spotless.FormatterFunc.apply(FormatterFunc.java:32)
        at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:78)
        at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:76)
        at com.diffplug.spotless.FilterByFileFormatterStep.format(FilterByFileFormatterStep.java:42)
        at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
        at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:201)
        at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:188)
        at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:71)
        at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskIm

Chris and I feel we are configured to retry excessively today - we do 3 attempts with 3 runs in each attempt - that's 9 'tries' in total. This is not good for rate limits. Errors also take a while to surface. Thinking of pushing this down to 3 attempts of 1 run each. Thoughts?

Hello team, any best practice to deploy airbyte in production given that there is no authentication, I want to link it to a subdomain of mine for my team to be able to access it, anyone have tried this before?

I'm wondering how it is that you're not defaulting to e.g. aiohttp or some other async http client in the python CDK?

Hello team, we just created a PR #5561 for the new

DynamoDB

destination. Please let us know if anything else needs to be done!

how do we update the seeds now if we make changes to

source_definitions.yaml

I see

./gradlew generateSeed

doesnt work anymore

what is the best way to run spotless on all projects?

hey all, is there a python wrapper for airbyte API?

@Jared Rhizor (Airbyte) do we have a timeout for our synchronous temporal jobs?

do we have a test oracle database lying about?

Liren, can I confirm something with you regarding https://github.com/airbytehq/airbyte/pull/4670. Cloud was previously on a version before this. We then updated to 0.29.12 (with configs existing in the database). My understanding is the configs were copied from the volume to the database as part of backward compatibility changes in the PR, so we should be good to delete the volume without any additional migration work. Is that right?