do we have a preferred unit testing solution for python?

this is a feature we could support pretty easily relative to singer. https://github.com/datamill-co/target-postgres/issues/138

Is it OK if I kill all the gitbook variants that we have? I have hope that they fixed their issue of creating new assets all the time after seeing john's PR

Is it expected that the base script for integrations don't support

write

?

I have a build issue on my branch https://github.com/airbytehq/airbyte/runs/1229054423?check_suite_focus=true

I quickly do you think we will have our existing our connectors migrated to our protocol? (contemplating not doing a compatibiliy layer for singer in the worker code)

anything i can do to help on the worker migration? it’s blocking merging a bunch of the destination stuff, but nbd, can push on to implementing BQ in java.

Are you OK if I add

abprotocol

temporarily in images that implement the airbyteprotocol ?

anyone have a good way to run the build for PRs submitted by open source contributors? Would love to merge christophe’s stuff.

is it intentional that integration tests now run as part of

./gradlew build

?

If a log is output from an integration docker container but not as an AirbyteLogMessage, what is the intended behavior? It just gets thrown out but the worker still carries on?

re: bug introduced by update to docker-compose: got to a place where i could get the app running in dev one time with the named volumes, but can’t get it to start a second time without having to clear out all the volumes first. will pick it up again tomorrow.

got the first sync to run on the new Airbyte Worker (the non singer one). With Jared’s exchangerate on the python base and my csv-destination on the java base!!!!

In our protocol should each message still be one a single line (like in singer) or are we allowing multiline?

docker run --rm -i -v /tmp/dev_root/workspace:/data -v /tmp/airbyte_local:/local -w /data/27/0 --network host airbyte/source-exchangeratesapi-singer-abprotocol:dev check --config tap_config.json
{"type": "LOG", "log": {"level": "INFO", "message": "Ping response code: 200"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Check succeeded"}}

@Jared Rhizor (Airbyte) shouldn’t this be returning a

StandardCheckConnectionOutput

@Jared Rhizor (Airbyte) - do you know if the python json schema lib you’re using supports references in the schema?

Should we commit generate type files to Git?

master is green

Is there a way to get imports to work properly for python in intellij? I set the module sdk to be the virtual env that I created for base-singer but intellij still is unhappy.

These two PRs should get us across the finish line on using airbyteprotocol with both java and python. We can now run a pair of integrations fully on the airbyteprotocol. Only blocker now is migrating our remaining two existing sources (pg and stripe) to use the new protocol and then we can switch (and nuke the old singer stuff): https://github.com/airbytehq/airbyte/pull/608 https://github.com/airbytehq/airbyte/pull/609

Is anyone working on the build?

what is the point of the

DataType

enum?

curious how people are feeling about caching the output of getSpec. it feels like having to run a worker for most api actions now because it depends on getting the spec from the integration image slows stuff way down.

is anyone active working on reorging the airbyte-integrations dir structure?

Airbyte protocol discussion: how is a source supposed to behave if a field or stream it was asked to sync is not present? e.g: I request to sync “students” table then when we run the sync, the “students” table is not present. With the capabilities we have today, do we: 1. fail the sync entirely 2. sync what we can without failing 3. other In an ideal world I think we should make this configurable by the user when they setup the connection. In either case I think it should notify them. But for the moment, I am leaning towards 2 since we don’t have a strong notification mechanism right now, and so the user has to manually check if sync fails and try to deduce why that is

The build is down to 8 minutes, which is awesome! @s is this from your change? or should we be worried 😉

The standard source test suite is in master. It’s pretty barebones, but it’s a place to start. You can pattern match off of the gradle.build file for

source-github-singer

for how to configure it. You run it with this command

./gradlew :airbyte-integrations:connectors:source-github-singer:standardSourceTest

. It is not integrated into the build yet, but can be used for developing new integrations. I’ll add a readme tomorrow, but I ran out of time today and want to make sure people stay unblocked.

Hello, what is the task/process that generates

airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py

? For some reason, when i run my

./gradlew build

it ends up with a new ownership by

root:root

user and makes my command fails afterward because of permission access... ⁉️

During the weekly sync on monday, I think it would be worthwhile for us to review the build situation again and see what we can do quickly to improve the situation. We’ve all mentioned this week that as we add integrations, the build times are going to get out of control really fast. @s did a dump of what he learned on the issue. I’ve also added a proposal for splitting up the build. I’d like to come out of this with something we could do in less than a day without requiring us to switch platforms.

Can

AirbyteRecordMessage

contain data for multiple rows of data or is it mandatory to have one row per message?