does anyone else have issues compiling the project locally. i’m getting c compiler issues when it attempts to run this task

Task :airbyte-integrations:connectors:source-postgres-singer:installReqs

hey all, if there any issues that are not already in our roadmap for this week that you think we should consider, make sure to have them ready to go for the sync at 11:30am!

2020-10-26 201430 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):108 - * - CRITICAL Salesforce has reported 13060/15000 (87.07%) total REST quota used across all Salesforce Applications. Terminating replication to not continue past configured percentage of 80% total quota.

10106

so another reason we may not want to run all integration integration tests with every build is that we’re going to run into quota issues. master build is failing currently because we’ve used up the quota from having built just a few times this morning.

Is there anything to be done quickly about mypy issues? i’m on master and running into this. It is preventing me from releasing.

> Task :airbyte-integrations:connector-templates:singer-source:mypyCheck FAILED
[python] .venv/bin/python -m mypy -m template_singer_source --config-file /Users/charles/code/airbyte/tools/python/.mypy.ini
         airbyte_protocol/__init__.py:27: note: In module imported here,
         template_singer_source/source.py:27: note: ... from here,
         template_singer_source/__init__.py:25: note: ... from here:
         airbyte_protocol/models/__init__.py:2:1: error: Skipping analyzing
         'airbyte_protocol.models.airbyte_message': found module but no type hints or
         library stubs  [import]
             from .airbyte_message import *
             ^
         airbyte_protocol/models/__init__.py:2:1: note: See <https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports>
         Found 1 error in 1 file (checked 1 source file)

Should I be setting a style guide for the python version of the airbyte project?

should

./gradlew :airbyte-integrations:connectors:source-github-singer:build

create a

.venv

dir in

airbyte-integrations/connectors/source-github-singer

or is there some other task that i should be running to do that?

has anyone figured out writing tests and handling test dependencies in python already in a way that is graceful?

is mypy supposed to catch import statements that are wrong, right (e.g. trying to import a class that does not exist in the specified module)

> Task :airbyte-integrations:connectors:source-postgres-singer:installReqs
[python] .venv/bin/python -m pip install -r requirements.txt
         Obtaining file:///Users/charles/code/airbyte/airbyte-integrations/bases/airbyte-protocol (from -r requirements.txt (line 1))
         Obtaining file:///Users/charles/code/airbyte/airbyte-integrations/bases/base-python (from -r requirements.txt (line 2))
         Obtaining file:///Users/charles/code/airbyte/airbyte-integrations/bases/base-singer (from -r requirements.txt (line 3))
         Obtaining file:///Users/charles/code/airbyte/airbyte-integrations/connectors/source-postgres-singer (from -r requirements.txt (line 4))
         Requirement already satisfied: PyYAML==5.3.1 in ./.venv/lib/python3.7/site-packages (from airbyte-protocol==0.0.0->-r requirements.txt (line 1)) (5.3.1)
         Requirement already satisfied: pydantic==1.6.1 in ./.venv/lib/python3.7/site-packages (from airbyte-protocol==0.0.0->-r requirements.txt (line 1)) (1.6.1)

Expiring Daemon because JVM heap space is exhausted
Daemon is stopping immediately JVM garbage collector thrashing and after running out of JVM memory

anyone intermittently getting

JVM heap space is exhausted

when building the project?

can i get a review on this pig of PR, please? it touches alot of files so i don’t want it to sit too long.

All right. 3 hours is the time to beat for building an airbyte integration from a singer one. here’s hubspot. 😉 anyone want to race?

Hi, there, when i run the gradle task

:airbyte-integrations:connectors:source-postgres-singer:installReqs

, I will always get the error (#shown as the followed log file), that will led to the fail of the gradle build process. Is that cause of the version of Python or something? Thanks.

After writing a connector, i created a file

airbyte-config/init/src/main/resources/config/STANDARD_SOURCE/778daa7c070c.json

with this content:

{
  "sourceId": "778daa7c070c",
  "name": "File",
  "dockerRepository": "airbyte/source-file",
  "dockerImageTag": "0.1.0",
  "documentationUrl": "<https://hub.docker.com/r/airbyte/source-file>"
}

However when i start the

docker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml up

command and open the browser, i don't see my source-file in there, am i missing something?

I am not going to get to SqlServer or migrating Postgres to use the Jdbc base today. Falling behind on getting MySql out and still need to start on google ads.

Hi, I had successfully built the project. And How I want to contribute to the project! Is there something I can do that is maybe not so complicated? And maybe as time I know this project better I can contribute more!

do we have a good PR example with a python integration written from scratch?

did we ever have a README.md to create java destinations?

What’s our preferred pattern on

.dockerignore

files? I thought we had a pattern where we started with

*

and then only surgically re-added the things we explicitly wanted to make available in the dockerfiles. Got some feedback asking to go the opposite way (e.g. just ignore

build

). I don’t have a preference on this once, but I’d like us to coalesce around a standard that I will follow.

@s should we kill the postgres singer source integration now that we have ours?

would someone be able to take a look at this PR this morning? it’s for handling configuration for normalization.

anyone able to take a look at the FB source PR? https://github.com/airbytehq/airbyte/pull/808

we are inconsistent in our

setup.py

in how we refer to internal deps:

install_requires=["airbyte-protocol"],

install_requires=["airbyte_protocol"],

Both work. I think the

-

version is more idiomatic in the python world. Please correct me if I’m wrong. I care most that we pick one way of doing it and stick with it.

Hey guys, Red highlighted links are not working for me can you cross check

@s could you please help looking at this PR? https://github.com/airbytehq/airbyte/pull/802 I've merged and pushed back the changes to put Charles' branches and mine together for the normalization process... However I've ran into a last stop block where i have to run with a bunch of excludes:

gradlew build -x spotlessSql -x spotlessSqlApply -x spotlessSqlCheck

to make it build successfully... (Spotless is messing up the DBT SQL code turning into syntax errors otherwise...) Thanks!

How long does spotless takes for you?

Is it correct, that all our integrations are currently poll based? So we execute sync every N minutes? Just a matter of interest - are we going to support webhook based integrations in future? Like psql wal logs or dynamodb streams?

Not sure if it's on the Front-end or backend but on master branch if i try to use the Source connector for Files, i get exceptions thrown because the value is not in the ENUM but it is selected from the UI (one of the value of the enum)...

I think my PR https://github.com/airbytehq/airbyte/pull/802 is mergeable, except for few questions i left if you want to review a last time (@Jared Rhizor (Airbyte) @Michel) thanks!

New issues will now be auto-assigned to the Airbyte project in the “Backlog” column. This isn’t part of the issue template (unfortunately not supported by github), so it takes a few seconds for it to be assigned after the issue has been created.

What's the point of

documentationUrl

in source/destination destinations?