https://linen.dev logo
c

charles

10/26/2020, 3:31 PM
does anyone else have issues compiling the project locally. i’m getting c compiler issues when it attempts to run this task
Task :airbyte-integrations:connectors:source-postgres-singer:installReqs
u

user

10/26/2020, 3:31 PM
which version of python are you running
u

user

10/26/2020, 3:32 PM
Copy code
➜  airbyte git:(master) python --version
Python 2.7.16
➜  airbyte git:(master) python3 --version
Python 3.7.8
➜  airbyte git:(master)
u

user

10/26/2020, 3:33 PM
is that wrong?
u

user

10/26/2020, 3:33 PM
am i supposed to symlink python to point to python3 or something?
u

user

10/26/2020, 3:35 PM
deleting this
airbyte-integrations/connectors/source-postgres-singer/.venv
seems to have fixed the issue.
u

user

10/26/2020, 3:38 PM
hmm maybe we have to set the executable to the venvs to python3 binary instead of python
u

user

10/26/2020, 3:39 PM
it seems possible that i wasn’t careful and created the venv in python 2.
u

user

10/26/2020, 3:39 PM
maybe copy pasta-ed off of the tap-postgres website or something
u

user

10/26/2020, 3:39 PM
the venv should be greated automatically by ./gradlew build
u

user

10/26/2020, 3:39 PM
oh
u

user

10/26/2020, 3:39 PM
i likely had an old one in there though.
u

user

10/26/2020, 3:40 PM
that i had created manually.
u

user

10/26/2020, 3:40 PM
oh ok
u

user

10/26/2020, 3:47 PM
I noticed some issues with that one when building psycopg2 binaries too (it might require us to install postgres and python3 devel gcc etc to build properly?)
u

user

10/26/2020, 3:47 PM
But havent been able to properly identify yet..
u

user

10/26/2020, 3:55 PM
do you need to run the build multiple times to get the build to pass all of the python checks?
u

user

10/26/2020, 3:55 PM
mypyCheck
has seemed very flakey.
u

user

10/26/2020, 4:00 PM
do you have an example of a failure
u

user

10/26/2020, 4:00 PM
haven't seen any mypy errors yet
u

user

10/26/2020, 4:01 PM
it shouldn't require multiple builds, if it does things are happening in the wrong order
u

user

10/26/2020, 4:01 PM
okay. i’m going to keep my fingers crossed that this is an issue with me switching back and forth between kinda old branches.
u

user

10/26/2020, 4:02 PM
i’ll report back if i see this consistently on new branches.
u

user

10/26/2020, 4:06 PM
nope. looks like still failing
u

user

10/26/2020, 4:06 PM
Copy code
> Task :airbyte-integrations:bases:base-singer:mypyCheck FAILED
[python] .venv/bin/python -m mypy -m base_singer --config-file /Users/charles/code/airbyte/tools/python/.mypy.ini
         airbyte_protocol/__init__.py:27: note: In module imported here,
         base_singer/singer_helpers.py:33: note: ... from here,
         base_singer/__init__.py:25: note: ... from here:
         airbyte_protocol/models/__init__.py:2:1: error: Skipping analyzing
         'airbyte_protocol.models.airbyte_message': found module but no type hints or
         library stubs  [import]
             from .airbyte_message import *
             ^
         airbyte_protocol/models/__init__.py:2:1: note: See <https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports>
         Found 1 error in 1 file (checked 1 source file)
u

user

10/26/2020, 4:07 PM
probably related. these two files are almost always dirty.
u

user

10/26/2020, 4:07 PM
what's the diff?
u

user

10/26/2020, 4:07 PM
there used to be license problems before
u

user

10/26/2020, 4:07 PM
but I thought those were fixed
u

user

10/26/2020, 4:09 PM
we probably need an additional gradle dependency to make sure generating the protocol files happens before spotlessapply
u

user

10/26/2020, 4:09 PM
and that those happen before later formatting checks
u

user

10/26/2020, 4:14 PM
it seems like specifically in the publish script it gets whack.y
u

user

10/26/2020, 4:14 PM
./tools/integrations/manage.sh publish airbyte-integrations/connectors/source-exchangeratesapi-singer
u

user

10/26/2020, 4:14 PM
after i run this script the diff happens.
u

user

10/26/2020, 4:14 PM
so i can’t run it against for another integration unless i first remove the idff.
u

user

10/26/2020, 4:16 PM
Copy code
→ ./gradlew integrationTest --dry-run | grep generate
:airbyte-integrations:bases:base-python:generateProtocolClassFiles SKIPPED
:airbyte-config:models:generateJsonSchema2Pojo SKIPPED
:airbyte-protocol:models:generateJsonSchema2Pojo SKIPPED

# jrhizor in ~/code/airbyte on git:master ✖︎ [9:15:52]
→ ./gradlew integrationTest --dry-run | grep spotless
u

user

10/26/2020, 4:16 PM
looks like the call to integrationTest depends on
python:generateProtocolClassFiles
but not on
spotlessApply
, so it's getting strange results.
u

user

10/26/2020, 4:16 PM
I guess
generateProtocolClassFiles
is probably in the wrong spot then?
u

user

10/26/2020, 6:22 PM
adding this to my zshrc fixed the OP
Copy code
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
u

user

10/26/2020, 6:23 PM
the issue was
Copy code
ld: library not found for -lssl
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
which just meant the psycopg2 install couldn’t find ssl
u

user

10/26/2020, 6:23 PM
on mac we probably want to add this option by default (since everyone is probably using openssl). Linux I’m not sure
u

user

10/26/2020, 6:23 PM
wasn't necessary on my computer :/
u

user

10/26/2020, 6:24 PM
We should at a minimum add these to a troubleshooting doc
u

user

10/26/2020, 6:30 PM
which OS are you running? and what does
which openssl
return?
u

user

10/26/2020, 6:31 PM
Copy code
→ which openssl
/usr/bin/openssl
u

user

10/26/2020, 6:31 PM
Mac 10.15.4