Is there anything to be done quickly about mypy issues i m o Airbyte #contributing-to-airbyte

Is there anything to be done quickly about mypy is...

charles

10/26/2020, 10:25 PM

Is there anything to be done quickly about mypy issues? i’m on master and running into this. It is preventing me from releasing.

Copy code

> Task :airbyte-integrations:connector-templates:singer-source:mypyCheck FAILED
[python] .venv/bin/python -m mypy -m template_singer_source --config-file /Users/charles/code/airbyte/tools/python/.mypy.ini
         airbyte_protocol/__init__.py:27: note: In module imported here,
         template_singer_source/source.py:27: note: ... from here,
         template_singer_source/__init__.py:25: note: ... from here:
         airbyte_protocol/models/__init__.py:2:1: error: Skipping analyzing
         'airbyte_protocol.models.airbyte_message': found module but no type hints or
         library stubs  [import]
             from .airbyte_message import *
             ^
         airbyte_protocol/models/__init__.py:2:1: note: See <https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports>
         Found 1 error in 1 file (checked 1 source file)

user

10/26/2020, 10:27 PM

it seems like an early step creates this diff

user

10/26/2020, 10:27 PM

Copy code

➜  airbyte git:(cgardens/release_v0_2_1) gst
On branch cgardens/release_v0_2_1
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   airbyte-integrations/bases/base-python/airbyte_protocol/models/__init__.py
	deleted:    airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py

user

10/26/2020, 10:28 PM

which causes the linter to fail.

user

10/26/2020, 10:29 PM

this is due to an interaction between generation, spotless, and the linter

user

10/26/2020, 10:29 PM

at some point you ran a step that depends on generation but not spotless

user

10/26/2020, 10:31 PM

i’m running the release_version.s script.

user

10/26/2020, 10:31 PM

which just runs

./gradlew clean build

user

10/26/2020, 10:31 PM

i’m not doing anything special.

user

10/26/2020, 10:35 PM

I think spotless should be part of build then

user

10/26/2020, 10:35 PM

it is?

user

10/26/2020, 10:36 PM

must be in the wrong order

user

10/26/2020, 10:38 PM

spotless runs as part of

build

user

10/26/2020, 10:38 PM

hmmm. i think the issue now is that this task isn’t ever completing:

Copy code

<============-> 94% EXECUTING [39s]
> :airbyte-integrations:bases:base-python:generateProtocolClassFiles

user

10/26/2020, 10:38 PM

is that something we’ve run into before?

user

10/26/2020, 10:41 PM

no I don't think so

user

10/26/2020, 10:41 PM

what do you get if you run

user

10/26/2020, 10:41 PM

Copy code

→ ./gradlew clean build --dry-run | grep "spotless\|generate"
:spotlessGroovyGradle SKIPPED
:spotlessGroovyGradleApply SKIPPED
:spotlessJava SKIPPED
:spotlessJavaApply SKIPPED
:airbyte-integrations:bases:base-python:generateProtocolClassFiles SKIPPED
:spotlessPython SKIPPED
:spotlessPythonApply SKIPPED
:spotlessSql SKIPPED
:spotlessSqlApply SKIPPED
:spotlessStyling SKIPPED
:spotlessStylingApply SKIPPED
:spotlessApply SKIPPED
:spotlessGroovyGradleCheck SKIPPED
:spotlessJavaCheck SKIPPED
:spotlessPythonCheck SKIPPED
:spotlessSqlCheck SKIPPED
:spotlessStylingCheck SKIPPED
:spotlessCheck SKIPPED
:airbyte-config:models:generateJsonSchema2Pojo SKIPPED
:airbyte-protocol:models:generateJsonSchema2Pojo SKIPPED
:airbyte-api:generateApiClient SKIPPED
:airbyte-api:generateApiDocs SKIPPED
:airbyte-api:generateApiServer SKIPPED

user

10/26/2020, 10:42 PM

does it show

:airbyte-integrations:bases:base-python:generateProtocolClassFiles

above

spotlessPython

user

10/26/2020, 10:42 PM

Copy code

➜  airbyte git:(cgardens/release_v0_2_1) ✗ ./gradlew clean build --dry-run | grep "spotless\|generate"
:spotlessGroovyGradle SKIPPED
:spotlessGroovyGradleApply SKIPPED
:spotlessJava SKIPPED
:spotlessJavaApply SKIPPED
:airbyte-integrations:bases:base-python:generateProtocolClassFiles SKIPPED
:spotlessPython SKIPPED
:spotlessPythonApply SKIPPED
:spotlessSql SKIPPED
:spotlessSqlApply SKIPPED
:spotlessStyling SKIPPED
:spotlessStylingApply SKIPPED
:spotlessApply SKIPPED
:spotlessGroovyGradleCheck SKIPPED
:spotlessJavaCheck SKIPPED
:spotlessPythonCheck SKIPPED
:spotlessSqlCheck SKIPPED
:spotlessStylingCheck SKIPPED
:spotlessCheck SKIPPED
:airbyte-config:models:generateJsonSchema2Pojo SKIPPED
:airbyte-protocol:models:generateJsonSchema2Pojo SKIPPED
:airbyte-api:generateApiClient SKIPPED
:airbyte-api:generateApiDocs SKIPPED
:airbyte-api:generateApiServer SKIPPED

user

10/26/2020, 10:42 PM

hmm

user

10/26/2020, 10:42 PM

generateProtocolClassFiles

should be fast, right?

user

10/26/2020, 10:43 PM

it's not as fast as you'd expect but it should be like 10-15 seconds I think

user

10/26/2020, 10:43 PM

hmmm. i’m giving up after a minute.

user

10/26/2020, 10:43 PM

so that seems like i’m giving it enough time.

user

10/26/2020, 10:44 PM

I think

from .airbyte_message import *

may be a bad thing too (being to vague for mypy) and it's throwing warnings on how the code should be written

user

10/26/2020, 10:44 PM

are you all able to clean build locally on master?

user

10/26/2020, 10:44 PM

can you run it with

--scan

user

10/26/2020, 10:44 PM

then we could see a timeline of when things are breaking on your side if it ends in the bad state

user

10/26/2020, 10:45 PM

does scan print anything incrementally?

user

10/26/2020, 10:45 PM

Copy code

➜  airbyte git:(cgardens/release_v0_2_1) ✗ ./gradlew :airbyte-integrations:bases:base-python:generateProtocolClassFiles --scan

> Task :airbyte-integrations:bases:base-python:code-generator:buildImage
Building image using context '/Users/charles/code/airbyte/airbyte-integrations/bases/base-python/code-generator'.
Using images 'airbyte/code-generator:dev'.
Step 1/5 : FROM python:3.7-slim
 ---> 217e85391449
Step 2/5 : RUN pip install datamodel-code-generator==0.5.39
 ---> Using cache
 ---> 2e547409dc5b
Step 3/5 : ENTRYPOINT ["datamodel-codegen"]
 ---> Using cache
 ---> 57c73643c2c6
Step 4/5 : LABEL io.airbyte.version=dev
 ---> Using cache
 ---> 96d999a96a4a
Step 5/5 : LABEL io.airbyte.name=airbyte/code-generator
 ---> Using cache
 ---> 18ec9e472078
Successfully built 18ec9e472078
Successfully tagged airbyte/code-generator:dev
Created image with ID '18ec9e472078'.
<============-> 94% EXECUTING [53s]
> :airbyte-integrations:bases:base-python:generateProtocolClassFiles

user

10/26/2020, 10:46 PM

no it just gives a link at the end

user

10/26/2020, 10:46 PM

so it doesnt help the never completing case

user

10/26/2020, 10:47 PM

maybe show more logs to debug that?

user

10/26/2020, 10:47 PM

if it's workiing I'd prefer to see scan

user

10/26/2020, 10:47 PM

it never completes.

user

10/26/2020, 10:47 PM

scan is a no go.

user

10/26/2020, 10:48 PM

so this just started?

user

10/26/2020, 10:48 PM

weird

user

10/26/2020, 10:48 PM

you don’t have this problem doing a clean build locally?

user

10/26/2020, 10:48 PM

./gradlew clean build

user

10/26/2020, 10:48 PM

nope

user

10/26/2020, 10:48 PM

gets past generation fine

user

10/26/2020, 10:49 PM

Copy code

2020-10-26T15:48:23.083-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:23.083-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:23.084-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:33.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:33.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:33.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:33.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:33.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:33.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:43.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:43.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:43.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:43.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:43.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:43.081-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:53.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:53.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:53.079-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:48:53.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:48:53.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:48:53.080-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:49:03.083-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:49:03.083-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:49:03.083-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2020-10-26T15:49:03.084-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2020-10-26T15:49:03.084-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2020-10-26T15:49:03.084-0700 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.

user

10/26/2020, 10:49 PM

🤯

user

10/26/2020, 10:50 PM

is something gradle-related still running in the background?

user

10/26/2020, 10:50 PM

not sure how that happened but you can probably kill it and re-run

user

10/26/2020, 10:51 PM

restarted docker and blew away all gradle things and it doesn’t hang anymore.

user

10/26/2020, 11:18 PM

thanks for your help!

user

10/26/2020, 11:19 PM

if you get stuck in a state where some command is not adding the license correctly to the generated files, please try to recreate with a scan though

4 Views

Open in Slack

Previous Next