https://linen.dev logo
c

charles

10/16/2020, 6:02 PM
@Jared Rhizor (Airbyte) - do you know if the python json schema lib you’re using supports references in the schema?
u

user

10/16/2020, 6:02 PM
i’m still parsing through this so forgive me if i’ve got the problem diagnosed wrong.
u

user

10/16/2020, 6:02 PM
i move the log file spec into its own file and used a ref in the airbyte message.
u

user

10/16/2020, 6:03 PM
Copy code
properties:
  type:
    description: "Message type"
    type: string
    enum:
      - RECORD
      - STATE
      - LOG
  record:
    description: "record message: the record"
    "$ref": "#/definitions/AirbyteRecordMessage"
  state:
    description: "schema message: the state. Must be the last message produced. The platform uses this information"
    "$ref": "#/definitions/AirbyteStateMessage"
  log:
    description: "log message: any kind of logging you want the platform to know about."
    "$ref": airbyte_log_message.yaml
u

user

10/16/2020, 6:34 PM
cleaned up the error message a bit:
u

user

10/16/2020, 6:34 PM
Copy code
Traceback (most recent call last):
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 383, in resolve_from_url
    document = self.resolve_remote(url)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 474, in resolve_remote
    result = json.loads(urlopen(uri).read().decode("utf-8"))
  File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 510, in open
    req = Request(fullurl, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 328, in __init__
    self.full_url = url
  File "/usr/local/lib/python3.7/urllib/request.py", line 354, in full_url
    self._parse()
  File "/usr/local/lib/python3.7/urllib/request.py", line 383, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'airbyte_log_message.yaml'
u

user

10/16/2020, 6:34 PM
Copy code
Traceback (most recent call last):
  File "/airbyte/base-python/base.py", line 7, in <module>
    from airbyte_protocol import ConfigContainer
  File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 19, in <module>
    message_classes = _load_classes("types/airbyte_message.yaml")
  File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 13, in _load_classes
    return builder.build_classes(standardize_names=False)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/__init__.py", line 148, in build_classes
    builder.construct(nm, self.schema, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 497, in construct
    ret = self._construct(uri, *args, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 586, in _construct
    self.resolved[uri] = self._build_object(uri, clsdata, parent, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 666, in _build_object
    typ = self.resolve_type(ref, ".".join([nm, prop]))
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 485, in resolve_type
    with self.resolver.resolving(ref) as resolved:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 366, in resolving
    url, resolved = self.resolve(ref)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 375, in resolve
    return url, self._remote_cache(url)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 385, in resolve_from_url
    raise RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: unknown url type: 'airbyte_log_message.yaml'
u

user

10/16/2020, 6:34 PM
is does look like a python issue.
u

user

10/16/2020, 6:35 PM
I think this is actually fixed in the new generator
u

user

10/16/2020, 6:35 PM
i haven’t been following, what’s the new generator?
u

user

10/16/2020, 6:35 PM
can you use the branch
jrhizor/switch-types
and see if it just works for you?
u

user

10/16/2020, 6:35 PM
ok
u

user

10/16/2020, 6:35 PM
oh
u

user

10/16/2020, 6:36 PM
my branch is really a message it’s going to be easier to run a test on your branch than for me to try to rebase now.
u

user

10/16/2020, 6:36 PM
actually
u

user

10/16/2020, 6:36 PM
well i’ll put up the pr for the new version of nested messages / enevelope
u

user

10/16/2020, 6:36 PM
and then we can probably rebase that onto your branch.
u

user

10/16/2020, 6:36 PM
k
u

user

10/16/2020, 6:47 PM
lmk if this looks good with you before we move forward with testing: https://github.com/airbytehq/airbyte/pull/597/files
u

user

10/16/2020, 6:51 PM
I think the true home of the spec should be in the protocol, not config
u

user

10/16/2020, 6:51 PM
it'd probably look this ugly, just in the opposite direction
u

user

10/16/2020, 6:56 PM
okay.
u

user

10/16/2020, 6:56 PM
can we do that later? it’s just a file move.
u

user

10/16/2020, 6:57 PM
yeah
u

user

10/16/2020, 7:08 PM
do you have a good way to test this on your branch?
u

user

10/16/2020, 7:09 PM
i unfortunately do not have a straightforward way of doing this.
u

user

10/16/2020, 7:10 PM
the only thing we want to do is make sure it generates the files properly, right?
u

user

10/16/2020, 7:11 PM
no.
u

user

10/16/2020, 7:11 PM
i think we need to make sure that whatever validation happen in python works.
u

user

10/16/2020, 7:11 PM
if the schema includes refs.
u

user

10/16/2020, 7:11 PM
just in terms of required fields and what not?
u

user

10/16/2020, 7:12 PM
actually maybe i’m jsut wrong
u

user

10/16/2020, 7:13 PM
is this error message during a runtime validation ^?
u

user

10/16/2020, 7:13 PM
that was my interpretation.
u

user

10/16/2020, 7:13 PM
but i could just be wrong.
u

user

10/16/2020, 7:13 PM
with statically generated files (the new way) that code won't run
u

user

10/16/2020, 7:14 PM
okay
u

user

10/16/2020, 7:14 PM
the resolved refs are "compiled"
u

user

10/16/2020, 7:14 PM
got it
u

user

10/16/2020, 7:16 PM
Looks like generation fails with your new message version
u

user

10/16/2020, 7:16 PM
Copy code
NotImplementedError: It is not support to use $id with $ref
u

user

10/16/2020, 7:16 PM
darn
u

user

10/16/2020, 7:18 PM
i’m not seing this when i run
./gradlew build
u

user

10/16/2020, 7:18 PM
I think we can reference the id
u

user

10/16/2020, 7:18 PM
do need to do something else?
u

user

10/16/2020, 7:18 PM
hmm I should add that to build
u

user

10/16/2020, 7:18 PM
it's part of buildImage
u

user

10/16/2020, 7:18 PM
:airbyte-integrations:base-python:generateProtocolClassFiles
u

user

10/16/2020, 7:18 PM
just that should belong to build
u

user

10/16/2020, 7:18 PM
yah
u

user

10/16/2020, 7:19 PM
kk.
u

user

10/16/2020, 7:20 PM
actually
u

user

10/16/2020, 7:20 PM
it might be better to find a solution that doesn’t require that.
u

user

10/16/2020, 7:20 PM
i dunno what’s going to happen in java land if we start doing that.
u

user

10/16/2020, 7:20 PM
i’m going to go eat something and clear my head. will be back at 1 and will beat my head against this.
u

user

10/16/2020, 7:25 PM
wait a second the problem isn't in ConnectorSpecification, it's in connectionStatus for some reason
u

user

10/16/2020, 7:57 PM
I think the opposite way is probably just easier
u

user

10/16/2020, 7:57 PM
have all of these here and have the java generator pull in the file $ref
u

user

10/16/2020, 7:59 PM
not sure i followl.
u

user

10/16/2020, 7:59 PM
want to hop on discord?
u

user

10/16/2020, 8:02 PM
lol. just noticed “is not support”
u

user

10/16/2020, 8:13 PM
i think i can get passed that initial error
u

user

10/16/2020, 8:13 PM
Copy code
> Task :airbyte-integrations:base-python:generateProtocolClassFiles
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 111, in sort_data_models
    unresolved_references, sorted_data_models, require_update_action_models
  File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 96, in sort_data_models
    model.name in model.reference_classes and len(model.reference_classes) == 1
RecursionError: maximum recursion depth exceeded in comparison

During handling of the above exception, another exception occurred:
u

user

10/16/2020, 8:13 PM
but then i get a stack overflow 🤣
u

user

10/16/2020, 8:15 PM
I got a late start to lunch. I’ll hop on in 5
u

user

10/16/2020, 8:15 PM
kk
u

user

10/16/2020, 8:16 PM
i might be misreading this, but i don’t think this function can ever work.