https://linen.dev logo
Join Slack
Powered by
<@U01ADKHBJKC> - do you know if the python json sc...
# contributing-to-airbyte
c
@Jared Rhizor (Airbyte) - do you know if the python json schema lib you’re using supports references in the schema?
u
i’m still parsing through this so forgive me if i’ve got the problem diagnosed wrong.
u
i move the log file spec into its own file and used a ref in the airbyte message.
u
Copy code
properties:
  type:
    description: "Message type"
    type: string
    enum:
      - RECORD
      - STATE
      - LOG
  record:
    description: "record message: the record"
    "$ref": "#/definitions/AirbyteRecordMessage"
  state:
    description: "schema message: the state. Must be the last message produced. The platform uses this information"
    "$ref": "#/definitions/AirbyteStateMessage"
  log:
    description: "log message: any kind of logging you want the platform to know about."
    "$ref": airbyte_log_message.yaml
u
cleaned up the error message a bit:
u
Copy code
Traceback (most recent call last):
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 383, in resolve_from_url
    document = self.resolve_remote(url)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 474, in resolve_remote
    result = json.loads(urlopen(uri).read().decode("utf-8"))
  File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 510, in open
    req = Request(fullurl, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 328, in __init__
    self.full_url = url
  File "/usr/local/lib/python3.7/urllib/request.py", line 354, in full_url
    self._parse()
  File "/usr/local/lib/python3.7/urllib/request.py", line 383, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'airbyte_log_message.yaml'
u
Copy code
Traceback (most recent call last):
  File "/airbyte/base-python/base.py", line 7, in <module>
    from airbyte_protocol import ConfigContainer
  File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 19, in <module>
    message_classes = _load_classes("types/airbyte_message.yaml")
  File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 13, in _load_classes
    return builder.build_classes(standardize_names=False)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/__init__.py", line 148, in build_classes
    builder.construct(nm, self.schema, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 497, in construct
    ret = self._construct(uri, *args, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 586, in _construct
    self.resolved[uri] = self._build_object(uri, clsdata, parent, **kw)
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 666, in _build_object
    typ = self.resolve_type(ref, ".".join([nm, prop]))
  File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 485, in resolve_type
    with self.resolver.resolving(ref) as resolved:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 366, in resolving
    url, resolved = self.resolve(ref)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 375, in resolve
    return url, self._remote_cache(url)
  File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 385, in resolve_from_url
    raise RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: unknown url type: 'airbyte_log_message.yaml'
u
is does look like a python issue.
u
I think this is actually fixed in the new generator
u
i haven’t been following, what’s the new generator?
u
can you use the branch 
jrhizor/switch-types
and see if it just works for you?
u
ok
u
oh
u
my branch is really a message it’s going to be easier to run a test on your branch than for me to try to rebase now.
u
actually
u
well i’ll put up the pr for the new version of nested messages / enevelope
u
and then we can probably rebase that onto your branch.
u
k
u
lmk if this looks good with you before we move forward with testing: https://github.com/airbytehq/airbyte/pull/597/files
u
I think the true home of the spec should be in the protocol, not config
u
it'd probably look this ugly, just in the opposite direction
u
okay.
u
can we do that later? it’s just a file move.
u
yeah
u
do you have a good way to test this on your branch?
u
i unfortunately do not have a straightforward way of doing this.
u
the only thing we want to do is make sure it generates the files properly, right?
u
no.
u
i think we need to make sure that whatever validation happen in python works.
u
if the schema includes refs.
u
just in terms of required fields and what not?
u
actually maybe i’m jsut wrong
u
u
is this error message during a runtime validation ^?
u
that was my interpretation.
u
but i could just be wrong.
u
with statically generated files (the new way) that code won't run
u
okay
u
the resolved refs are "compiled"
u
got it
u
Looks like generation fails with your new message version
u
Copy code
NotImplementedError: It is not support to use $id with $ref
u
darn
u
i’m not seing this when i run 
./gradlew build
u
I think we can reference the id
u
do need to do something else?
u
hmm I should add that to build
u
it's part of buildImage
u
:airbyte-integrations:base-python:generateProtocolClassFiles
u
just that should belong to build
u
yah
u
kk.
u
actually
u
it might be better to find a solution that doesn’t require that.
u
i dunno what’s going to happen in java land if we start doing that.
u
i’m going to go eat something and clear my head. will be back at 1 and will beat my head against this.
u
wait a second the problem isn't in ConnectorSpecification, it's in connectionStatus for some reason
u
I think the opposite way is probably just easier
u
have all of these here and have the java generator pull in the file $ref
u
not sure i followl.
u
want to hop on discord?
u
lol. just noticed “is not support”
u
i think i can get passed that initial error
u
Copy code
> Task :airbyte-integrations:base-python:generateProtocolClassFiles
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 111, in sort_data_models
    unresolved_references, sorted_data_models, require_update_action_models
  File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 96, in sort_data_models
    model.name in model.reference_classes and len(model.reference_classes) == 1
RecursionError: maximum recursion depth exceeded in comparison

During handling of the above exception, another exception occurred:
u
but then i get a stack overflow 🤣
u
I got a late start to lunch. I’ll hop on in 5
u
kk
u
i might be misreading this, but i don’t think this function can ever work.
u
Open in Slack
PreviousNext
Powered by