Channels
c
charles
10/16/2020, 6:02 PM
@Jared Rhizor (Airbyte) - do you know if the python json schema lib you’re using supports references in the schema?
u
user
10/16/2020, 6:02 PMi’m still parsing through this so forgive me if i’ve got the problem diagnosed wrong.
u
user
10/16/2020, 6:02 PMi move the log file spec into its own file and used a ref in the airbyte message.
u
user
10/16/2020, 6:03 PM Copy code
properties:
type:
description: "Message type"
type: string
enum:
- RECORD
- STATE
- LOG
record:
description: "record message: the record"
"$ref": "#/definitions/AirbyteRecordMessage"
state:
description: "schema message: the state. Must be the last message produced. The platform uses this information"
"$ref": "#/definitions/AirbyteStateMessage"
log:
description: "log message: any kind of logging you want the platform to know about."
"$ref": airbyte_log_message.yamlu
user
10/16/2020, 6:34 PMcleaned up the error message a bit:
u
user
10/16/2020, 6:34 PM Copy code
Traceback (most recent call last):
File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 383, in resolve_from_url
document = self.resolve_remote(url)
File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 474, in resolve_remote
result = json.loads(urlopen(uri).read().decode("utf-8"))
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 510, in open
req = Request(fullurl, data)
File "/usr/local/lib/python3.7/urllib/request.py", line 328, in __init__
self.full_url = url
File "/usr/local/lib/python3.7/urllib/request.py", line 354, in full_url
self._parse()
File "/usr/local/lib/python3.7/urllib/request.py", line 383, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'airbyte_log_message.yaml'u
user
10/16/2020, 6:34 PM Copy code
Traceback (most recent call last):
File "/airbyte/base-python/base.py", line 7, in <module>
from airbyte_protocol import ConfigContainer
File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 19, in <module>
message_classes = _load_classes("types/airbyte_message.yaml")
File "/airbyte/env/lib/python3.7/site-packages/airbyte_protocol/__init__.py", line 13, in _load_classes
return builder.build_classes(standardize_names=False)
File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/__init__.py", line 148, in build_classes
builder.construct(nm, self.schema, **kw)
File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 497, in construct
ret = self._construct(uri, *args, **kw)
File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 586, in _construct
self.resolved[uri] = self._build_object(uri, clsdata, parent, **kw)
File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 666, in _build_object
typ = self.resolve_type(ref, ".".join([nm, prop]))
File "/airbyte/env/lib/python3.7/site-packages/python_jsonschema_objects/classbuilder.py", line 485, in resolve_type
with self.resolver.resolving(ref) as resolved:
File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 366, in resolving
url, resolved = self.resolve(ref)
File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 375, in resolve
return url, self._remote_cache(url)
File "/airbyte/env/lib/python3.7/site-packages/jsonschema/validators.py", line 385, in resolve_from_url
raise RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: unknown url type: 'airbyte_log_message.yaml'u
user
10/16/2020, 6:34 PMis does look like a python issue.
u
user
10/16/2020, 6:35 PMI think this is actually fixed in the new generator
u
user
10/16/2020, 6:35 PMi haven’t been following, what’s the new generator?
u
user
10/16/2020, 6:35 PMcan you use the branch
jrhizor/switch-typesu
user
10/16/2020, 6:35 PMok
u
user
10/16/2020, 6:35 PMoh
u
user
10/16/2020, 6:36 PMmy branch is really a message it’s going to be easier to run a test on your branch than for me to try to rebase now.
u
user
10/16/2020, 6:36 PMactually
u
user
10/16/2020, 6:36 PMwell i’ll put up the pr for the new version of nested messages / enevelope
u
user
10/16/2020, 6:36 PMand then we can probably rebase that onto your branch.
u
user
10/16/2020, 6:36 PMk
u
user
10/16/2020, 6:47 PMlmk if this looks good with you before we move forward with testing: https://github.com/airbytehq/airbyte/pull/597/files
u
user
10/16/2020, 6:51 PMI think the true home of the spec should be in the protocol, not config
u
user
10/16/2020, 6:51 PMit'd probably look this ugly, just in the opposite direction
u
user
10/16/2020, 6:56 PMokay.
u
user
10/16/2020, 6:56 PMcan we do that later? it’s just a file move.
u
user
10/16/2020, 6:57 PMyeah
u
user
10/16/2020, 7:08 PMdo you have a good way to test this on your branch?
u
user
10/16/2020, 7:09 PMi unfortunately do not have a straightforward way of doing this.
u
user
10/16/2020, 7:10 PMthe only thing we want to do is make sure it generates the files properly, right?
u
user
10/16/2020, 7:11 PMno.
u
user
10/16/2020, 7:11 PMi think we need to make sure that whatever validation happen in python works.
u
user
10/16/2020, 7:11 PMif the schema includes refs.
u
user
10/16/2020, 7:11 PMjust in terms of required fields and what not?
u
user
10/16/2020, 7:12 PMactually maybe i’m jsut wrong
u
user
10/16/2020, 7:13 PMis this error message during a runtime validation ^?
u
user
10/16/2020, 7:13 PMthat was my interpretation.
u
user
10/16/2020, 7:13 PMbut i could just be wrong.
u
user
10/16/2020, 7:13 PMwith statically generated files (the new way) that code won't run
u
user
10/16/2020, 7:14 PMokay
u
user
10/16/2020, 7:14 PMthe resolved refs are "compiled"
u
user
10/16/2020, 7:14 PMgot it
u
user
10/16/2020, 7:16 PMLooks like generation fails with your new message version
u
user
10/16/2020, 7:16 PM Copy code
NotImplementedError: It is not support to use $id with $refu
user
10/16/2020, 7:16 PMdarn
u
user
10/16/2020, 7:18 PMi’m not seing this when i run
./gradlew buildu
user
10/16/2020, 7:18 PMI think we can reference the id
u
user
10/16/2020, 7:18 PMdo need to do something else?
u
user
10/16/2020, 7:18 PMhmm I should add that to build
u
user
10/16/2020, 7:18 PMit's part of buildImage
u
user
10/16/2020, 7:18 PM:airbyte-integrations:base-python:generateProtocolClassFilesu
user
10/16/2020, 7:18 PMjust that should belong to build
u
user
10/16/2020, 7:18 PMyah
u
user
10/16/2020, 7:19 PMkk.
u
user
10/16/2020, 7:20 PMactually
u
user
10/16/2020, 7:20 PMit might be better to find a solution that doesn’t require that.
u
user
10/16/2020, 7:20 PMi dunno what’s going to happen in java land if we start doing that.
u
user
10/16/2020, 7:20 PMi’m going to go eat something and clear my head. will be back at 1 and will beat my head against this.
u
user
10/16/2020, 7:25 PMu
user
10/16/2020, 7:57 PMI think the opposite way is probably just easier
u
user
10/16/2020, 7:57 PMhave all of these here and have the java generator pull in the file $ref
u
user
10/16/2020, 7:59 PMnot sure i followl.
u
user
10/16/2020, 7:59 PMwant to hop on discord?
u
user
10/16/2020, 8:02 PMlol. just noticed “is not support”
u
user
10/16/2020, 8:13 PMi think i can get passed that initial error
u
user
10/16/2020, 8:13 PM Copy code
> Task :airbyte-integrations:base-python:generateProtocolClassFiles
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 111, in sort_data_models
unresolved_references, sorted_data_models, require_update_action_models
File "/usr/local/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 96, in sort_data_models
model.name in model.reference_classes and len(model.reference_classes) == 1
RecursionError: maximum recursion depth exceeded in comparison
During handling of the above exception, another exception occurred:u
user
10/16/2020, 8:13 PMbut then i get a stack overflow 🤣
u
user
10/16/2020, 8:15 PMI got a late start to lunch. I’ll hop on in 5
u
user
10/16/2020, 8:15 PMkk
u
user
10/16/2020, 8:16 PMi might be misreading this, but i don’t think this function can ever work.