fast-leather-13054
04/29/2021, 4:51 PMDataJobSnapshotClass
DataJobInfoClass
DataJobInputOutputClass
Could you please explain how you imported links to airflow or any custom properties in your demo environment ?
Will these missing classes be available soon ?mammoth-bear-12532
mammoth-bear-12532
millions-engineer-56536
04/29/2021, 4:55 PMmammoth-bear-12532
millions-engineer-56536
04/29/2021, 4:56 PMmillions-engineer-56536
04/29/2021, 4:56 PMmillions-engineer-56536
04/29/2021, 4:57 PMDataJobSnapshotClass
DataJobInfoClass
DataJobInputOutputClass
do not seem to have all the properties avaialblemillions-engineer-56536
04/29/2021, 4:57 PMmammoth-bear-12532
record DataJobInfo includes CustomProperties, ExternalReference {
mammoth-bear-12532
fast-leather-13054
04/29/2021, 5:02 PMname: str,
type: Union[str, "AzkabanJobTypeClass"],
description: Union[None, str]=None,
in DataJobInfoClassmammoth-bear-12532
gray-shoe-75895
04/29/2021, 5:03 PMfast-leather-13054
04/29/2021, 5:05 PMmammoth-bear-12532
fast-leather-13054
04/30/2021, 8:06 AMfast-leather-13054
04/30/2021, 8:07 AMfast-leather-13054
04/30/2021, 8:08 AMfast-leather-13054
04/30/2021, 2:46 PMmammoth-bear-12532
"com.linkedin.pegasus2avro.metadata.snapshot.DataFlowSnapshot": {
"urn": "urn:li:dataFlow:(airflow,dag_abc,PROD)",
gray-shoe-75895
04/30/2021, 4:48 PMfast-leather-13054
04/30/2021, 4:57 PMmammoth-bear-12532
git log | head -n 1
give?gray-shoe-75895
04/30/2021, 5:01 PM(cd .. && ./gradlew :metadata-events:mxe-schemas:build) && ./scripts/codegen.sh
mammoth-bear-12532
class DataJobInfoClass(DictWrapper):
"""Information about a Data processing job"""
RECORD_SCHEMA = get_schema_type("com.linkedin.pegasus2avro.datajob.DataJobInfo")
def __init__(self,
customProperties: Dict[str, str],
name: str,
type: Union[str, "AzkabanJobTypeClass"],
externalUrl: Union[None, str]=None,
description: Union[None, str]=None,
):
millions-engineer-56536
04/30/2021, 5:32 PMclass DataFlowInfoClass(DictWrapper):
"""Information about a Data processing flow"""
RECORD_SCHEMA = get_schema_type("com.linkedin.pegasus2avro.datajob.DataFlowInfo")
def __init__(self,
name: str,
customProperties: Optional[Dict[str, str]]=None,
externalUrl: Union[None, str]=None,
description: Union[None, str]=None,
project: Union[None, str]=None,
):
and
class DataJobInfoClass(DictWrapper):
"""Information about a Data processing job"""
RECORD_SCHEMA = get_schema_type("com.linkedin.pegasus2avro.datajob.DataJobInfo")
def __init__(self,
name: str,
type: Union[str, "AzkabanJobTypeClass"],
customProperties: Optional[Dict[str, str]]=None,
externalUrl: Union[None, str]=None,
description: Union[None, str]=None,
):
millions-engineer-56536
04/30/2021, 5:33 PMmillions-engineer-56536
04/30/2021, 5:35 PMtype: Union[str, "AzkabanJobTypeClass"],
? we have many more job typesfast-leather-13054
04/30/2021, 5:38 PMgray-shoe-75895
04/30/2021, 5:40 PMmetadata-models/src/main/pegasus/com/linkedin/datajob/DataJobInfo.pdl
- any specific job types you’d want to be included?gray-shoe-75895
04/30/2021, 5:42 PMmillions-engineer-56536
04/30/2021, 5:50 PMmillions-engineer-56536
04/30/2021, 5:51 PMmammoth-bear-12532
mammoth-bear-12532
mammoth-bear-12532
mammoth-bear-12532
gray-shoe-75895
04/30/2021, 5:57 PMgray-shoe-75895
04/30/2021, 5:58 PMfast-leather-13054
04/30/2021, 6:19 PMmillions-engineer-56536
04/30/2021, 6:25 PMmillions-engineer-56536
04/30/2021, 6:26 PMgray-shoe-75895
04/30/2021, 6:49 PMfast-leather-13054
04/30/2021, 6:50 PMgray-shoe-75895
04/30/2021, 6:51 PM