hallowed-truck-92074
04/12/2023, 10:59 AMbusy-ghost-93490
04/12/2023, 12:07 PMfull-toddler-1726
04/12/2023, 2:42 PMfull-toddler-1726
04/12/2023, 2:43 PMbland-orange-13353
04/12/2023, 4:52 PMbland-orange-13353
04/12/2023, 5:31 PMsteep-alligator-93593
04/12/2023, 6:55 PMbrave-judge-32701
04/13/2023, 2:45 AMbin/spark-shell --master local[*] --deploy-mode client
I can’t see any log about DatahubSparkListener
and I got WARN message about :
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
my log4j.properties config is :
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.logger.org.apache.spark.repl.Main=WARN
log4j.logger.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver=WARN
log4j.logger.datahub.spark=DEBUG
log4j.logger.datahub.client.rest=DEBUG
bland-orange-13353
04/13/2023, 8:52 AMhelpful-van-67650
04/13/2023, 9:08 AMdatahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml
, It list the following error,May I ask what I should do next?dry-thailand-78553
04/13/2023, 4:58 PMdry-thailand-78553
04/13/2023, 4:58 PMfast-midnight-10167
04/13/2023, 7:44 PMproud-dusk-671
04/14/2023, 6:47 AMdocker/datahub-frontend/env/docker.env
. Secondly, I do not want to kill my docker container in fear of data loss. Can you confirm killing the frontend docker container not result in any data loss. Finally, is there any better way to test Google Authentication for the app?future-holiday-32084
04/14/2023, 5:06 PMspark.sql("select * from <database>.<table_source>").write.mode("append").format("parquet").saveAsTable("<database>.<table_sink>")
The lineage, as shown in the image below, has been inferred perfectly for the sink table. However, the source table displays the location on my Hadoop Data Lake, even though I'm reading from a table, not a path.early-hydrogen-27542
04/15/2023, 6:34 PMmetadata_aspect_v2
) for any versioned aspects? If so, the metadata store should theoretically hold all the lineage information, right? Looking at that table, it looks like the upstreamLineage
aspect accomplishes that. My initial plan is to:
• Ingest metadata_aspect_v2
into Redshift
• Parse the json in the metadata
column
• Model the results to show lineage relationships (e.g. one row per relationship)
Am I missing something in terms of native DataHub capabilities that might make this smoother?bland-orange-13353
04/17/2023, 11:07 AMearly-area-1276
04/18/2023, 12:20 AMcolossal-football-58924
04/18/2023, 12:20 AMbrave-judge-32701
04/18/2023, 2:49 AMfull-car-89338
04/18/2023, 3:28 AMcess to fetch at '<http://localhost:8080/api/graphql>' from origin '<http://localhost:5174>' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
colossal-autumn-78301
04/18/2023, 2:42 PMnamespace be.kqe.ste
and these files are in the same directory metadata-models-custom/src/main/pegasus/be.kqe.ste/CustomAspect_1.pdl
the command ../gradlew build
fails with the following exception. Also, with existing custom model DataQualityRules
it does not work. But if I remove dots
in the namespace such as bekqeste
then it works. any ideas
/Users/idris52/Git/datahub/datahub/metadata-models-custom/src/main/pegasus/be.kqe.ste/CustomAspect_1.pdl cannot be resolved.
13,15: Type not found: CustomAspect_1
at com.linkedin.data.schema.generator.AbstractGenerator.parseSources(AbstractGenerator.java:170)
at com.linkedin.data.avro.generator.AvroSchemaGenerator.generate(AvroSchemaGenerator.java:191)
at com.linkedin.data.avro.generator.AvroSchemaGenerator.run(AvroSchemaGenerator.java:165)
at com.linkedin.data.avro.generator.AvroSchemaGenerator.main(AvroSchemaGenerator.java:123)
Caused by: java.lang.IllegalArgumentException: be.kqe.ste.CustomAspect_1 has namespace that does not match file path '../metadata-models-custom/src/main/pegasus/be.kqe.ste/CustomAspect_1.pdl'
at com.linkedin.data.schema.generator.AbstractGenerator.validateSchemaWithFilepath(AbstractGenerator.java:221)
at com.linkedin.data.schema.generator.AbstractGenerator.parseFile(AbstractGenerator.java:194)
at com.linkedin.data.avro.generator.AvroSchemaGenerator.parseFile(AvroSchemaGenerator.java:221)
at com.linkedin.data.schema.generator.AbstractGenerator.parseSources(AbstractGenerator.java:132)
important-bear-9390
04/18/2023, 6:34 PMadamant-sugar-42217
04/19/2023, 7:36 AMenough-football-92033
04/19/2023, 6:15 PMdatahub docker quickstart
on v0.10.2 using this guide https://datahubproject.io/docs/quickstart/bland-orange-13353
04/19/2023, 11:27 PMrich-crowd-33361
04/19/2023, 11:27 PMrich-crowd-33361
04/20/2023, 12:33 AMrich-crowd-33361
04/20/2023, 12:33 AM