Hi Does Datahub support code with Spark in Kubernetes operat DataHub #ingestion

Hi. Does Datahub support code with Spark in Kubern...

millions-sundown-65420

09/03/2022, 7:14 AM

Hi. Does Datahub support code with Spark in Kubernetes operator: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator I get this exception:

Copy code

22/09/02 10:16:00 ERROR DatahubSparkListener: java.lang.NullPointerException
        at datahub.spark.DatahubSparkListener$3.apply(DatahubSparkListener.java:258)
        at datahub.spark.DatahubSparkListener$3.apply(DatahubSparkListener.java:254)
        at scala.Option.foreach(Option.scala:407)
        at datahub.spark.DatahubSparkListener.processExecutionEnd(DatahubSparkListener.java:254)
        at datahub.spark.DatahubSparkListener.onOtherEvent(DatahubSparkListener.java:241)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at <http://org.apache.spark.scheduler.AsyncEventQueue.org|org.apache.spark.scheduler.AsyncEventQueue.org>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

millions-sundown-65420

09/04/2022, 4:01 PM

I am using the helm chart for datahub: https://github.com/acryldata/datahub-helm/tree/master/charts/datahub should I import the datahub-spark-lineage jar file into datahub explicitly for this to work? io.acryldatahub spark lineage0.8.44

millions-sundown-65420

09/05/2022, 9:26 AM

Hi @dazzling-judge-80093 & @helpful-optician-78938 - any help on this please - really appreciate it. thanks struggling to get our Spark code emit events to Datahub now :(

Open in Slack

Previous Next