hi! I have been messing around with the spark listener and I can get the spark application to finish. It got hung every single time and the onApplicationEnd never get called. Performing some trial and error I have been able to track down the issue and I think is it related to the emitter and the thread pool not being properly stopped and shutdown. If I do so on the SparkListenerSQLExecutionEnd event, everything goes well. But that is not an option because of many other problems it entails, any clue?

have you called spark.stop() at end in your code? This allows proper shutdown of emitter threads

oh I dont