Hi all, my team and I are migrating our AWS KDA to...
# troubleshooting
t
Hi all, my team and I are migrating our AWS KDA to the recently supported flink-1.15.2. And we are facing some random runtime exceptions, which can’t be reproduced in my local environment. It happens at our Scala code portions that invoke List.map or Seq.map. The exception is: java.util.NoSuchElementException: head of empty list _at scala.collection.immutable.Nil$.head(List.scala:469) at scala.collection.immutable.Nil$.head(List.scala:466) at scala.collection.immutable.List.map(List.scala:293)_ Similar exceptions happen when we use Case Class as data record between streaming task: the record fails to be serialized as it invokes similar code to List.map(). We are suspecting the Scala implementation at List.scala:297 tail ne Nil was not correctly executed, but can’t investigate further. Have you ever faced such error in the latest KDA? Please let me know. Thanks in advance. Note: we are using Scala: 2.12.15.
m
If you use the Flink Scala APIs, the latest Scala version that is supported is 2.12.7. Given that the Scala APIs are deprecated, I would recommend to start using the Java APIs because that allows you to use any Scala version. See https://flink.apache.org/2022/02/22/scala-free.html
t
Thanks, but I am afraid the issue is related to generic scala runtime execution in KDA, not about Flink Scala API.
m
I'm by no means a Scala expert, but since there's no binary compatibility between Scala 2.12.7 and 2.12.15, isn't it more logical that this might cause the problem?
➕ 2
h
Does the code work when running locally with RocksDB as state backend? It seems there is a known issue when serializing a Scala List with Kryo serializer https://stackoverflow.com/questions/25380517/unable-to-iterate-over-scala-list-serialized-with-kryo and RocksDB as state backend will trigger serialization whenever items are put into state
👍 2
I don’t have experience debugging Scala serialization, but seems like case classes should be supported. It’s possible List or the contents of the List is not supported? https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/fault-tolerance/serialization/types_serialization/ Maybe this might help with registering a custom serializer: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/fault-tolerance/serialization/third_party_serializers/
a
Hi Max
It happens at our Scala code portions that invoke List.map or Seq.map. The exception is: java.util.NoSuchElementException: head of empty list
Additionally if you are seeing it
Hi Max Do you have a specific class loader config setting in your local runtime?
t
@Hong Teoh, thanks. The stackoverflow link you sent confirms my doubt over the matter. I have looked into the implementation of List.map():
Copy code
final override def map[B, That](f: A => B)(implicit bf: CanBuildFrom[List[A], B, That]): That = {
  if (isLikeListReusableCBF(bf)) {
    if (this eq Nil) Nil.asInstanceOf[That] else {
      val h = new ::[B](f(head), Nil)
      var t: ::[B] = h
      var rest = tail
      while (rest ne Nil) { // --> Problematic Condition
        val nx = new ::(f(rest.head), Nil)
        <http://t.tl|t.tl> = nx
        t = nx
        rest = rest.tail
      }
      h.asInstanceOf[That]
    }
  }
  else super.map(f)
}
The problem is at the while loop condition, where ne operation is performed. In scala, ne and eq are comparisons by memory location. During debugging, the error happened randomly with List of 0 or more elements (I dumped the list content in a try / catch). Hence, I concluded that the while loop condition returns true, even when the rest variable has no more element. One way to explain (according to my limited knowledge) this is that: Nil is case object in Scala, which is translated into a singleton static class / object in Java. However, JVM can still create different singleton objects through different separated class loaders. To investigate deeper about this when I have time. Anyway, we have worked around the problem by avoiding List.map() (using foldLeft / foreach instead) and the issues happened no more. However, for long-term, I feel that scala's List implementation is not really fit for distributed computing. Other notes: • Not sure why similar problem does not happen in KDA 1.13.2 • Case class serialization also passed through similar implementation (listMap with ne operation), and hence throwing the exception, making our app crashed and restated all the time.
👍 2
Hi @Ahmed Hamdy, no I don't use custom class loader in the app. I used the same jar file as the one on KDA, and did not find the issue in my local.
h
Wow thanks for the deep dive! What you said makes sense. Just one thjng to add, in Flink we can register custom serializer, and it will no longer fallback to Kryo. That might help side step this issue for serialization That might also be why it works on Flink 1.13.2 and not 1.15. https://flink.apache.org/2022/02/22/scala-free.html This change was made in 1.14
> For example, the Kryo serializer, which we always extended to support certain Scala types, now only includes them if an application uses the Scala APIs.
👍 3
n
This thread was super useful to me to solve my scala application issue around deserialisation. For anyone who is struggling around deserialisation of data into case class, please check if it using Kryo. If yes consider including Flink-scala and chill library in your classpath.
🙌 2