set the channel description: All relevant information to big data living together with kotlin language. The channel created to group people that are interested to use kotlin in this area

set the channel topic: Joining to see how relevant it is in Big Data direction. Share some news and thoughts.

Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure if anyone has been able to figure out. When importing that dependency my code get's a class not found error but works fine from running it in IntelliJ. When opening the Jar I can see it in the class path so stumped on what the issue is. Even weirder is I have a file that works perfectly fine with spark-core and the RDD api, however it breaks with the same error when I import the spark-sql library. As far as I understand that library is dependent on core and should be packaging it as well.

Hi folks, We are happy to share with you the Kotlin Spark API we’ve been working on: https://github.com/JetBrains/kotlin-spark-api.  Right now it is in beta stage and we need your feedback before we create a pull request to the official Apache Spark repository.  Why Kotlin Spark API? While you can use Kotlin with the existing Apache Spark Java API, Kotlin Spark API significantly improves the developer experience. For instance, this API allows you to use such Kotlin features as data classes and lambda expressions.  On top of that Kotlin Spark API adds some helpful extension functions. Use

withCached

to perform arbitrary transformations on a Dataset without it being recalculated, and don’t worry about your Dataset unpersisting at the end. Kotlin Spark API also allows you to have unnamed tuples that you can call with

c()

function that takes a variable number of arguments. You can add these to one another like in Python. Check out the Quick Start Guide to quickly set up all the needed dependencies using Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md Check out some code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples Try it and share your feedback with us either in #kotlin-spark, or via GitHub issues: https://github.com/JetBrains/kotlin-spark-api/issues. Thread in Slack Conversation

Hi all! I'm new here and I'm assessing the capabilities of Kotlin when working in the Big Data field. Any suggestions? Thanks!

Kotlin mostly does everything out of the box. The main difficulties you might have if you are less experienced is converting examples from Scala or Java into Kotlin.

Did anyone come to Kotlin from Scala or Clojure... why did you switch?

I'm really hesitant to get involved with Kotlin since I think the big data industry is moving in a different direction language wise (?) but others on my team want to use it , wanted to get any alternative perspectives here

Hi there guys I have an idea to use BigData notebook for graphical representation(charts etc) of data from postgres but I can’t find source for postgres in big data tools pluing Have I miss-understood something in this case

hey mates, i am new student of university, i am learning big-data analyse, does anybody know which lib i can learn and tools for Various algorithms, but also other machine learning or deep learning algorithms, such as: random forest, specific data analysis and data visualization There must be a process of data preprocessing, and corresponding visualization assistance must be carried out during data preprocessing, and the characteristics of the dataset must be observed and analyzed (for example, drawing a box plot or violin plot to observe whether there are outliers in the data features, drawing a scatter plot or heat map matrix to observe the state between the features of the dataset, drawing a histogram or histogram to understand the distribution of data features, etc.). now i know https://github.com/JetBrains/kotlin-spark-api and https://github.com/breandan/kotlingrad