Hi people, is there a helper util for converting s...
# troubleshooting
c
Hi people, is there a helper util for converting scala case classes to
RowData
objects? right now I do it manually with a
.setField
but it would be nice if I could handle it programmatically for many different scala case classes. something like look up case class fields, create a GenericRowData, copy over the fields with the correct datatype?
n
I wrote something that does this but for Kotlin. The code here says "Delta", but it's just
RowData
so should work fine. Not sure how Kotlin's sealed classes will translate though. It's a bit much to share sensibly here, so if you want me to share everything then send a DM. After adding the things I wrote you basically just add a schema like this:
Copy code
object SomeSchema {
    const val VERSION = "0.1"

    val PARTITIONS =
        listOf(
            "version",
            "year",
            "month",
            "day",
        )

    private const val ROW_ARITY = 28

    val ROW_TYPE =
        RowType(
            List(ROW_ARITY) { position ->
                when (position) {
                    0 -> RowField("version", VarCharType(false, VarCharType.MAX_LENGTH))
                    1 -> RowField("year", VarCharType(false, VarCharType.MAX_LENGTH))
                    2 -> RowField("month", VarCharType(false, VarCharType.MAX_LENGTH))
                    3 -> RowField("day", VarCharType(false, VarCharType.MAX_LENGTH))
   ... snip
...}
and then in a mapper or whatever function:
Copy code
private fun create(
        input: SomeInput,
        rowType: RowType,
    ): DeltaTableRow =
        DeltaTableRow(
            rowType,
            setOf(
                DeltaTableField("version", string(SomeSchema.VERSION)),
                DeltaTableField("year", string(input.endedAt.year())),
                DeltaTableField("month", string(input.endedAt.monthValue())),
                DeltaTableField("day", string(input.endedAt.dayOfMonth())),
                DeltaTableField("field_1", string(input.sessionId)),
                DeltaTableField("field_2", string(input.field2)),
The
DeltaTableField
and
DeltaTableRow
above are part of the utility I created. The
string
function is also part of the utility, I added all the types we need including handling optionals etc. If you're interested just shout. Basically, we have a ton of jobs that write to Delta tables, and it became so messy and cumbersome to add more fields or whatever that I couldn't stand it anymore. So any new table we just literally add the mapper and schema as shown above. Nothing else required. fyi I don't know any Scala.