Clemens Valiente
02/18/2025, 2:58 AMRowData objects?
right now I do it manually with a .setField but it would be nice if I could handle it programmatically for many different scala case classes.
something like look up case class fields, create a GenericRowData, copy over the fields with the correct datatype?Nicholas Erasmus
02/18/2025, 4:05 PMRowData so should work fine. Not sure how Kotlin's sealed classes will translate though.
It's a bit much to share sensibly here, so if you want me to share everything then send a DM. After adding the things I wrote you basically just add a schema like this:
object SomeSchema {
const val VERSION = "0.1"
val PARTITIONS =
listOf(
"version",
"year",
"month",
"day",
)
private const val ROW_ARITY = 28
val ROW_TYPE =
RowType(
List(ROW_ARITY) { position ->
when (position) {
0 -> RowField("version", VarCharType(false, VarCharType.MAX_LENGTH))
1 -> RowField("year", VarCharType(false, VarCharType.MAX_LENGTH))
2 -> RowField("month", VarCharType(false, VarCharType.MAX_LENGTH))
3 -> RowField("day", VarCharType(false, VarCharType.MAX_LENGTH))
... snip
...}
and then in a mapper or whatever function:
private fun create(
input: SomeInput,
rowType: RowType,
): DeltaTableRow =
DeltaTableRow(
rowType,
setOf(
DeltaTableField("version", string(SomeSchema.VERSION)),
DeltaTableField("year", string(input.endedAt.year())),
DeltaTableField("month", string(input.endedAt.monthValue())),
DeltaTableField("day", string(input.endedAt.dayOfMonth())),
DeltaTableField("field_1", string(input.sessionId)),
DeltaTableField("field_2", string(input.field2)),
The DeltaTableField and DeltaTableRow above are part of the utility I created. The string function is also part of the utility, I added all the types we need including handling optionals etc.
If you're interested just shout.
Basically, we have a ton of jobs that write to Delta tables, and it became so messy and cumbersome to add more fields or whatever that I couldn't stand it anymore.
So any new table we just literally add the mapper and schema as shown above. Nothing else required.
fyi I don't know any Scala.