Hey everyone :wave: I am using Pinot in conjuncti...
# general
f
Hey everyone 👋 I am using Pinot in conjunction with Debezium to ingest change capture data in realtime into tables. I use the upsert functionality to ensure that only the latest version of the row is preserved in the table. Since this works by checking primary keys, I run into the following scenario with Debeizum. For primary key columns that are of type BIGINT, when the integer stored reaches beyond a certain range, it is converted into a base64 encoded string. In other words, a primary key field can be an integer or a string in the Kafka stream. The Groovy script below shows how to get the BigInteger back.
Copy code
import java.math.BigInteger;

class Ideone {
    static void main(String[] args) {
       def number = "eA=="
       
       if (number.toString() ==~ /\d+/) {
       	def bi = new BigInteger(number.toString())
       	// return bi
       } else {
       	def bytes = Base64.getDecoder().decode(number)
       	def bi = new BigInteger(bytes)
       	// return bi
       }
    }
}
My question is: can Groovy scripts return BigIntegers for storage in Pinot or would they have to be converted into strings?
Looking at the available
ScalarFunctions
, the following transformation could potentially work:
Copy code
"bytesToBigDecimal(base64Decode(jsonPath(payload, '$.after.id')))"
However, the present implementation of
base64Decode
accepts byte array and not a string. Perhaps adding an overloaded version which accepts strings might do the trick? Happy to raise a PR.
x
I prefer to just use string as primary key and you can query it with your client side logic.
BigDecimal is a bit different than BigInteger
f
@Xiang Fu Ah, thank you for noticing that it's a
BigDecimal
and not a
BigInteger
. Could you please point me to the transformation function to convert an input value to string?
x
Just configure column as string, then Pinot should keep it as string
f
Hey @Xiang Fu, what does this document mean when it says that ".function." must be somewhere in the project? Also, what version of Java should the project use?
One of the ways to achieve what I've outlined in Groovy above would be to ship a custom JAR and put it in the
plugins
folder of the cluster.
x
yes, you need to make sure the package of your class is under
org.apache.pinot.*.function.*