Hi Team, Please help with below in dedup config, ...
# troubleshooting
a
Hi Team, Please help with below in dedup config, while creating realtime table
Copy code
"dedupConfig": { 
        "dedupEnabled": true, 
        "hashFunction": "MD5",
        "dedupTimeColumn": "created_at",
        "metadataTTL": 86400000
   },
I used above but after table creation i see
Copy code
"dedupConfig": {
      "dedupEnabled": true,
      "hashFunction": "MD5"
    },
below got removed somehow.
Copy code
"dedupTimeColumn": "created_at",
        "metadataTTL": 86400000
Can someone please help me understand if I am doing something wrong? Why is the TTL config not showing after table got created? How to control it?
m
Perhaps you are using a version of Pinot that does not have TTL support?
a
I see https://github.com/apache/pinot/issues/15163 It seems the TTL was not there in Pinot v1.2.0 But there from v1.3.0
@Mayank - In this case, will the dedup hash table keep growing in server JVM ? How to plan the size then? Please let me know Any way to manually delete just the primarykey mapping from JVM to free up space ( for example, every 5days or so) ?
m
No, upgrade to the version with TTL
a
Thank you @Mayank