Hi, Has anyone looked into tokenizing event attri...
# troubleshooting
j
Hi, Has anyone looked into tokenizing event attributes in a high throughput event stream? There's a clear path towards hash tokenization using a cipher like AES, which can scale and run entirely in a Process Function. But this won't meet all compliance requirements, such as GDPR right to be forgotten. What's being considered is a lookup table with a pseudorandom/random unique identifier. The lookup table needs to be accessed quickly and securely persisted. We'd have to make constant reads to avoid key collisions, so parallelization might not be possible. Would like to see if the community has worked on something similar before, and how they were able to leverage certain properties in Flink to enable this!