https://datahubproject.io logo
Join Slack
Powered by
hey, is there a way to keep only `n` last versions...
# ingestion
b
hey, is there a way to keep only 
n
last versions of aspects? Mysql is growing rapidly in size and I don’t really have any use for old aspects
b
Great question!! We've currently had implementing retention policies on our backlog so glad to hear you guys will need this. Would you have a preference about how retention policies are implemented? They can most likely be either a) time-based (after a certain TTL, aspects are removed) or b) max-version based (where you delete all aspects greater than version X). We are leaning toward #2 for the initial impl & sounds like you are too
b
No preference, really. I’ve been reading through other threads - this one to be specific and is it true that if the aspect is exactly the same gms shouldn’t persist another version of it to mysql?
although when I think about it, I would probably lean towards #2. Some tables might lie there for months untouched, so I assume their aspects would get deleted if they exceed some ttl - in my use case that would be very undesirable :)
b
It is true that if nothing changes, we do not duplicate the info
Thanks for the feedback!
b
So I have this issue with ingesting looker data:
as far as I can tell the only thing that changes is the order of 
customProperties.upstream_fields
ok, so this also propagates to 
upstreamLineage
aspect for looker urns as well.
b
Ah got it - order changes can indeed cause an overwrite. @mammoth-bear-12532 @miniature-tiger-96062 Maybe we can track this issue? Seems to be every time ingestion is running we are adding new aspects
m
@bumpy-activity-74405: thanks for flagging this. We’re taking a look.
b
Hey@big-carpet-38439! Just remembered this thread and I was wondering whether you guys have implemented any sort of retention policy?
b
@early-lamp-41924 Is working on a PR for this right now 🙂 Should be out in the next week
👍 1
Open in Slack
PreviousNext
Powered by