https://datahubproject.io logo
Join Slack
Powered by
Hi team, i have a question about storing metadatas...
# ingestion
c
Hi team, i have a question about storing metadatas into mysql. Why datahub stores pipeline metadatas by keeping all versions instead of overriding? We have so many dags and some of them are running every 5 min, 10 min etc. I’ve checked the mysql and saw that we have 1500 version number in specific task created in 4 days
e
We are tackling version and time based retention in our local db in the next two weeks. Aiming to have a solution by then.
Curious tho, have you checked diff between the versions? is there timestamp stored in properties?
c
@early-lamp-41924 I think the only difference is lastmodified field in systemmetadata column and createdon columns. Only these values change and it becomes new version…
If we continue without any change, we will have about +1m row in a month and this doesn’t makes sense since versions are meaningless
e
Hmn systemmetadata changes should not create a new version
cc @green-football-43791 this is the case right?
Can you see if there are any other changes? for us, we saw that the properties bag sometimes has a field that changes on every run
g
correct- system metadata changes should not create a new row
c
@green-football-43791 The aspect is ownership and i also see that in metadata column, “time” also changes.
g
ah, it must be that they are updating the ownership timestamp inside the aspect as well
c
Copy code
{
  "owners": [
    {
      "owner": "urn:li:corpuser:xxx",
      "type": "DEVELOPER",
      "source": {
        "type": "SERVICE",
        "url": "dag.py"
      }
    }
  ],
  "lastModified": {
    "actor": "urn:li:corpuser:airflow",
    "time": 1635798300000
  }
}
the only changing part is time
g
got it.
c
what should i do
g
what source is this? airflow?
the pipelines you are concerned about
c
yess
g
and are you ingesting using the airflow source?
or lineage backend?
c
lineage backend
g
ok, let me take a look
i think i may have identified the fix
here's my proposal for what should be done
I'll let some of the more airflow lineage experts weigh in though
c
Ok thank you
b
Thanks Gabe!!
c
@green-football-43791 I think we need to wait for next release in order to get this update, right?
Open in Slack
PreviousNext
Powered by