im finding that sometimes, upon ingesting a new ve...
# troubleshootb
better-orange-49102
06/29/2022, 7:19 AMim finding that sometimes, upon ingesting a new version of an aspect, the original version of the aspect will disappear from mySQL. But if i were to follow up and emit a slightly different value again, the newly inserted aspect does get incremented correctly. What are some possible areas to check?
These aspects are not using ingestion sources, but generated mcps from the python library.
I did not set a retention policy yet, and there are no errror messages in the frontend/gms logs. I also didnt specify a version# in the MCP.
better-orange-49102
06/29/2022, 11:16 AMso this error is happening to older records in mysql (created before we recently updated Datahub) - when a new schemaMetadata aspect is emitted, the existing version should have been copied to a new record. (calling this case 1)
For newly created datasets, it behaves correctly -> the old aspect is copied to a new record and version 0 is updated (case 2)
looking at the query logs, it appears that mysql is being sent the correct set of instructions each time, yet the outcome is different for the 2 cases.
for case 1, after i emitted the aspect, i only have v0 for schemaMetadata
for case 2, i have both v0 and v1
any ideas?
better-orange-49102
06/30/2022, 8:16 AM@big-carpet-38439 @orange-night-91387 any suggestions would be greatly appreciated, thanks!
better-orange-49102
06/30/2022, 9:59 AMnever mind, finally traced it to the datahubRetention record. i previously mounted a retention policy in ${HOME}/.datahub/plugins/retention/retention.yaml, it created records in DB. i subsequently restarted the container without ENTITY_SERVICE_ENABLE_RETENTION=true and thought that it wasn't active anymore.
The retention was working in a weird way also
But removing the retention records in DB solved the problem
🙌 1
2 Views