hi all, quick question, in the table detail page, whats the difference between "last synchronized date" and "last observed date" under schema section ?

hey William! great question. The last synchronized (on the top of the profile sidebar on the right side of the page) will represent the last time datahub heard about this entity as a whole from ingestion. So the most recent time any aspect of this entity was ingested. Last observed in the schema tab represents the last time that datahub heard about the schema of this table specifically. Usually the two will equal each other, but if you ever ingest aspects that don't include the table's schema, then they could diverge

Thanks for the response @bulky-soccer-26729! We've scheduled the the ingestion recipe to run daily . Currently "last observed" / schema info is up to date while "last synchronized" is a week behind. Any idea what could have caused the behavior? we've also noticed some lineage is broken (some nodes disappeared sometime during last week) so it seems it could be related to some ingestion issue as well.

yeah that could be the case.. when coming up with "last synchronized" we check the

systemMetadata

column in mysql and find the most recent

lastObserved

out of all the aspects where

runId

in the

systemMetadata

column is not empty or the default (

no-run-id-provided

). Can you check to ensure that a

runId

is being generated for these out of sync entities after ingestion runs?

ok will do, thanks!