Hey everyone! Lately my team has been seeing some issues in one of our datahub environments, namely it appears data is not being pushed to our graph database (neo4j community edition) when new items are ingested. The main example I have is for the UpstreamLineage aspect. When ingesting a set of these aspects, we're seeing the data show up in mysql, but not neo4j. Additionally, when we hard delete the entity from datahub using the CLI. It is removed from mysql, but is not removed from neo4j. However, the connection between the gms service and neo4j seems to be working fine for standard queries because whatever dat is present in neo4j is visible in the frontend UI. The following are some steps and results while identifying and debugging this issue to create a timeline for you all: 1. Ingested new dataset entities. They appeared in mysql, neo4j, and the UI. 2. Ingested lineage data from these new dataset. All of the lineage appeared in mysql, but only a subset of the lineage appeared in neo4j & the UI (seemingly all oracle tables). 3. Reindexed a single urn for a downstream dataset. The DownstreamOf relationship now appears in neo4j for the reindexed dataset, and the correct lineage is shown in the UI. 4. Ran the RestoreIndices kubernetes job for all aspects. Job ran for ~9 hours and completed successfully however no knew relationships appeared in neo4j or the UI. 5. Restarted neo4j, no effect. 6. Manually added one of the missing edges to neo4j. The correct lineage then appeared in the UI. 7. Did a hard-delete on one of the dataset entities. The dataset was deleted from mysql and elasticsearch, and no longer was present in the UI, however the node and relationships were still present in neo4j. 8. From this point on we switched over to kafka emitter as the rest emitter was seemingly related to similar problems in the past. 9. Reingested the deleted dataset. It reappeared in the UI with the partial lineage info it had before being deleted. 10. Manually deleted the lineage relationships from neo4j for that dataset and reingested the UpstreamLineage aspect. The aspect appeared in mysql, but the relationships were not recreated in neo4j or the UI. 11. Tried several combinations of restarting datahub, restarting neo4j, reindexing, and reingesting. No effects. 12. We've also seen some validations aspects be created in mysql after ingestion, but not appear in the UI. We've seen an issue like this pop up in the past that appeared to be related to the REST sink. The REST sink was used for the first seven steps of this timeline, but we have switching to kafka emitter now. When similar issues occurred in the past we were able to resolve them by reindexing the database and restarting our graph db a few times, but that does not appear to be working here. If anyone has any thoughts or ideas regarding directions to move in on this issue, I'd love to hear them. Thanks in advance!

cc @orange-night-91387 Please help us digest this^^

This is v0.9.5 by the way

For John: To try to reproduce

Hi! 1 ) Are you running with MAE Consumer in embedded or standalone mode? 2) Have you seen any errors on the MCL processing side? Would be either GMS logs or MAE logs in standalone

Hey Ryan, 1. We're running it in embedded mode as part of the gms container at the moment. We have two replicas of our gms pod in this environment if that matters 2. I haven't seen any yet, but I'll run another test here in a bit to be sure and will share anything I find

Specifically look for:

UnsupportedOperationException("Remove edge not supported by Neo4JGraphService at this time.")

and check this env var:

GRAPH_SERVICE_DIFF_MODE_ENABLED

Thanks for the info. Sorry for the delay here, we're currently experiencing some unrelated infrastructure issues in this environment that are preventing me from checking for this right now. As soon as I get the application back online I will report back. Thanks!

(by default going forward we should have this addressed)

Okay so I got our environment stood back up, ran some tests, and have a few things: 1.

GRAPH_SERVICE_DIFF_MODE_ENABLED

is not an env variable that exists in our gms pods 2. I checked our logs for the past month and found several instances of

UnsupportedOperationException("Remove edge not supported by Neo4JGraphService at this time.")

in them 3. We ran into an issue with kafka yesterday where it was unable to acquire a lock file for its data volume (likely related to a cluster upgrade that was done). We had to wipe the kafka data volume to fix it, but when I stood the env back up today I did some more tests and was able to get the lineage to load into neo4j. I'm not sure why, but this time when I hard deleted everything it actually disappeared in neo4j and then when I reingested the data again via the kafka emitter it finally made it's way into neo4j and thus the frontend as well. With this finding, I'm not sure if this was related to the issue you guys found or not.

GRAPH_SERVICE_DIFF_MODE_ENABLED

defaults to true, so you'll have to set it

Oh okay I'll look at that. Can you give a brief overview of what this toggle does. I assume it makes it so only new changes are pushed to the graphdb and not all changes. Is that correct?

Our historical logic for graph updates does a remove-all -> re-add all for the edges. Whenever this was first implemented we didn't send across the previous aspect state so it would have required a read-modify-write to do a difference, but since we send the previous state in MCLs now we can introspect and do targeted adds/removals.

Okay thanks for the insight