Hello , I am having trouble about neo4j backed lineage store. when lineages are nested and circuit, gms container always run into oom. then i find out the lineage shortest path check is in metadata-io, but not in neo4j side. current code in Neo4jGraphService.java:

final String multiHopTemplateDirect = "MATCH (a {urn: '%s'})-[r:%s*1..%d]->(b) WHERE b:%s RETURN a,r,b";
final String multiHopTemplateIndirect = "MATCH (a {urn: '%s'})<-[r:%s*1..%d]-(b) WHERE b:%s RETURN a,r,b";

should it become?:

final String multiHopTemplateDirect = "MATCH shortestPath((a {urn: '%s'})-[r:%s*1..%d]->(b)) WHERE b.urn <> '%s' AND b:%s RETURN a,r,b";
final String multiHopTemplateIndirect = "MATCH shortestPath((a {urn: '%s'})<-[r:%s*1..%d]-(b)) WHERE b.urn <> '%s' AND b:%s RETURN a,r,b";

and also remove:

// It is possible to have more than 1 path from node A to node B in the graph and previous query returns all the paths.
// We convert the List into Map with only the shortest paths. "item.get(i).size()" is the path size between two nodes in relation.
// The key for mapping is the destination node as the source node is always the same, and it is defined by parameter.
neo4jResult = neo4jResult.stream()
.collect(Collectors.toMap(item -> item.values().get(2).asNode().get("urn").asString(), Function.identity(),
            (item1, item2) -> item1.get(1).size() < item2.get(1).size() ? item1 : item2))
        .values()
        .stream()
        .collect(Collectors.toList());

hey Bowen! how many entities do you have that are nested and causing this OOM situation? also I would definitely suggest trying this out yourself and seeing if it fixes your situation! we would be more than happy to accept a contribution around this 🙂 the tough thing with neo4j is that the datahub team uses elastic search and we're therefore less familiar with neo4j and it's harder to support

hi Chris, entities i have are around 12000+, but nested or circle lineage between these datasets cause oom or gc problem. I made a perf PR and having more detailed explanation in commit https://github.com/datahub-project/datahub/pull/7219

okay gotcha nice! thanks for putting that up. someone from our team will definitely take a look. But you're seeing this fix working locally for yourself?

yes this fix work for my local build, 12 hops query results from neo4j decrease from 7 m*illion to* actual 1500+ using shortestPath

oh wow yeah we would absolutely appreciate a contribution for time filtering for neo4j! also appreciate the contribution for shortest path. We'll make sure to get a review on that soon!

Sorry for digging this out. We see the enriched

r_

edges are nice for lineages, but we also realise that duplicate edges are now added to all entity relationships, which do not appear very useful and make it more difficult to view the graphs (see pic below), and also lead to higher consumption of Neo4j storage. Would it be a good idea to limit the

r_

edges to lineage relationships only? Even if so I’m still wondering, for lineage, if there is a strong rationale to introduce reversed edges, while the original edge directions seem sufficient to me (as Neo4j works bidirectionally for planning shortest paths). Could we simply enrich the original edges with the timestamp/actor/etc. properties? Thanks! FYI @bulky-soccer-26729 @echoing-airport-49548 @silly-accountant-1288

hi Xuelei @fierce-guitar-16421, it’s ture that r_ edges will double the storage for Neo4j 😂. in case of

table_A

<-Consumes-

job1

-Produces->

table_B

table_A

<-DownstreamOf-

tableD

table_A

-DownstreamOf->

table_E

using bidirectional cypher is really hard to find upstream lineage or downstream lineage. please share if you have better solutions!!🥰

I see! The job entity is the problem since it cannot be collapsed into a simple uni-directional edge which can be included in the shortest path query. I don’t really have a great idea at this moment. Maybe a reversed edge “IsProducedBy” can be added in addition to “Produces”. Then the shortest path query could roughly look like:

MATCH shortestPath((a {urn: "%s"})-[r:DownstreamOf|Consumes|IsProducedBy*1..%d]-(b)) WHERE b.urn <> "%s" RETURN a,r,b

and the shortest path query can walk through both Datasets and DataJobs. No further types of edges seem to be needed. But still, at least it might be worth considering to avoid adding the

r_

edges to the non-lineage relationships, since they don’t appear to be very useful for now.