Hey everyone! So somewhat recently my team upgraded to datahub v0.9.3, but we've started seeing the following issues: • Attempting to view the lineage tab on a dataset results in a 503 error. However, when clicking "Visualize Lineage" the lineage visualization loads just fine. Maybe because these pages use two different graphql queries to pull data? • There are virtually no logs from the gms containers. I can see the initial startup logs when the container comes online, but then there are no logs until the lineage tab is loaded a few times and the following error occurs:

java.lang.OutOfMemoryError: Java heap space

. When this happens the entire application freezes until those containers are manually restarted. I've tried tripling the RAM for these containers and saw no change. • When I create a new policy it says it was successfully created, but the new policy does not appear in the list nor do its permissions take effect. When I check the MySQL database though, I can see the rows for the new policy which was created. I'm unsure if there are other issues, this just what we've observed over the past few days. I downgraded one of our instances to v0.9.2 and all of these issues went away. Are there any thoughts on what may have changed between v0.9.2 and v0.9.3 when could have caused these issues? Also, I saw that v0.9.4 just recently came out. Is there reason to believe that some of these issues may be resolved by upgrading to that version?

Hi @bland-balloon-48379, 0.9.4 still has some kinks being worked out, so I’d hold off on updating until we’ve resolved those.

Hey Paul, thanks for that update. I appreciate the input

Do you think you could document this with screenshots here next time it happens?

Hi Paul, sorry for the delay on this. I'll look into doing this for sure

@bland-balloon-48379 thanks for the updates here, this is super helpful!

@astonishing-answer-96712 So I just spent some time trying to debug this issue and now I'm unable to reproduce it. I'm not sure if the upgrade to v0.9.5 took a bit to take hold for this particular issue or what, but the lineage tab seems to be working as expected now. Appreciate the help!

Hi @bland-balloon-48379, we’ve been seeing an uptick in neo4j issues lately. Do you have any logs you could provide ?

Hey Paul, I can provide a few of the logs with portions of specific urns redacted. Here's what I have as far as relevant logs go:

13:22:21.566 [pool-13-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3AREDACTED) - batchGet - 200 - 2ms
13:22:21.626 [pool-13-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Adataset%3A%28REDACTED%29) - batchGet - 200 - 57ms
13:22:21.633 [pool-13-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3AREDACTED) - batchGet - 200 - 3ms
13:22:21.711 [Thread-35782] INFO  c.l.m.graph.neo4j.Neo4jGraphService:247 - MATCH (src {urn:"urn:li:dataset:(REDACTED)"})<-[r:Asserts ]-(dest )
13:22:21.711 [Thread-35781] INFO  c.l.m.graph.neo4j.Neo4jGraphService:247 - MATCH (src {urn:"urn:li:dataset:(REDACTED)"})<-[r:Asserts ]-(dest )

I got these from loading up a dataset, navigating to the lineage tab, encountering the error, and then opening up the lineage visualization. I don't have anything else relevant on my end.