https://datahubproject.io logo
Join Slack
Powered by
Hi folks, maybe this is more of a deployment relat...
# all-things-deployment
c
Hi folks, maybe this is more of a deployment related question.
s
One concern was Neo4j required a license IIRC that was the big reason for the switch to elastic for graph queries. Elastic was already being used for search. So removing Neo4j removed an extra thing in deployment to manage
l
@calm-airplane-47634 we have found that for most of our current use cases we just do a one-hop graph query and Elastic serves as a good enough graph store for that
There are some advanced use cases which require multi-hop queries from a single node and a specialized graph database like neo4j will perform better (e.g: fetching all impacted down streams several hops away if an airflow dag fails). We are hoping elastic will work acceptably well for these use cases too and will do some benchmarking to verify
Where graph databases really shine is in the multi source multi-hop use cases. At the moment we think this is better than by exporting all the metadata to your data lake/warehouse and running queries there instead of doing them online
Overall, simplicity of the stack was the motivating factor for supporting elastic as a graph store as well
c
Got it, it looks like overall the recommendation is to start and stick with “elastic search” till there is a real reason to do so i.e. neo4j is more of a fringe case at the moment. In my experience, people who want to do analytics workload against metadata usually do that alongside other datasets which are not already are in data warehouse.
unrelated question: What is the mysql database used for?
l
Open in Slack
PreviousNext
Powered by