https://datahubproject.io logo
Join Slack
Powered by
I basically want to know how datahub uses elastic ...
# getting-started
s
I basically want to know how datahub uses elastic search and MySQL to store index or data in datahub?
b
hey Nishchay! we have some docs for these questions and hopefully they can help you out. I'd first start with an overview of overview of the architecture which should lead you to checking out an overview of the components involved in our architecture where you can follow some links to get more specific details. Here you can find out more about how we use elasticsearch and what types of data are stored in the document store (mysql)
s
Yeah Already go through it
but I want to know how datahub uses mysql+elasticsearch, what kind of data format stored in MySQL (as rows or not) or elasticsearch does point to MySQL rows using primary or secondary index ?
what is main use case of using kakfa as service deployment with elasticsearch, what kind of data kafka stored?
@bulky-soccer-26729
can you please help
b
so ultimately mysql is going to be your document store which stores your metadata aspects. one columns is an 
urn
(looks like: 
urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)
) and another is 
aspect
which will be what type of data this row will store (such as 
datasetProperties
) and another column is 
metadata
which is where we'll store the data related to that aspect. I would highly suggest getting a mysql viewer (I use Sequel Ace) to give you a better understanding of what's in there and how it all related
elasticsearch is used as a search and a graph index. This will be used to store search index information as well as relationship indexes (such as who the parent node of a glossary term or who the children are of a glossary term group). We also store timeseries based data in elasticsearch as well - here's more info on timeseries aspects
s
sure , @bulky-soccer-26729 thanks
7 Views
Open in Slack
PreviousNext
Powered by