https://datahubproject.io logo
Join Slack
Powered by
Hi, I'm new to DataHub and I'm trying to understan...
# getting-started
s
Hi, I'm new to DataHub and I'm trying to understand the DB model and source of truth. I saw that this file contains a table: https://github.com/linkedin/datahub/blob/b8e18b0b5d56b4fa69b4bc35e8055176f9577dee/docker/postgres/init.sql However I don't fully understand if all of the data (even if there are 10B+ records), will be contained there or not and if you saw that it is a method that scale, or other tables are generated on the fly. Is it also append only table?
b
This is the single table that will hold all the metadata. No schema evolution is needed. That said, MySQL (or other traditional RDBMS) isn't really designed to scale beyond 100s of millions of rows. We're actively working on a NoSQL-based implementation which should scale beyond 100s of billions of rows.
m
@shy-lizard-17779 : do you need those 1B+ records to be in the DB or do you just need a metadata log that can contain all this data and get shipped off to big data lake (with interesting subset of that metadata stored in db)?
s
Basically are 2-3 large metadata tables that we are collapsing into a system similar to datahub. The data is internal for datascientists. I was implementing a datahub myself (based on postgres+elasticsearch) before encountering into this. We have about 7B rows subject to grow in a 3TB Postgres cluster. To scale the DB we use Citus plugin for postgres that allows to very easily partition the data among multiple machines with a "multi master" approach that we found very convenient.
m
Got it. Do you think you cannot do this with the way Datahub uses MySQL?
s
I thought it was DB agnostic (supporting both). We are trying to move away from MYSQL since we believe Citus made Postgres quite superior from a cluster manageability perspective/performance
m
Yes it is agnostic though at LinkedIn we run it on MySQL. I’m of course very curious about the 7B things you have in there :) are they entities or relationships or events?
s
Just entities and relationships that our datascience would search on
m
Got it
So do you see anything in the ddl that makes you think Citus wouldn’t be able to scale this?
s
Our first internal prototype between postgres and elatic works very well for what we have to do (from the point of view of how fast the queries are, and how easily we can index). However schema evolvability is not really easy, and we are trying to understand how to not do "not invented here"
m
Hah right.. which is why we do blobby storage for the things we don’t need indexes for in the rdbms and delegate that to elastic
s
I don't fully know what the query model is in datahub
m
I don’t know exactly how your prototype is set up, but I assume you tell Postgres everything about your model
s
we split the table in two, one for the aspects, the other one for the relationships, so they can be two ways
Yeah, we don't have the best model management yet
m
We use graphdb (neo4j) for relnship queries
MySQL/Postgres for primary key lookup
And elastic for search and secondary index lookup
There is work in progress to support strongly consistent secondary indexes in rdbms in a way that doesn’t create agility challenge with schema evolution
I would also ask, how fast is metadata changing and are you getting a constant stream of changes?
s
We don't need strongly consistency for the query usually. Given the numbers, the updates are fairly frequent. Mostly in batches.
b
This is actually quite an impressive size you have there. Do you mind to share your company name?
m
Cool .. as you must have noticed the “metadata” column is the blobby piece, the aspect,urn,version represent the pkey
b
I'm assuming it's mostly operational metadata or jobs-related metadata?
DataHub supports MySQL, Postgres, MarinaDB out of the box. The DB driver should support MS SQL & Oracle in theory but we have not tested those.
m
@bumpy-keyboard-50565 I think you meant MariaDB
b
lol yes sorry typo.
I'm not familiar with Citus but I'm assume it's similar to Vitess (https://github.com/vitessio/vitess)?
m
@shy-lizard-17779 : we support customizable retention of versions, so you can have last N versions of each metadata entry to be retained so that table doesn’t grow beyond bound
s
A few things: • versioning is great, in our DB we added an end date (of when the data is replaced), so we can easily look up by date with a simple query • In our elastic we index not only the entity, but also all of the entities that it is related to (so we can give more powerful queries). Is it possible in datahub? I guess those queries should all be done on neo4j given that in the metadata db everything is serialized Yeah it sounds similar to Vitesse. Actually Citus is now owned my MSFT and also comes with azure :) Sorry can't disclose yet
b
More high-level information on retention/versioning: https://github.com/linkedin/datahub/blob/master/docs/what/aspect.md While it's possible to combine metadata from multiple entities into one search index, we generally encourage using graph db for that kind of query patterns.
m
@shy-lizard-17779 : yes better to let graphdb do all the relnships
s
Do you feel neo4j will scale? We feel it mighthat neo4j would just crash under that. e.g. a query that returns 100k entities, joining on 4 tables, each with a few conditions. Queries in the future will also involve some text (descriptions). Elastic search just scales seamenglessly. No matter how much load we added to it, it just worked, and with the SQL extension it is just wonderful to use
m
Does elastic support joins now? I was under the impression that it is a single table abstraction
So you would need to denormalize
s
Some joins but we don't use it. As was saying before, we denormalize on level of relationships in nested fields
m
There is nothing preventing you from doing the same denormalization in datahub as well
🙌 1
s
That you can run queries x document. E.g. SELECT entity FROM index WHERE nested(nicknames, nickname.name="love jeff")
m
Depending on how your metadata change event looks, You will have to tap into the indexing pipeline and “materialize” the denormalization into the index. If that makes sense.
s
I see. So to tap in the indexing pipeline, is there already any code saying "when this entity updates, signal the identities that are connected"?
m
You can customize it to your hearts content
m
Specifically your index builder can "subscribe" to changes in several entities by updating the list of snapshots here https://github.com/linkedin/datahub/blob/master/metadata-builders/src/main/java/com/linkedin/metadata/builders/search/DatasetIndexBuilder.java#L26
b
neo4j 4.x now claims to be horizontally scalable. Internally we have our proprietary in-memory graph DB that scales to LinkedIn's economic graph scale (billions or nodes & edges) and maintains millisecond query time. Performance in graph is purely a function of how much to hold in memory.
s
@bumpy-keyboard-50565 interesting! are you using/plan to use neo4j internally for datahub or a custom DB? Do you mind sharing if you did any scaling testing?
b
We're currently using neo4j and plan to transition to our custom DB in the next few quarters. The DAO is designed in such a way that it's mostly implementation agonistic. The metadata graph we have currently is still small O(10M) nodes so no scaling issue so far.
Open in Slack
PreviousNext
Powered by