Hi folks, We're exploring DataHub as a data catalog service and we were wondering if we could simplify the deployment / dependency footprint a bit. From what I understand, we need to deploy the frontend, gms services as well as Kafka, a database (e.g MySQL / Postgres), Search svc (Elasticsearch) and a graph service (e.g Neo4j). Curious if we can skip some of these as it's a bit cumbersome for us to spin up all these dependencies upfront. Im wondering if we can: • Skip Kafka if we're ok using Rest based ingest and don't have any MAE consumers at the start • Store the graph index in ES like suggested in the docs - not sure how much of a risk this is perf wise

You can definitely omit neo4j, but all other components are required.

Thanks for clarifying @early-lamp-41924. Are there any docs around how we can configure the Kafka topic etc for the GMS <-> ES queueing?

Are you referring to the topic names?

yeah

So we do have this doc but seems a bit outdated. (I’ll updated this today) https://datahubproject.io/docs/how/kafka-config/#configuring-topic-names But here is our code that generates the topics! https://github.com/datahub-project/datahub/blob/master/docker/kafka-setup/kafka-setup.sh Hope this works as a starting point, while I update the docs!

Ah great that should work. It looks like we need to spin up 4 topics. Are all 4 of these required for the GMS -> ES piece or just the metadata audit event topic?

So the ones used internally are

ENV DATAHUB_USAGE_EVENT_NAME="DataHubUsageEvent_v1"
ENV METADATA_CHANGE_LOG_VERSIONED_TOPIC="MetadataChangeLog_Versioned_v1"
ENV METADATA_CHANGE_LOG_TIMESERIES_TOPIC="MetadataChangeLog_Timeseries_v1"

@bitter-lizard-32293: another thing to keep in mind is the serialization and schema registry integration. What do you use at your company?

@mammoth-bear-12532 yeah good question. We're largely a protobuf shop and are currently in the process of spinning up the schema registry project within Stripe for general use. It's likely that for Datahub we'll end up sidestepping the schema registry piece to start off with and just reach out to spin up the relevant Kafka topics and allow gms to send messages in the ser format that it chooses. We took the same route with schema registry (as it uses kafka to persist and doesn't use proto for the messages).

got it, makes sense

I wrote an medium blog about deploying OSS Datahub with mini. setup.

@mammoth-bear-12532 - I think I got a bit more context on your question now (slightly belated 🙂 ). I didn't realize at the time that DataHub requires a hard dep on either the confluent schema registry / aws glue schema reg (if I'm correct tracked in this https://feature-requests.datahubproject.io/b/Developer-Experience/p/remove-required-dependency-on-confluent-schema-registry). Is this work still planned for q2? Any pointers for folks who want to disentangle this?