hello I've got a few questions on functionality that I can't work out from the docs or the demo site. 🧵

Hi @brave-forest-5974 great questions… answers inline .. 1. Ranking: We will have basic ranking functionality in the OSS project but the more advanced version which uses a continuously trained model which takes into account various signals (usage, graph connectivity, data quality etc.), is only available in the Acryl Data managed version of DataHub. We have made the interfaces pluggable so that open source adopters can plug-in their own implementations if necessary. 2. ETL: This is very easy to do, just subscribe to the two Kafka topics: 

MetadataAuditEvent_v4

and

MetadataChangeLog_Versioned_v1

and ETL them to your lake or data warehouse. 3. Lineage: We currently cap the lineage viz to 100 upstreams and downstreams for a single node, so the UI shouldn’t suffer too much. We have plans to make the UI and API support filtering  search experience within the lineage graph to allow for navigating really dense graphs. 4. GraphQL API security: We are targeting to release support for access tokens in the next couple of months. cc @big-carpet-38439 who can add more context here 5. Upgrades: For primary data, we recommend using managed services (like AWS RDS or similar) and using its backup-restore functionality. For restoring indexes, we have a helpful guide here (https://datahubproject.io/docs/how/restore-indices/). In our managed offering, we follow a similar approach for our customer deployments. 6. Scaling: The datahub services are stateless and so are theoretically infinitely scalable horizontally. In terms of scaling bottlenecks, currently the bottlenecks that we are aware of are in the metadata service (for batch ingesting tons of metadata -> e.g. ingestion of 2K+ Looker entities takes about 15 seconds with parallel REST calls: quite a bit of low hanging fruit here), for up to a million entities with an average metadata footprint of 1MB per entity, we have seen customers being comfortably served with the large instance versions of the specific technologies (RDS, Elastic, Kafka) with a minimum of 3 hosts for the distributed systems. UI usage is typically not where you will see any bottlenecks, you will only see performance being a concern when you start using this for programmatic use cases. Obviously at that point, the workload matters a lot (e.g. are these predominantly primary key based queries or search queries etc). [all primary key based reads go to MySQL / Cassandra. Only graph queries / search queries will hit elastic]

4. The ability to generate an access token for use against the GraphQL API via an Authorization header will be coming late December or early January. I'd love to know how you would like to use the API 🙂

❤️ thank you both very much