Hi everyone, apologies for the beginners’ question...
# getting-startedb
brief-cricket-98290
09/27/2021, 10:31 AMHi everyone, apologies for the beginners’ questions as I am just starting my data governance journey:
• Is there a Vertica plugin for metadata ingestion?
• If not, what would be a good starting point for making a custom ingestion plugin? This documentation page I suppose?
• Is it possible to ingest Kafka metadata about consumers, namely consumer groups and list of the consumers?
Thanks!
l
loud-island-88694
09/27/2021, 3:26 PM
Hello @brief-cricket-98290 - welcome to the community! We currently don't have a vertica plugin and would love a contribution from you. That is indeed the correct documentation page - we're happy to help you along as you make progress.
loud-island-88694
09/27/2021, 3:30 PM
As for Kafka consumers - that is an interesting use-case. Are these long running Kafka consumers? I think you can model this using time-series aspects i.e emit consumer-id, consumer-group, count of events over a window periodically using the emitter API. Is this more of a monitoring use-case for you?
loud-island-88694
09/27/2021, 3:30 PM
cc @helpful-optician-78938 who can help more
h
helpful-optician-78938
09/27/2021, 5:08 PMHi @brief-cricket-98290, please take a look at https://github.com/linkedin/datahub/blob/master/docs/modeling/metadata-model.md#timeseries-aspects for the documentation on the time-series aspects. I will be happy to help more if you have specific questions.
c
cool-afternoon-48566
09/30/2021, 9:50 AMHello @helpful-optician-78938, @loud-island-88694 Can you share more on this. I am still not sure how can we model all this in Acryl SAAS, I guess we can't as of now. Do you suggest something else like properties. Etc.