An FYI in case anyone else runs into issues with setting up analytics in a datahub deployment without the standard helm charts using AWS MSK. If you notice every third click in the datahub UI hanging, you can look into the networking and check if the

trace

call is hanging(should timeout in 60 seconds and you can make one more call). If it is, you may also notice the following stack trace if you add a callback to the event tracking kafka emit:

org.apache.kafka.common.errors.TimeoutException: Topic DataHubUsageEvent_v1 not present in metadata after 60000 ms.

To fix this, the following configs need to be set to talk to kafka correctly: https://github.com/datahub-project/datahub/blob/34b36c0fe17f6ed6195ba5a0b57f41853fc60532/datahub-frontend/conf/application.conf#L158 Will update the docs tomorrow to add this info, but hopefully someone will find this useful if doing a search through slack one day 🙂

thank you!!!

After doing this, you may notice a bunch of 500 errors in your analytics page. To resolve these 500 errors, you'll need to follow the steps highlighted here: https://datahubspace.slack.com/archives/C029A3M079U/p1637601894470900?thread_ts=1637363019.462500&cid=C029A3M079U To delete all the indicies, the following deletions need be run(with authentication of choice)

curl -XDELETE https://$ELASTICSEARCH_HOST/datahub_usage_event-000001
curl -XDELETE https://$ELASTICSEARCH_HOST/_template/datahub_usage_event_index_template
curl -XDELETE https://$ELASTICSEARCH_HOST/_opendistro/_ism/policies/datahub_usage_event_policy

followed by running the create-indicies script then restart the service(NOTE: at this point as long as no one is navigating to the service, you don't need to tear down the service. This is a huge problem in our company as tearing down a service is a pain, so instead just re-direct traffic if you can for a bit or update this when users may not be using the service) Now analytics should be working(hopefully) 🥳