Hi - this is not really a feature request but not ...
# feature-requests
m
Hi - this is not really a feature request but not too sure in which other channel to slot it. I got 2 questions: 1. We are struggling in the team to use Datahub to represent regional differences. In particular, we have some datasets that exists only in some regions and not others (for example in US and not in EU). On the other hand, the same dataset can exist in multiple regions at the same time (EU and US) --> it would make sense to have 2 representations of the dataset because there may be slight differences in the schemas + also it would make sense to have a different 'Validation" tab for each of them as the tests would run independently for each dataset. To represent this, we were thinking of using the region in the "Environment' Part of the URN. So for example 'PROD-EU' or 'PROD-US'. However, when in the recipe file we try to put that we receive an error saying that only a set of values is allowed, like
['PROD', 'DEV', 'TEST' etc etc]
. Is there any reason why the list is limited to those? Are those used internally by Datahub in any place? Could we not freely specify values there, so that we can also include the region? 2. We enabled DBT tests for our project, and we're so happy to see them appear in the Validation tab!! The only thing is, we use quite heavily
dbt-expectations
link - so the names of the test in the UI come up as the SQL as opposed to the nicer test name. The tests here are actually named almost identically to the
great-expectations
tests, would there be any way to re-use the naming from there?
l
for 1, please use DataPlatformInstance
for 2, will let @mammoth-bear-12532 comment
m
@millions-notebook-72121: for 2. the tests will appear correctly if you move to the latest quickstart images for server... and when the next server release comes out. For example on https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:bigquery,calm[…]jaffle_shop.orders,PROD)/Validation?is_lineage_mode=false you will see this dbt-expectations test as shown in the screenshot.
there is also a new PR that I'm reviewing that is trying to make this even better -> https://github.com/datahub-project/datahub/pull/5214
For context, this was the PR that fixed the UI issues (https://github.com/datahub-project/datahub/pull/5155)
m
wow that's amazing, thank you @mammoth-bear-12532! @loud-island-88694 I see let me explore the DataPlatformInstance.