wooden-spring-14915
06/24/2022, 6:19 PMmodern-account-72538
06/24/2022, 6:19 PMusers
table I may expand the Description column so that it's very wide, but when I'm viewing a purchases
table I may expand the Profiler/Distributions column so that it's wide. When I come back to view these tables in future it would make my user experience 2x better if those column widths were the same as they were last time I changed them. Same situation if I've applied filters, sorted, etc.
• My #1 request, something I haven't seen anywhere else ("column lineage" gets somewhat close sometimes) and I that I think would have a huge positive impact. Let's say I've got a data mart in our warehouse named users
which pulls in data from a dozen different sources; it has aggregations and whatnot built in dbt, data joined from our production db orders table and Intercom, etc. Where do I document all of these columns?
◦ Take an example of a column account_created
that orginates from our production postgres db users
table...obviously, a logical place to document this column would be there: Catalog -> Postgres -> db -> main -> users. So I head there, write a nice long description of what exactly the account_created
field is, great. What's the problem? The problem is that our BI developer builds his visualizations from our BigQuery marts, not our prod db. So they look at Catalog -> BigQuery -> prod -> users and see no description of what this account_created
field is.
◦ A good long-term solution to this is column lineage. If we happen to use an ETL tool that Secoda integrates with, and do our warehouse transformations using a tool like dbt that Secoda integrates with, eventually you'll be able to show column lineage. But that's a long term, difficult problem to solve.
◦ A quicker, easier, and better solution in some use cases would be to allow me to do the following:
1. Write the column description once where it should live: Catalog -> Postgres -> db -> main -> users
2. Then allow me to @ tag that description anywhere throughout Secoda. In this example @ tag it in Catalog -> BigQuery -> prod -> users, in the Description column for account_created
3. Now when our BI dev is looking at the BigQuery users
table documentation, in the account_created
Description column they'll see @Postgres - db - main - users - account_created. Ideally hovering over this would show the description, and clicking would open a new browser tab to the prod db users table
◦ Even once column lineage is built, this @ tagging ability would be highly used. I will @ tag the description of some column, a dictionary term, and a collection, all inside the description of some column of some data mart, so that end users of that mart can easily find all the info they need about itelegant-house-93198
modern-account-72538
06/27/2022, 6:58 PMthe root issue you’re experiencing is that you have a description that you’d like to have shared across similar tables/columns right?@elegant-house-93198 yes, that's most of the use case
modern-account-72538
06/28/2022, 3:03 PMage_group
for users, sprinkled throughout various tables/marts. If I'm looking at the main users
table documentation I get to see the description of this field, but if I'm looking at any other table documentation e.g. users_purchaseanalytics
then of course it's not availableelegant-house-93198
modern-account-72538
06/28/2022, 6:34 PMelegant-house-93198