Secoda Community

Hey <@U0227NF4Z7Y> welcome! Feel free to take a look in our <https://trello.com/b/FXuyzLmX/secoda-roadmap|public roadmap> and vote on any features you think might make Secoda more useful for you and your team

Currently, you'd have to do one integration per database. How many databases do you have in Snowflake? We've never had anyone request an automated way of going through each database

22 currently, expected to grow - just seems like unnecessary steps/work to wire each new one up

adds some new challenges to self-service workflows as well - as folks either need to be secoda admins or send off requests for new databases :thinking_face:

You're correct, it's an issue. Like I said we've never had someone encounter the problem, but now that we have we'll look into what a crawler could look like so you just specify a warehouse and it will get all associated databases

Yes, there's a multi step process for the extractions. It's failing on the lineage extraction where we parse SQL queries. That's a bug on our end which should be caught as an exception and continue with the extraction.

You should be able to still see the tables in your catalog though. Can you confirm if you can see them

ah excellent - yes. Real bummer on lineage - what is the timeline on fixing that?

We are pushing a new version this evening so you can expect this to be fixed tomorrow morning. I'll circle back here when it's been fixed

image.png

Hi Chris, you can go to *Integrations &gt; Snowflake Integration &gt; Schema* and check which schemas you'd like to import (and uncheck the PR run schema). Hopefully this is what you're looking for

I actually set up the dbt connection first, without Snowflake.  This is pure dbt API info

Oh okay that's interesting. So short answer for dbt is we currently don't allow that. However, if you integrate Snowflake you'll have that option. You're the first to ask about limiting what comes in from dbt so we'll have to add that same functionality to the dbt integration

Ah.  If I were to remove the dbt data, add Snowflake, and then readd dbt, would it only pull in info related to existing Snowflake tables?

The connector automatically pulled in all environments data, including the PR and Dev environments, which is not useful for cataloging.

That's correct, you need to have both the dbt and Snowflake integration to pull in metadata from both sources. When you delete the dbt integration, all of the metadata is also deleted.

You're feedback is 100% correct, we aren't experienced users of dbt so adding an option to limit by env was an oversight. But not a difficult change to implement

I've tried to communicate how we run extractions on Looker in this <https://secoda.notion.site/Looker-Integration-5deded350dc74125a5fa4776260776b0|document>. If you have any suggestions about how we can improve it please let me know. Right now we try to connect the Looker dashboards to Snowflake tables by parsing queries run in  Snowflake but it's not as effective as we'd like it to be.

ooh, this looks helpful.  I’ll give it a read

digging through the Looker API docs and pulling sample data, they are not making your life easy at all.

When pulling explore LookML, I only get the actual SQL table name for the parent table for that explore, none of the tables joined.  We have our joins set up with the joined fields in our model lookml, so that’s the only thing that shows up in the API data.  It doesn’t surface the table references like it does for the other views.

If lookml view names perfectly mapped to DB table names, this wouldn’t be too bad, but they don’t, and I’m not sure how you’d get that info from the API.

Spectacles <https://www.spectacles.dev/> figures some of this out by using the API to run column header queries (limit, 0=1), which may be a decent approach for determining what the joins in an explore actually map to.

<https://github.com/spectacles-ci/spectacles/blob/master/spectacles/lookml.py>

Some more thoughts on how to stitch this together:
1. GET`/lookml_models`  will give you explore:name, explore:description, and explore:hidden (useful for a config option)
2. GET`/lookml_models/{lookml_model_name}/explores/{explore_name}`  will give you the views joined into the explore.  You can snag sql_table_name for the base view/table, and joins:name, joins:from (if its an aliased view), and TBD joins:sql_table_name.  Grab sets:name and sets:value to use in creating a query
3. use `create_query` to make a Looker API query, and get an ID for step 4
```{
  "model": {lookml_model_name},
  "view": {lookml_view_name}, from your explore LookML
  "fields": ["{lookml_field_name}"], from the set:value list
  "limit": "0" we just need a query ID
}```
1. (step 4) GET`/queries/{query_id}/run/{result_format}` if you format the results as SQL, you’ll get a clean `from schema.table` back from the API

I think that should allow you to scrape the Looker API for actual DB table names.  you may also need to pull in the DB connection info, in case someone is using multiple DBs that you want to tie through to

If you want to just tie a Look or Dashboard to tables, that should be simpler.

The look/dashboard has a query slug, which the Look and Dash API calls will give you.  You can use GET`/queries/slug/{slug}`  to give you the query ID, then you’re back to step 4 above, with a clean set of table refs to use

You can add these roles to a service account in GCP. I've included screenshots below

It should just say "Job Viewer". GCP has changed it since the documentation was created, I need to update it to reflect what's in GCP

I had issues with this setup too, I am doing my service account setup via Terraform. So I wanted to post here for info on the actual full role information that is required:
```roles/bigquery.metadataViewer        roles/bigquery.resourceViewer        roles/cloudjobdiscovery.jobsViewer
roles/logging.viewer        roles/logging.privateLogViewer        roles/bigquery.dataViewer        roles/bigquery.jobUser```
<@U01MHQR7DE2> it would be great to have this listed in the documentation for the BigQuery setup, so it is a lot clearer in what to setup for permissions.

Thanks for posting <@U03Q7DWHS75>, I’ve updated the BigQuery integration documentation to reflect this <https://docs.secoda.co/integrations/big-query?q=bigquery>

This was a feature that was temporarily removed, which is being added back in this week. We just did not remove the filter option.

Hey <@U02FR9DEVFG> right now it’s manual. I will send you a DM with instructions

<@U01MHQR7DE2> circling back here if you could DM me those instructions that would be great.

Screenshot 2021-11-04 at 21.15.51.png

Oh you're right, that's misleading. It's intended to show that you can add as many as you would like, but that's definitely confusing. Will change this! 

This may be better in the <#C02H3N7G26Q|> section of the slack group

fair enough, should have noticed that myself.  sorry!

You can do this by adding “collections” to fields. Collections are similar to tags and are quiet flexible. They also will allow you to see the group of fields in the matching collection. I know <@U028E7B1U31> has been using Secoda in this way. 

Yea, we ran into this problem too...when dbt rebuilt views Secoda was losing visibility at the schema level. So you are correct that the reference commands need to be at the schema level.

<@U031A287CR5> Not sure if this is helpful for your support ticket, but I had problems on the same step. (FYI <@U01MHQR7DE2>)

Thanks <@U02QJG595KM>. I'm essentially trying to setup Secoda with a catch-all command as I know a few other data discovery tools use the `snowflake` database to generate all documentation and lineage. Which is a lot simpler to setup
```GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ROLE;```

The feature isn’t available but it’s an interesting idea. Just to make sure I’m understanding correctly, you’re saying you’d like to impersonate someone who is a member of your workspace?

Exactly. The use case is to check on visibility of assets and objects as the user. It's helpful for troubleshooting.

Hey Justin, we chatted about this feature internally this morning. I think the ideal solution here is to select the user role and groups for the user you’d like to browse as. Doing full impersonation for a specific user does open up a ton of holes for security, whereas this solution doesn’t have the security concerns. Let me know what you think.

Such as:
User A is a Viewer and a Member of Groups X and Y

So I can do "View as `Viewer` , Member of `Groups: X and Y`"?

Our team is not on Enterprise although we have discussed upgrading!

We’ve come up with a better solution that doesn’t required `ACCESS_HISTORY` so it works for all Snowflake users.

Hey Richard, you can find our changelog here <https://secoda.notion.site/Secoda-Change-Log-a377d8df003d440eaf30e186807d62ca>

Hey Iqbal, happy to provide context. We pull in all of your Metabase dashboards, questions (charts), and queries and then try to extract the lineage information between Metabase and the database/datawarehouse tables.

The Data Engineering Podcast
The Analytics Engineering Podcast
Software Daily
The OSS Startup Podcast

Good tech podcasts but not necessarily data related but still highly recommend:
Stack Overflow Podcast
Non-technical

Oof definitely a bunch I can look for some highlights!

I think that <@U02C14D2KKJ> might be able to shed some light on this and how his team is using the product. The “ask a question” feature is used quite a bit at Partnerstack

i’ll send over the documented process we use

basically it’s like a ticketing system that’s integrated with slack, that we review every morning as a team, and asign the questions

that way it is manageable / doesn’t get overwhelming

<@U02C14D2KKJ> how does your team about SLA’s? Do you share what the expected response / solve time is for a ticket?

some questions we had hanging for several months tho lol

all depends on if the data is readily available or not

Data_Operations__Data_Request_Process.pdf

<@U0394G0FZGS> to me, this feels like a really clean experience for the “viewer”. They don’t even need to leave Slack + they get notified in Slack as soon as they question is being worked on or is complete

That's pretty clean, I like it.

<@U02C14D2KKJ> thanks for sharing. How big is your data team by the way?

we are getting useful metrics on our data team as well from this process

This might be worth exploring more in a case study <@U01PM6182AX> <@U032YJQ9TEU>? There’s some good metrics here that are great to see!

a few of us have been toying around with it. Big fan of Transform, but I’m still trying to figure out MetricFlow’s place in all this.

I don’t think/have seen anyone productionize just MetricFlow yet given that it’s just the CLI.

From the Secoda perspective, do you see yourself wanting to integrate with MetricFlow or the whole Transform suite?

Yes we do. Checkout this video of the integration we built with dbt metrics: <https://www.loom.com/share/7b57b7c0d60444d4bce3e3db8a201a28>

You read our minds <@U02739P8464>! We just pushed a fix that will only notify you on failure

hmm I don’t think there’s a good place I could point you for that. Really cool question. Maybe I can look  to see if we can make our internal workspace accessible to you?

+1 to this - would be cool to see what we can shoot for!

The original post this is a response to: <https://scientistemily.substack.com/p/scaling-data-teams?s=r|https://scientistemily.substack.com/p/scaling-data-teams?s=r>

Screen Shot 2022-06-02 at 2.10.17 PM.png

Screen Shot 2022-06-03 at 12.03.14 PM.png

The two that we give you as pre-built embeds are DBdiagram (recommended) and <https://www.diagrams.net/>. You can embed them by using the / command in a doc

Ask me about secoda - 3.png

Ooh good question. I absolutely love that we can have a shared source of truth for data. Our previous solution was “try to search for documents in Google Drive - I dare you.” It wasn’t working well for us, but Secoda is now our source of truth. Anyone who wants to know anything about our data resources has only one place to look for answers.

The general idea of "the Notion of data documentation" would be one, but a small thing that I'm finding really valuable is the ability to @ tag resources. For example when I'm writing a description of some table/view in our data warehouse, I will @ tag the primary source table from our production database. As more ELT/ETL integrations are put in place sure you'll be able to see this in lineage, but I like being able to quickly navigate; for example this is a description of a view in bigquery which primarily gets its data from our production postgres `users` table:

although I’m inherently biased on making the data team more productive, we chose and adore Secoda because the ux is focused on delivering understanding to the stakeholder/end-user. collections enriched with different types of resources help bridge the technical understanding in a friendly way. youre never more than 2-3 links away from finding out “what happens with data”

I believe that the existing integration of different functionalities in Secoda makes a difference. Analysts want to answer questions; the relevant tables &amp; columns to do so are only vehicles - the ability to link Q&amp;As to given resources provides way more context than mere table &amp; column descriptions could do. Plus it helps to draw analysts into the ecosystem of Secoda for data related Q&amp;As. Otherwise the discussions will happen in Slack (or any other messenger tool) and the tribal knowledge will most likely not make it into the data catalog.

On my side, I love the Dictionnary functionnality. All words and terms used on daily basis at Kard ("User", "Active User",  .... ) are defined and linked with a sql formula to compute or retrieve the information. When one of these word is used during a presentation, the link towards the secoda entry is included in the slide. We all aligned and speaking about the same thing.
There are also collections helping us grouping words and kpis by team but you said one thing :slightly_smiling_face:

Same with <@U03JFBY7CDU>. Currently we are focusing on dictionary implementation to make alignment between metrics definition. From the users, they love to use Slack integration so can easily to find the definition of a metrics when needed.

We are still in the process of getting Secoda fully implemented and rolled out but one of the biggest draws for us was the friendly user interface. Getting business users engaged will help us be successful so we are hoping this will help drive great adoption. :raised_hands:

Hey All
Thank you so much for all the great feedback! It has been a very busy couple of days so far with the traveling and workshops at our offsite, but the things highlighted here are very helpful and we’ll be sure to keep them top off mind when having the Secoda discussions with our colleagues!

We do have a number of BI integrations that report usage like Tableau and Looker. So we can also add it to the Mode integration since their API supports it.

In terms of our roadmap, we don’t have one that’s publicly available but I’m happy to give you more details if you’re interested.

Where would I find usage information for looker assets?

It should be available on <https://app.secoda.co/search> page. We’re expanding it to be on <https://app.secoda.co/catalog> and on the resource page itself

What is the data stack you’re using outside of apache nifi? Have you considered Airflow?

yes we do use airflow and dbt however nifi is something that is regularly used on a large scale. Hence was wondering if there are any plans to bring its integration.

We’ve never heard of it in the past, so this one is new for us. We’ll have to dig in to the docs, but currently it hasn’t been requested

Understood. Thanks for the clarification. Following is the link for its official documentation <https://nifi.apache.org/>