Secoda Community

image.png

Thanks for the feedback <@U02739P8464> we’ll get this fixed right away

Hi Kevin, thanks for this! Pushing an improvement to production now.

Hey Gerald, apologies about that. They aren’t removed from the database so we’ll get this fixed so that they are showing again. Look out for an update tonight or early tomorrow

Hi Gerald, thanks for your patience, I'm just finishing this up for you. I'll let you know once it's in production.

<@U02GZEAV18D>
1. The extended properties and references links are not wrapped anymore. Is the intention to keep the rows compact? Some of our columns will have more than 10 references, meaning that the user will need to scroll to the right quite a bit.  
2. Some of the extended properties for particular columns are shown duplicated

Hi <@U028E7B1U31> - I've made some changes to make the table component so that it can better handle large amounts of metadata. I think this will resolve your issue so please take a look. :slightly_smiling_face:

Yep the references and extended properties are wrapped now. Looks better. Would it be possible to left align them and sort them if it's not too much trouble. Thanks <@U02GZEAV18D>

Hey Julie, can you send the id of the dictionary term that’s causing issues. It should be the URL when you go to it

The first thing that comes to mind is that the search indices did not update properly. You can trigger a reindex for dictionary terms by modifying the title or description of the term.

Great! We’re doing an update to the search index logic to avoid this issue

Following up - this is happening fairly frequently still. Some terms still don't appear after changing the name to re-index them

Thanks for letting me know Julie, I’ll take another look and let you know what I find. Is it just with terms or is it with all resources?

Thanks! It's primarily been noticed with dictionary terms, but I'll see if it is happening with other resources

Screen Shot 2022-01-21 at 2.55.00 pm.png

Hey <@U028E7B1U31> this looks like a bug on our end, we’ll work on a fix today and let you know when it has been completed.

Hey <@U028E7B1U31> this has been fixed now. What are the endpoints you are most interested in? That way we can create some code samples for you. cc <@U02N4NXTTJR> 

The samples in the API doco are sufficient for now. I'll let you know if I need more

<@U01MHQR7DE2> We want to manage our users in google instead. Do you have any google integration?

I'm sure I follow what you mean <@U028E7B1U31> . Do you mean Google SSO or something else?

You can login with Google on the login page. If you’ve used email you can still login via Google and it will link the account you already have.

Sorry not an expert on this matter. We want to manage our users in our google suite instead of creating the users directly in Secoda

For Secoda you can think of the Google SSO as a way for Secoda to know which users are part of your Google suite. For example:
• Say there’s Employee A who is part of your Google Suite
• When they log in with Google SSO Secoda is linking to that user’s details
• When Employee A is removed from your Google Suite they will no longer be able to log into Secoda.
In order for that user to work properly in Secoda, we make a copy of their details in our database along with supplemental properties that our app requires. That’s why when you remove them from the Google Suite you can still see them in Secoda, but are not able to log in. Hopefully that provides some clarity.

That's good info. The part which I'm missing is can we add the users in our Google Suite and it automatically create a user in Secoda? Or do we always need to send an invitation from Secoda?

It depends, if you want them to be a certain role you’ll need to invite them. Otherwise, if they use Google SSO to login and their email domain matches with yours then they can automatically join the workspace as a viewer. From a security perspective we require emails to be verified so someone can’t just put in &lt;name&gt;@courseloop.com and join your organization, they actually have to be part of your Google Suite.

The only thing you need to do is add your domain to *Settings &gt; Workspace &gt; Allowed Domains* and add <http://courseloop.com|courseloop.com>

To add here, looks like the S3 permissions are being denied

<@U02GZEAV18D> can we look into this this afternoon if you have a chance. Thanks!

Hi Vishnu, they should be back, it seems they were in base64 image type, which may have caused some issues. Should be all good now!

It appears to be doing this w/ any newly created question

Hey Alex, thanks for letting us know about this. I’ve pushed a fix that will fix those questions which are not showing up.

First thing tomorrow morning we’ll push a fix that resolves this for all new questions.

<@U02PBHLUDU6> do you know if these questions were submitted via the Slack app or on the actual Secoda application?

Okay thanks, we’ll be pushing a permanent fix tomorrow morning to this issue

Hey Benoit, thanks for pointing this out. The reason why this is happening is because the catalog page is retrieving data from Elasticsearch whereas the actual resource is being retrieved from our database. The update that’s occurring to keep elasticsearch in sync with the database is why you’re seeing a discrepency.

The main thing though, is that the items that you’ve pointed out are indeed bugs and we’ll be fixing them early this week. I’ll keep you updated in this thread as we make progress on them.

Hey <@U02HEA5UD7F> this issue should be resolved now. If you find it happens again please let us know.

Hello, me again, I still have  discrepancies for at least 3 tables, including `staging.stg_prod_checkout_history` for example

Are the discrepancies on the published/draft field? If that’s the case I’d toggle between them and then if there’s still a discrepancy let me know. The bug fix didn’t retroactively fix things with discrepancies.

Yes it is. It might be that it’s a remnant of the old bug

This was an oversight on our part, as you cannot edit the title. We’ll also push a fix for this today since it is important functionality.

<@U02PBHLUDU6> thanks for reporting, we’ll fix this issue today. I’ll update you when it’s resolved.

Capture d’écran 2022-01-31 à 18.17.22.png

Hey <@U02K8BY6RCK> we’re pushing a hotfix for the blank screen cc <@U02GZEAV18D>

<@U02K8BY6RCK> the blank screen should be resolved. Please let me know.

Hey Alex, for Docs you have to explicitly set who can edit the Doc. Being Public means it’s viewable by anyone. In the screenshot you can see that I own the Doc and Etai can edit it

Ah, I didn't understand that adding someone as a viewer from that box would enable me to add them as an editor. Thanks!

<@U01MHQR7DE2> Is there any way to override this so admins can see everything? It feel tedious to add all of my team members to each doc I create

Or at least be able to add a group to the doc

Hey Benoit, which integration type is the profiling failing for?

Also there currently isn’t a way to profile all tables. The reason why we chose this is because it’s fairly compute intensive and we didn’t want to increase people’s compute bills. However, we can add a button in the data settings to run the profiler on all columns and just warn people about the query it’ll be running against all the columns.

it’s failing for any table really. It worked fine the first time I tried the feature but since then I have been unable to use it on another table

Ah that explains it, we’re trying to profile with the dialect “dbt” instead of bigquery. I’ll push out a quick fix for this.

Hey Kiara, this is a bug on our end but there is a workaround if you don’t want to wait on us to fix the issue.

To get the session id you can follow these instructions:
1. Sign out of Metabase
2. Open up the Developer console (Right click and click “Inspect”)
3. Go the the “Network” tab and check the box for “Preserve log”
4. Login to Metabase
5. Go back to the Developer console and click the `google_auth` request. Go to the “Response” tab and copy the `id`
6. Go to Secoda and add the Metabase integration and instead of a password use a session token
Here’s a video demonstration as well <https://www.loom.com/share/fd3f422022b041a0a757045064a2a2d6>

it doesn’t seem to be working. I can wait for a fix - is there an estimated completion date for a fix on this?

I will get it out so it’s ready on Monday!

<@U02739P8464> do you have any insight on this? I know you’ve played around with the permissions in Snowflake more than I have. If not I can take a look into this. Just wanted to check first.

Thanks Andrew! In a nut shell, I just want to know whether i need to explicitly grant access to every database we have or if there's just a simple catch all command I can run. We also use blue-green deploy so I'm not sure how the lineage might work in this case, but ideally we don't need to grant access to the staging db in a blue-green deploy

I’ll look into this this morning for you <@U031A287CR5>

<@U031A287CR5> based on the research it looks like the query you’ve pasted above would do the trick. You just have to associate `ROLE` with the user you connect to Secoda with.

gotcha, I'll try that now and let you know if I run into any issues! Would make it a lot simpler for future Snowflake users to onbaord as well IMO!

<@U01MHQR7DE2> so it looks like it's only pulling in the schemas within the default `SNOWFLAKE` database. i.e. `account_usage` `organization_usage` ,etc. And not actually extracting the production databases that can be derived from this DB

<@U031A287CR5> let me make a quick update to our Snowflake queries to make it so that works properly. It shouldn’t take too long and you’re right that it will make things easier for future users. I’ll let you know when it’s been completed. Hopefully in the next hour

Thanks! For now I've explicitly created a Snowflake integration for each database we plan to sync

We have a bunch of interim tables that Airbyte uses to build source tables, is there a way to deselect and not sync all of these tables?

Yes, if you go to the Integration in Secoda and then the Schema tab you can select which schemas and associated tables come into Secoda.

If you want we can update that UI to also include tables?

Yes that would be great - we have many `_scd` suffixed tables that Airbyte uses for incremental syncs and a lot of `_airbyte` prefixed tables (that I can't seem to search for in the search bar) that shouldnt be included. In the image below I've searched for `_airbyte_` it's saying there are 722 results but only shows 2 pages of them, none of which are the airbyte prefixed tables

For the screenshot, are those the results after searching `_airbyte`? It would be good to get a full screenshot if possible to get a better understanding of what’s going on.

Yup, those are the results after entering `_airbyte` in the searchbar

That’s helpful, should be a quick fix on our end. I’ll let you know when it’s been fixed.

Awesome, just adding on, similarly if I search for `core.` it would display 143 results on the right but only 10ish in the list with no pages to select

I’ve also forwarded this to our engineering team. A solution while you’re waiting on the fix is to select the “Tables” option on the right side. That should show all the tables.

The “All” bug should be fixed now. Let me know if you continue to experience the issue with it.

Hey Andrew is this available yet? I've got probably around 600 tables that are temp tables I don't want to include in any of the searches or catalogs
<https://secodacommunity.slack.com/archives/C01PT4WP0TE/p1644257802323409?thread_ts=1643998298.824089&amp;cid=C01PT4WP0TE>

It’s not available yet, some additional work came up today. But I still have it on the top of my todo list so hoping to get to it tomorrow morning.

Thanks for bringing this back up <@U02QJG595KM>, I’ve added this to our backlog and slotted it in for the upcoming sprint cycle next week. <https://linear.app/secoda/issue/SEC-276/snowflake-lineage-parsing>

Hey <@U02QJG595KM> <@U02UNRGSVNC> we’ve updated our Snowflake integration with improved SQL parsing to populate lineage courtesy of <@U03113A05T6> who has been working on improving lineage overall in Secoda.

If you run an extraction, you should see much more lineage information on your Snowflake tables now!

Good question. Collection can hold any resource but documents (coming soon). The way to add them now is by going to the resource and adding them to the collection. In the near future we’ll also be adding a widget on the collection page to add them to the collection from there.

Thanks Andrew, is there a way to mass apply a list of resources to a collection? I'm searching for a specific schema and I'd want all of the tables in that schema to be in a specific collection

Unfortunately not at the moment. You’re bringing up all the items that are very high priority on our backlog. In February we’ll be adding support for bulk editing on the catalog page bulk editing so you can add collections to multiple tables at once. cc <@U02GZEAV18D>

Also for the dbt integration as well, how long should I expect the delay between integration and Secoda populating my underlying tables with my dbt docs

For the collections question, it looks like there a bug where it won’t show up until you refresh. In the meantime while we’re fixing this bug if you refresh it will show up as a workaround.

For the dbt question, it should be done immediately once the extraction is completed. If it’s not let us know.

Awesome, yeah it hasn't shown up, for the dbt docs.

Even after the extraction completed? And for both the table and column descriptions?

No need to apologize about the questions, all the feedback is really helpful! At the moment we don’t allow linking via the definition, but you can link the term via the long form documentation of a dictionary term.

It will help us to parse lineage information between Looker views and Snowflake tables.

Are there any plans to set this up with a Looker deploy key instead of uploading the repo?

Could you elaborate on the Looker deploy key. How do you imagine we could use it to replace this current method?

The reason why we chose this method is because to integrate with a GitHub repo you need to provide full access (read, write, delete) just to look at the file contents. So we thought most people would not want to give us that level of access to their repos.

Select Star has an option to add a deploy key into the Looker repo without write access. That's how they parse lineage information, might be worth looking into!

That’s helpful feedback, thanks Kyle. We will definitely look into that method!

I didn't upload the LookML repo but it looks like Secoda is still able to parse lineage? Is the Repo for lineage between Looker and our database?

It’s able to parse lineage within Looker, so between views, explores, and dashboards. You’re correct that the repo is for lineage between looker and the database.

Hey Kyle, we recently did a migration for notifications and overlooked this one. So we're working on pushing a fix for it. 

Hello Kyle, we just pushed an update to fix this. Sorry for the inconvenience.

Hey Kevin, thanks for reporting this issue. I can reproduce the issue with the catalog page but can’t reproduce the issue on the search page. Are you clicking next while “All” is toggled?

I’ve created a ticket for this issue Kevin. You can subscribe to the ticket to receive updates once we fix it. <https://linear.app/secoda/issue/SEC-252/pagination-broken>

We’ve been able to reproduce the issue on our end, just need to fix it now

Hey Kyle, this is UX issue on our end. We first run an extraction to pull the environments/jobs. Once you select the environments/jobs you want us to parse the `manifest.json` you can run an extraction by clicking “Run Extraction”

Nothing yet, about 40 mins since extraction success

It looks like there was something extracted based on our logs, is it still blank <@U031A287CR5>?

<@U01MHQR7DE2> some of the data came through but not as I expected. It's populating a database called `STAGING_DB` that we use for blue-green deployments, when it should've actually filled out the fields from `PROD_DB` . Is ther a way to point the dbt extract to a different database?

Hi Iqbal, this appears to be a bug on our end. I’ve added a ticket so that you can follow progress on it. <https://linear.app/secoda/issue/SEC-266/slackbot-search-not-working>

Ah okay. Please let me know as well if it has been fixed. Thanks. <@U01MHQR7DE2>

Hi <@U01MHQR7DE2>, want to know, is there any update on this bug?

Hi <@U032EFBHWBD> we’re pushing a fix for this issue tonight, I’ll let you know when it’s live.

Cool. Looking forward on it <@U01MHQR7DE2>!

Hey <@U032EFBHWBD>, this has been fixed. Let me know if it works now for you!

Hi <@U01MHQR7DE2>! It's working! Thanks!
I have one question, may I know if the resource that we can browse here for all resource or only table? Because I try to search a metric that I already saved in Dictionary but it's not appears in the top list even I use exact name of it.

It should be searching across all resources, let me look into why that’s not the case for you Iqbal.

Cool. In the specific, what I expected is we will get the most relevant one in the top of Slack chat. But turn out I need to click see all and go to app to get the relevant resource.

Yeah that makes sense, so it’s likely a bug on our end because it should be returning exactly what you see in <https://app.secoda.co>

Oh I see.. If you need an example case I can share.

&gt; Yeah that makes sense, so it’s likely a bug on our end because it should be returning exactly what you see in <https://app.secoda.co>
By the way just want to make sure, on this case, it should be on the issue that will be fixed right? <@U01MHQR7DE2>

It’s actually a separate issue, so I’ve created a different ticket here <https://linear.app/secoda/issue/SEC-332/slack-search-doesnt-search-all-resources>

Hi <@U01MHQR7DE2>! Seems the first bug happened again in my side. Can you check please?

For update, it works again. Idk what the issue to make previous error. Thanks

Hi Alex, did the text come back after you refreshed or is it still gone even after the refresh?

Still gone unfortunately. It's not _that_ big of a deal in this specific case but curious what caused this considering I can't just revert to an earlier version

Hi Alex, I dm'd you the recovered response (I believe). It seems there may have been a migration issue with that specific response. I'm working on a fix now.

Fix is being deployed to production. The issue was editing and saving a reply in quick succession. Issue is resolved going forward.

Correction: adding tags from this screen does not appear to work at all I think? The collection tag still does not apply to the table event after adding the collection manually

Hey Alex, you’re correct. I’ve added the issue which we’ll fix today. You can follow the issue by subscribing to the link <https://linear.app/secoda/issue/SEC-302/collections-on-table-page-broken>

Hey <@U02LVDYD1AS> this issue has been resolved now

Capture d’écran 2022-02-16 à 17.55.11.png

Hi Marin, thanks for reporting this. It’s a bug on our end. Can you refresh your page and see if it’s been fixed now.

It works! Amazing response time :slightly_smiling_face:

I’ve tried many variations of login credentials -
1. with and without the `SECODA_ROLE`  
2. two different warehouses (`compute_wh` and my `AIRBYTE_WAREHOUSE`)
3. different account formats
    a. `ab12345.us-east-2`
    b. `ab12345.us-east-2.aws`

Hey Sawyer, do you know if you ran the queries specified in this documentation while being `ACCOUNTADMIN` <https://help.secoda.co/en/articles/5013624-snowflake>?

Also you need to whitelist our IP address `35.175.75.15/32`

Yup.  executed the instructions in Secoda’s docs line by line

should it be
    a. `ab12345.us-east-2`
    b. `ab12345.us-east-2.aws`


Can you send the URL of your Snowflake instance in a DM pls. I’ll be able to answer with that info

the same happens with deselecting databases, they've come back!

Did you click the “Save” button after selecting/unselecting them?

The schemas still appear in my "google search" after deselecting them, should i just reset the connection?

When you say they appear in the “google search” do you mean the schema tables or the schema filter options?

If it’s the latter, I wouldn’t reset the connection. It’s a bug on our end.

It's the former, the schema tables still appear when I search for a table using the searcxh bar

Can you try again now and let me know if it’s fixed. I’ve run a manual search index update on your workspace.

Did you check if that schema is unchecked in Snowflake _and_ dbt, it might be checked in one of them.

Great point! It was selected in dbt, thanks Andrew

This was something we haven’t invested in given the lack of engagement on the jobs pages, but now that people are using it, it makes sense to pull more metadata. Besides the runs and links, is there anything else you’d like to see?

I’ve added it to our backlog <https://linear.app/secoda/issue/SEC-310/additional-metadata-for-dbt-jobs>

Hi <@U01MHQR7DE2>, I've never used Linear before. Is this a way for us to track Secoda's work on particular features? When using the link, it just brings to a page that looks like I'll be creating my own Linear App workspace

My interpretation was the you could see the “Subscribe” button in Slack and if you subscribe you’ll be notified about changes. Can you not see that button?

This is the second time this has happened. The first happened on a different account earlier this week. This makes me hesitant to put any lengthy analyses in Secoda. Might suggest a use case for some form of version control on questions? Though I can see that being a PITA too

Hey Alex, do you know if there was any action that you performed that caused the answer disappear? The version control point makes sense as a fail safe mechanism, although it shouldn’t be happening in the first place on our end. I’ve added a bug to resolve this issue. <https://linear.app/secoda/issue/SEC-312/question-answers-are-disappearing>

I don't _think_ so. I was editing an answer to the question yesterday. I don't believe I would have deleted it nor do I believe anybody else on my team might have either (though it's possible).

Can you dm me the question title and the time you edited the answer?

Plot twist - I have the old tab I was editing it still open. So I can just copy / paste. Still odd that I can't see the edits in new windows though