Secoda Community

Hi Andre, thanks for the feedback, I'll create a ticket for this so we can explore further.

Would you mind elaborating on your second point about how pressing enter only picks up changes in your current field?

I edit database to: x (I leave it without pressing enter)
then I edit title to: y and press enter in the title field.

Only the title = y filter is then applied.

This is the one that accidentally got deleted: <https://app.secoda.co/dictionary/cc1ccca9f5db42fe8d999e8074dfb274>

We've restore it for you <@U04C7Q1H022>. Let me know if you can see it again

Untitled.txt

Hi Richard, going to create a ticket for this and get back to you on what the issue is.

Hey <@U04DHB29WBB> in our logs we see that the latest extractions were passing successfully and pulling tables into Secoda. If you have any other issues please let us know

Hello, I'm trying to configure a connection to SQL server on an azure vm but getting this error. I have you whitelisted on vm

<@U02739P8464> we're investigating the issue now, do you have schema name that's been effected?

I just manually executed the command where you all pull the columns etc under our Secoda account, and returned columns from the missing schemas, so I dont think its a snowflake permissions issue

Ok thanks for the info Kevin, we're investigating and will update you shortly

Are you able to DM me and <@U02N4NXTTJR> the list of schemas that you're seeing from the query <@U02739P8464>

Thanks <@U02QJG595KM> I've created a ticket for this issue

I should probably note here that initially I was pointing the validations store to the data_docs validations, which contained html pages with validation results. The tests view in Secoda didn't change though when I made the change to point to the json results. The view updated with the new run time, so the extraction was successful. I'm a little curious as to whether the tool is looking for html or json results, or if it's configured to handle both

Hi <@U04DU2UAZBJ>, we pull the validation JSON files from the *Validations Store* and parse those files to get results. We then determine the lineage by looking at the table/schema/database names specified in the file along with the datasource mapping :smile:

Any idea why those validations might be getting marked as failures incorrectly? I'm happy to provide the JSON results or anything that might be helpful

We just released a copy / paste functionality in Secoda to bring stuff from Notion to Secoda. Bringing stuff from Secoda would be a feature on Notions end <@U03JFBY7CDU>

Hey Ryan, are the permissions enabled for Secoda to index the other schemas? This is usually a good place to double check

Hi Etai, I like to ask for clarifications on what this mean?
&gt; permissions enabled for Secoda to index the other schemas?
I've checked the service account roles and it does have the proper roles/permissions. I've also specifically looked if the service account has the permissions to view a table in a dataset (thru BigQuery UI) that I can't see on Secoda after extraction (but which I can still see on the Schema page on Integration)

Hey <@U0454PR5754> would you be able to DM me an example schema(dataset) that you're not seeing any tables from on Secoda. Thanks!

Screen Shot 2022-12-21 at 12.47.17 PM.png

One common place some stumbling happens is in selecting the source in the top right hand corner of the query block. Did you select Snowflake from the dropdown?

<@U01MHQR7DE2> you may want to take a look here. Could be a permissions issue

Hey <@U04FMGG0L1M> there's an issue on our end showing when there's an error with the query. In the meantime I'd recommend adding the database and schema to the table reference on the query, i.e,
```SELECT * from PC_FIVETRAN_DB.MARIA_PROD_DEBTPAYPRO.CONTACTS
LIMIT 10;```

Hey Amit, we’ll have a look at the issue today and update you when we’ve pushed a fix. Thanks for flagging it

Hey Amit we weren’t able to resolve the issue yesterday but will be continuing to investigate the problem today

Hi <@U02739P8464> - we don't currently have resource level exports prioritized for this sprint cycle, but we can take a look at prioritizing it in the future. When were you hoping to have this feature live?

What’s the goal with the feature <@U02739P8464>? Would be helpful to understand how you want to use this

Use case here is a hackday we will be sponsoring, users involved will not be given secoda accounts as its a one off. We need to be able to provide documentation to participants.

Aside from that, there are use cases where non frequent users or leadership might want a quick excel file for a table instead of having to sign up for an account etc.

I think so, it would be great if the workbook also had a sheet for the documentation page as well

On the notifications settings, you can turn the individual notifications off. These must be on for folks in your workspace

Thanks Etai, so Admin can't change these? Are they enabled by default?

You can also uncheck the *Send Announcement* option when publishing a version of your workspace

Ah that's probably what I want. Thanks <@U01MHQR7DE2>

These are the two IP addresses <@U04HBMDJVJB> `35.175.75.15/32 3.122.13.89/32`

Hey <@U03M9F8GP8X> are you able to share the link you have saved that's providing that error, I'll be able to see what the issue is with that info

Hey Ben, the search should be filtering out draft resources. We've logged this bug ticket and should have it resolved quickly. When we do I'll let you know.

Hey Ben - just wanted to make sure that you knew this bug was resolved with the most recent update!

Hi <@U0454PR5754>, I'll create a ticket and we'll have a support engineer help troubleshoot this week.

Hi <@U0454PR5754>! We do have validations/tests on the Table page, are you seeing the GE validations there? In the screenshot I sent, you should be seeing a Tests tab if there are Tests available.

We don't currently display validations on the Catalog page, but we've added it as a feature request.

Hi Sarah, I am not seeing it. I was able to map the datasource to our BigQuery integration, if that info helps

Also, might be a unrelated problem, I've been fiddling around my path directory and see if that helps make the validations show after extraction. It seems that my jobs are stuck on queue

Thanks for the updates <@U0454PR5754> - we'll take a look and get back to you

Hi <@U0454PR5754>, we pushed some fixes to the Great Expectations integration 4 days ago, just wanted to check if you had a chance to re-run the integration and see if the issue has been resolved?

If you find that you still cannot find the validations, please let me know and I’d be happy to help :smile:

Hi <@U03113A05T6>, I'm still encountering the issue after re-mapping the integration and re-running the extraction. I do wonder if it has something to do with case sensitivity? datasets in Secoda shows in lowercase but in BigQuery, the dataset is in uppercase

<@U0454PR5754> <@U03113A05T6> this is likely the issue, <@U03113A05T6> we'll have to make sure to de-normalize the data

<@U0454PR5754> do you have an example BigQuery table since the one you linked is Postgres

Hi <@U01MHQR7DE2>, the integration is mapped to BigQuery. I am trying to map it into a sample dataset and table that I validated using Great Expectations.

Thanks for the info <@U0454PR5754> are you able to share the Secoda URL of that BigQuery table

Hi Secoda team, are there any updates to this?

Hi <@U0454PR5754> if you're able to provide the link to the BQ table in Secoda that'd be helpful for debugging. Thanks!

Hey <@U0454PR5754> <@U03113A05T6> will be investigating this week. We'll keep you posted

Hey <@U04HBMDJVJB> two thoughts here:
1. We have an open ticket to support parsing the `meta` property to automatically add owners, collections, etc
2. In your dbt cloud integration preferences, there's an option to import tags from dbt which will bring them in and add them to their respective models

It happens the same to me but in Collections pop-up. when starting writing the description is get stcuk and I need to reload the webpage

Hey <@U03C85WHYSX> <@U04HBMDJVJB> we're looking into this issue this morning and have logged it as an urgent issue

Hi Gauthier, I am going to work on this issue this morning

Hey <@U04HBMDJVJB> would you be able to share a screenshot of what these two items look like in Fivetran. It would help give me an idea of how to modify the integration to allow you to distinguish what is a destination

Sure. Fivetran allows setting different ‘destinations’ aka landing databases for the ‘same’ connector. It does not mean the same conector writes in two places but there are two ‘instances’ of this connector created down each ‘destination’. so destination should be a key metadata to attach to the jobs. Here a screenshot of my sandbox  with some notes

For sure it is not critical at all, at least for me

Ok, thanks for the context Daniel. I'll create a ticket to update the Fivetran integration to reflect this behaviour

FYI I noted that `requiredGroup` parameter in the job URL in fact identifies the destination but in Fivetran fashion, using some odd, random combination of two words as id to avoid duplications. `<https://fivetran.com/dashboard/connectors/>&lt;a given connector&gt;/status?requiredGroup=&lt;some odd two words combination&gt;`
Objective would be to get the mapping between this `requiredGroup`  parameter and the true,  user-friendly ‘destination name’

Hey <@U02739P8464> we don't currently support that capability, but what I'd recommend is to create a collection, bulk add Tableau items to that collection and then set the permissions on the collection. It's much faster to bulk add resource to a collection on the <https://app.secoda.co/catalog|catalog page>. Let me know if that works as a temporary workaround

<@U04JPJM9FAN> I believe that Rearc is using an isolated instance, so you'd want to replace the URL <https://api.secoda.co/> with your isolated instance's URL. If that's not the case let me know and I can investigate further

<@U01MHQR7DE2> Thanks. If I hit `https://&lt;custom&gt;.<http://secoda.co|secoda.co>` for the web console, is the api at `/api` under that? Or `&lt;custom&gt;-<http://api.secoda.co|api.secoda.co>`? Or is there somewhere I can look up what the api prefix should be for us?

<@U02GZEAV18D> do you know the answer to David's question, I'm fairly certain it should be `https://&lt;custom&gt;.secoda/co/api/v1` but I'll let Carter confirm

Hi David. Your python code is syntactically correct. However, since Rearc is on a managed instance, you will have to hit the endpoint <https://rearc.secoda.co/api/v1/>. For example:

```requests.get('<https://rearc.secoda.co/api/v1/table/tables/>', headers={'Authorization': 'Bearer &lt;token&gt;'})```


image.png

Hey <@U04C7Q1H022> it seems like your company's email server is blocking the invitation emails

Hey <@U04AD9BUCM7> how'd you trigger this exception? Was it a search or a question?

I typed /secoda purchases
Seems like /secoda is broken

Thanks for that context, we've created a ticket to address this issue <https://linear.app/secoda/issue/SEC-2120/slack-search-and-question-submission-broken>

Hey <@U04AD9BUCM7> this issue is happening because someone disconnected the Slack application from your workspace. Once you re-connect it will start working again.

<@U01MHQR7DE2> ah. I checked in with our IT team and it's blocked because it is unapproved by Slack, they won't override.

I can submit it to Slack now to get approved, hopefully the turnaround time isn't long. I'll keep you updated

Hey <@U04LR81AUG3> it depends on the database/datawarehouse dbt is connected to. Since dbt doesn't generate column level lineage, we're parsing the queries on the data warehouse to determine column lineage

what I want is for the documentation to be inherited so I do not have to embed a doc block everywhere a field is in a view.

does secoda support that or know of a way to do that with dbt or perhaps snowflake is supported?

I see there is a bulk update to do what I am asking about above?  Is there a way that this can happen automatically?  Can you point me at some docs?  Does this get retained upon a refresh?

<@U04LR81AUG3> here's the best way to accomplish what you're describing <https://www.loom.com/share/f716545a3efb4afb84c79c35d468a525>

Thanks <@U01MHQR7DE2> - I found this yesterday but will it retain the data after a refresh - I could not find anything to confirm this?  If I go to the trouble of using this feature then I do not want to see all the descriptions get deleted.  Also, what happens if I add a description to a filed that was bulk updated in say dbt and then do a refresh - will the dbt description take precedence?  I would be happy to read any docs but could not find any.

Hey <@U04LR81AUG3> thanks for the ping, so descriptions that are in Secoda would not be overwritten by descriptions that are in dbt unless you toggle the description preference under your dbt integration settings. If that's toggled then dbt will be used as the source of truth for descriptions

<@U01MHQR7DE2> - any docs besides the video on this?  If the toggle is set to NOT Always use the integration source I assume NO descriptions will be brought in and we do all of our work in secoda?

or will it bring in what it finds if there are no descriptions and then if we use secoda it will not overwrite what was changed?

There aren't any docs but <@U041UMTNUV7> it would be a good idea to create a doc to specify how importing descriptions works in more details.

In terms of your question <@U04LR81AUG3>. If the preference is not checked, the description will be brought in from the source if no description exists in Secoda. If a description exists in Secoda it will not be overwritten.

Hey <@U04JPJM9FAN> if you go into the Secoda UI and go to the Redshift integrations, the UUID is in the URL

Do you mean no API solution to get the UUID?

If you specify the `integration` parameter in the request, it'll filter by the specific integration which should resolve your problem.

Sorry, yes, I meant no way to look up integrations to get the UUID. E.g., something like `GET integration/integrations/` -&gt; `[{'id': uuid, 'name': 'blah', ...}, ...]` or `GET integration/integrations/&lt;uuid&gt;/` like exists for e.g. looking up tables

For the moment, pulling the UUID's from the website will suffice, but it would be nice to at least have readonly access to integration metadata via the API

Ok that makes sense, we can add a ticket for the ability to query the integrations but redact the sensitive information like the credentials.

Screen Shot 2023-01-30 at 11.22.37 AM.png

Hey <@U02QJG595KM> do you have the email of the user? You can DM it to me if you'd like. We can investigate once he have that info

The same applies to the import of tags (annotating existing columns in a schema with specific tags). Find the right database/schema in secoda and annotate specific columns of tables inside this schema through an API call.

Hi <@U04MZDFF5CY>, you could make a `GET` request to the `<https://api.secoda.co/table/columns/?table_schema=>&lt;schema-name&gt;&amp;table_database=&lt;database-name&gt;&amp;tags=&lt;tag-uuid&gt;`
To get all the columns in a certain database schema that have your desired tag.

To update the tags on a specific column, you could make a `PUT` request to the following endpoint
`<https://api.secoda.co/table/columns/column-id/>` with the following payload

```{
"tags": ["uuid1", "uuid2"]
}```
If you are on the EU instance, you would have to use `<https://eapi.secoda.co>` instead of `<https://api.secoda.co>`

Also if you would like to get the list of columns as a CSV without having to paginate through all of them, you could add this header to your `GET` request headers:

```Content-Type: text/csv```

Oh nice, thanks <@U02N4NXTTJR>!
Does `column-id` on the put request refer to a column uuid (I assume so as the table isn't specified)?

Can I pass multiple tags in `tags` on the GET request? How to separate, if so?

Yes, you could separate them with a comma `,`

Last question - probably an easy one - how do I obtain the `uuid` for a given tag (or all tags that are defined, so I can filter myself)

You could do a get request to `<https://api.secoda.co/tag/>` and it should return the list of all tags

Why wouldn't I find that info on `<https://api.secoda.co/api/schema/redoc/>` when searching for `tag` though? I am feeling a little dumb right now because your answers were pretty straight forward :joy:

Hi again! Since I need columns-ids to tag specific columns, I would first need to translate from a column name to it's id. Is there an API to retrieve all columns of a given table (ideally by the table's name, not it's uuid, because that will make me translate all table names to IDs first, too....).

Hi, yes you could get all the columns in a table by making a GET request to this endpoint:

```<https://api.secoda.co/table/columns/?table_schema=>&lt;schema-name&gt;&amp;table_database=&lt;database-name&gt;&amp;table_title=&lt;table-name&gt;```
This response is paginated to 50 results per page. You could also download all the tables all at once as a CSV by setting this http header

```Content-Type: text/csv```

thanks! I tried `table_name` and it didn't work.... who would have thought it was title...

[redacted random mumbling about how I didn't see a bug in my program :confused: ]
please disregard

Hey <@U02H1V13V99> we don't currently have this capability, but I'm curious to hear more about the use case. Do you prefer editing the documentation in Word vs Secoda, and if so it'd be good to hear what's leading to that preference

Nope, its not for editing. We need to present the tables info along with its documentation to the external customers without having access to the Secoda UI.

and also have one more question, why does secoda extract of a view changes the column order? (*Example*: The view might have defined to have a,b,c columns but after secoda extract of this view it follows random order).