Secoda Community

Hi <@U01TLJ5NDC2> - going to investigate this today and will provide an update!

Thank you! We found a work-around to publish the empty question and _then_ we can add a query into the question when going back and editing

Screen Shot 2022-10-17 at 8.18.17 AM.png

Hey Daniel, we’re working on a fix right now. Thanks for flagging

Currently, we only allow the chart as the default. Would it help to add the table as default? <@U041UMTNUV7> can you add this as a feature in the backlog?

<@U01PM6182AX> that's what is a bit confusing that for some resources "Chart" is the default, but for others is the "Table" the default

Thanks Daniel for raising, I'll add this note to the ticket so we can investigate further

Did you set the default as chart <@U03DA7NBWEB>?

<@U01PM6182AX> i am not even sure how to set a default, that was my original question, how can I set either Chart or Table as default?

Screen Recording 2022-10-26 at 5.59.24 PM.mov

Table is the default once you run a query. If you create a chart, you can follow this video to “set it as default”

:fire: excellent thanks so much <@U01PM6182AX>

np, sorry it took a while. I don’t think I understood the question correctly

```Initialization: 
'bytes' object has no attribute 'get'
Traceback (most recent call last):
  File "/app/databuilder/models.py", line 89, in process
    connection.extract(databuilder_job=self)
  File "/app/integration/connections/base.py", line 529, in extract
    self.extract_metadata()
  File "/app/integration/connections/dbt/dbt.py", line 387, in extract_metadata
    self._extract_manifest_metadata()
  File "/app/integration/connections/dbt/dbt.py", line 80, in _extract_manifest_metadata
    for table_node, manifest_node in manifest.get("nodes", {}).items():
AttributeError: 'bytes' object has no attribute 'get'```

Thanks for reporting Morgan, cc <@U01MHQR7DE2>

Hey <@U032DLWHQNN>, we’ll investigate into this issue and let you know when we’ve found a resolution. Thank you for reporting it

it seems like I am similarly unable to upload my manifest.json, as this:
```response = <http://requests.post|requests.post>(
        "<https://api.secoda.co/integration/dbt/manifest/>",
        files={
            "manifest_file": open(
                f"{os.getcwd()}/sources/{dataset}/target/manifest.json", "rb"
            )
        },
        data={
            "integration": &lt;my_integration_id&gt;,
        },
        headers=headers,
    )```
returns a 500 response. Even if I try to supply the file manually to the integration via the web UI, the extraction runs but nothing from dbt appears anywhere

another note, the documentation says the json key is `"integration_id"` but the endpoint requires `"integration"`

Hey <@U032DLWHQNN> we don’t have an update right now, but I’ll be working on a fix first thing tomorrow.

Similar for your issue <@U048TP20PPT>, so I’ll update you both in this thread after we push those changes tomorrow.

Hi <@U032DLWHQNN> - this should be resolved now, let us know if you experience any further issues!

<@U041UMTNUV7> we have error 500 now on our side

<@U048KN3EHFG> - thanks for letting us know - <@U01MHQR7DE2> flagging this for you

Thanks <@U048KN3EHFG> we’ll take another look into the issue today

Hey <@U048KN3EHFG> when you say you have an error 500 you’re not referring to the integration right? I’m having difficultly creating a 500 error for the dbt core integration

It’s not, but it helps us get more information on the source. are you having trouble seeing lineage?

yeah, I can view the tables from my BQ dataset but don't see a lineage tab. I wasn't sure if there was something else I needed to do

<@U01MHQR7DE2> can we take a look at why the lineage tab is not showing for kevin?

lineage does appear for the demo data, which is what makes me think I'm missing something

I think Etai DM’d you <@U048TP20PPT>, but just in case, you’ll have to add the following roles to the service account you used to connect BigQuery to Secoda
```BigQuery Metadata Viewer
BigQuery Resource Viewer
Job Viewer
Logs Viewer
Private Logs Viewer
BigQuery Data Viewer
BigQuery Job User```

Yep, I was using a service account managed by someone else that was missing a few of these roles. Sorry!

Screen Shot 2022-10-28 at 12.52.12 PM.png

<@U03113A05T6> can you look at this. Never seen this before

Thanks for bringing this up <@U048TP20PPT>! :pray:

Just checked Safari and I'm seeing it there too.

By the way, I think secoda is awesome and I totally didn't mean to show up out of nowhere just to point out issues or be confused :melting_face:

We’re stoked to see you using and loving Secoda! The feedback from the community is extremely helpful so thanks again! :star2:

now i have 500 code error same as this one : <https://secodacommunity.slack.com/archives/C01PT4WP0TE/p1666900908126999?thread_ts=1666723528.625549&amp;cid=C01PT4WP0TE>

Hey <@U048KN3EHFG> we have an open ticket for this issue. We’ll hopefully be able to get to it today. Thanks for reporting it

This would be an excellent quality of life upgrade, particularly for my business users who want to search the dictionary but don't always think in _exact_ terms

Hi <@U01TLJ5NDC2> - going to add a ticket for this as an improvement, we'll take a look and get back to you on this! Thanks for the feedback and thanks for the +1 <@U02QJG595KM>!

<@U03C8DUQEHF> sort of relevant to some of the stuff you’re working on this week

image.png

Hey <@U01TLJ5NDC2> this is available in the v5 update. If you have time this week I’m happy to walk you through the update

We’ve also released GCS support for dbt core and the Slack integration for self-hosted customers so lots of good updates!

Screen Shot 2022-11-08 at 9.40.13 PM.png

Hey <@U03DA7NBWEB> we’ll be updating the instance tomorrow with a bug fix. cc <@U02GZEAV18D> for updating to 5.4.6

<@U03DA7NBWEB> this should be resolved. Let me know if your extractions run successfully now

success :white_check_mark: - as always thanks so much <@U01MHQR7DE2>

<@U02GZEAV18D> deserves the credit here, I’m just the messenger

Screen Shot 2022-11-09 at 12.43.27 PM.png

<@U02GZEAV18D> <@U01MHQR7DE2> might have celebrated too early :grimacing:

the Integration was successful, but no tables are being pulled:

If you remove the filter does it show the tables <@U03DA7NBWEB>?

no it wasn't :confused: but i ran the integration a couple more times and on the 3rd time it is now showing up

Screen Shot 2022-11-09 at 12.40.05 PM.png

not sure if this is related to the issue, but also all of our Collections are showing up with 0 resources, eventhough they have tables published inside of it

<@U03DA7NBWEB> which instance is this happening on? The non-cloud one?

Can you also double check in your Redshift integration schema settings that the schemas are not all unchecked

Hey <@U01MHQR7DE2>
We are experiencing exactly the same issues described above by Daniel.

Hey <@U02BYJ618LW> I’ll coordinate with Kash and B.K for resolving this issue

Hey <@U01MHQR7DE2> <@U02N4NXTTJR> - this is unfortunately happening again after being fixed for 1 day. Yesterday I got a few people to test it after the fix and we were happy that the Redshift tables are back, but today they are missing again.

Thanks for updating us, I’ll coordinate with Kash and B.K to get it resolved today <@U02BYJ618LW>

Hey <@U04BM2D18AU>, I’m going to take a look into your issue now. I’ll update you shortly

What’s your workspace name <@U04BM2D18AU>?

it's called `Contractbook-Test`
I've created BigQuery integration. the last dataset that is fetched by Secoda is `dbt_konovalenko_xxxx`
while we still have a lot of other datasets below that last dataset

<@U04BM2D18AU> if you refresh the page all of your schemas should be there now

hi <@U01MHQR7DE2>
awesome! I can see all of our datasets, thank you so much :bow:

Nice stuff <@U04BM2D18AU> Glad to have you using Secoda :smile:

it's pretty cool :slightly_smiling_face: we're still in the middle of evaluating Secoda. thanks!

It should be in the tables that are in Secoda and transformed in dbt. If you check one of your tables and the lineage on those tables, you should see the BQ + dbt lineage. Let me know if it’s clear on your view. If not, I’m happy to take a look.

On the BQ table, you should see:
• column names
• table names
• tags
• owners
• lineage
• dbt tests
• queries
Let me know if it would help to put a view to show you how this looks on our end.

<@U01MHQR7DE2> not sure if you saw this because of the time

Hi <@U04AD9BUCM7> - we've got someone on the team looking into this right now, we'll provide updates with a solution soon

Hi <@U04AD9BUCM7> - can you rerun extraction and see if your assets are coming through now?

Hi <@U04AD9BUCM7> - just wanted to check in to see if you were able to re run the extraction.

Sorry, missed this update.
In queue. Did you trigger this? :shrug:  I'll letcha know!

If it's queued after our Snowflake metadata extraction it'll be a few hours

I believe <@U03C8DUQEHF> has a fix for this pending for the next release. Should be out shortly! Sorry about that

Hey Trevor, I pushed the fix yesterday, hope it’ll be out soon once it goes through our PR process!

<@U03C8DUQEHF> Thanks, that's good news.
Should I be looking out for that today or later this week?

Hey all, this has been fixed and should now be in the product

<@U03C8DUQEHF> this may be something to take a look at as a bug

<@U04AD9BUCM7> can you try now please. The reason why they weren’t sending is because the preferences for the Slack channel are managed here <https://app.secoda.co/settings/slack> and everything was turned off.

The personal Slack messages currently have a bug that’s disabling them from being sent, which we have in our backlog.

<@U01MHQR7DE2> actually when I click the New Question button nothing happens.

I had to go to Preview &gt; Ask a Question.

Then it did create a question, but it was staged to publish. I published. No alert at either time.

Sure,
• Dictionary: this a place you can define your metrics and common terms
• Documents: This is a place you can use for general analysis and documentation. Some examples include 
    ◦ How to onboard to snowflake 
    ◦ Analysis of why DAU is lower
    ◦ Everything you need to know about x
• Collections: This is a place to group any information into one place. Some common use cases for collections are:
    ◦ Team spaces (data team collection, marketing team collection etc.)
    ◦ Project spaces
    ◦ Onboarding collections
    ◦ Collections for “blessed” dashboards 

Are resources in collections only visible to groups assigned to the collection? Or is it more like a tagging solution? (why is this different than tags?)

Correct. We think about the differences between tags and collections as:
• Tags: Static ways to label resources with useful context that others will want to know about a resource (ie. PII, Do not delete, dbt data) etc.
• Collections: ways to assign visibility and add context around groupings of resources (ie. this is a folder of resources that only the sales team should know about and see) 

ooh i didn’t realize adding resources to collections would remove them from general access. Is there a way to turn this off, or does it not apply (aka available to all) if there is no group added?

That’s correct, it does not apply (aka available to all) if there is no group added

<@U02N4NXTTJR> I know you’re working on Sigma. Can you confirm if datasets are pulled in?

<@U04AD9BUCM7> we do not currently pull in datasets, but we have a ticket to update the integration to do so. The only issue with Sigma’s APIs is the dataset information is very limited. They don’t include columns or lineage information from the dataset, i.e, which workbooks use the dataset.

ooh ... hmm.

So many (most) of our users will only have access to datasets that are curated in Sigma.

Giving them visibility into what a table is/is not used for and the definitions of the columns is a major use case for us.

Dashboards is good too, but datasets is really the primary use case.

~Any idea when that might be? Is it high on the roadmap?~

Oh geez no wait, we just read Sigma's API docs.
We'll complain to them! :shrug:

Yeah, Sigma’s API could be more detailed lol

<@U01MHQR7DE2> can I manually add a table to the resource list? (in a perfect world assigning the source to Sigma)

Otherwise where's the best place to hack missing Sigma datasets do you recon?

We were messing around with creating a collection called Sigma Datasets and manually writing in important column names. Open to suggestions here!

Hey <@U04AD9BUCM7> would you be ok with having the datasets without the columns? We can push a quick update to the Sigma integration for that.

Yeah, I think so? I can perhaps type the important column definitions in the description

Sorry, I'm actually far more enthusiastic about trying this out than that sounded, slack multitasking :zany_face:

Yes please! I'd love to try that!

Screen Shot 2022-11-16 at 3.19.26 PM.png

I also updated a dictionary term once, about 3 hours ago and it generated 20 notifications in my inbox. The "new question" button is also not working in the Questions section. Clicking on it yields no action for both the admin and editor who have tried

Screen Shot 2022-11-16 at 3.28.25 PM.png

Screen Shot 2022-11-16 at 3.27.16 PM.png

Ooo. Interestingly, in the analytics tab I can see dashboard resources listed in the tile. When I click it I get a permission denied error. I am an admin for reference.

Hey <@U02QJG595KM> we're looking into all three issues right now, with the missing dashboards first

Looker dashboards are appearing now, but no sigma dashboards yet

Hey <@U02QJG595KM>, was just going to update you on the Looker dashboards. We're currently working on the Sigma integration, it should be available soon

Hey <@U028E7B1U31> we are investigating this issue right now. We'll get back to you once your information is restored

<@U028E7B1U31> all of the documentation has been restored to it's normal state

Hi <@U01MHQR7DE2> thanks for this fix. Everything is back except for the column collections. We use collections at the column level to flag deprecated columns and this is very important for us. Could you have a look at this please?

Yes, we're taking a look now <@U028E7B1U31>

Hey <@U028E7B1U31> do you remember any of the columns that were in a collection, off the top of your head?

<@U028E7B1U31> the collections should be back now, please let me know if it's reflected on your end

<@U04AD9BUCM7> I'd recommend the following:
1. Create a "Data Team" which you can create as a group in <https://app.secoda.co/settings/members>
2. Update the global share settings in <https://app.secoda.co/settings/sharing> that only shares resources by default with the Data Team
3. If you open the share settings (top right corner of collection page) for a collection that contains the dictionary terms that you'd like the share, then you can set specific share permissions on that collection which will take precedent over the global share settings

I changed the global permissions to Data Team.

and I created a Sigma Users collection with restricted access, and a separate slack channel

but when I `/secoda user source` on that channel I still get all possible return values

Oh ok, so I asked `<mailto:kelly.chien@babylist.com|kelly.chien@babylist.com>` to try `/secoda product catalog` , whom I tried to set as a `Sigma User` , and she's still seeing some low-level assets I don't want her to get.

I definitely am not successfully restricting access?

<@U01MHQR7DE2> any thoughts on what I'm doing wrong?

<@U04AD9BUCM7> after investigating, the issue is that we are not able to link the user that's requesting the Slack search request to the user in Secoda. There are two options that we can do to resolve this:
1. We can infer who the user is based on their Slack email, but this could be inaccurate if there are inconsistencies
2. We can allow a user to manually specify their Slack ID in Secoda but this requires more user input
We can prioritize #1 since I imagine it will work for almost all cases since the users sign up to Secoda with their business email.

Ohhh I was hoping I could get there through the slack channel. I connected the Sigma Users channel to a separate `#tmp-secoda-sigma-users` .

#1 is for sure the preferred option of the two you mentioned, we all use a consistent email across SaaS products.

What _does_ the slack channel assigned to a collection do?

It will send updates for the collection, i.e, description change, to the respective Slack channel

We're investigating this issue <@U04AD9BUCM7>. I'll get back to you shortly

Or at least, it has gone longer than the previous failed attempts

Not at the moment. Just Viewer / Editor mode

<@U02QJG595KM> brought up this same issue in a previous thread. Again, due to limitations on the Sigma API, we can't pull in the workspace that's associated with a workbook. I've mentioned this to Julie, but the plan is to refactor the Sigma integration to use their internal APIs so that we can get access to more information. ETA is hopefully before the end of 2022

Ok :crossed_fingers:!

So then it feels like this is again a permissioning / access solve.

I'm less concerned w/ everyone on Data Team &amp; Expert Creators being able to see all dashboards provided the Sigma Users _can't._ Data + Experts can manually add important dashboards to the Users collection.

Think that would work? Could I get a hand setting that up correctly?

That would work <@U04AD9BUCM7> if the users are accessing within Secoda. If they're searching on Slack it will still run into the same issues as I described in the other thread

<@U04AD9BUCM7> - I've sent you a separate message about setting up some time to check in :slightly_smiling_face:

You are correct <@U04AD9BUCM7> I'll make a ticket for this. Can you please link the URL of the dictionary term so that I can include it in the ticket

<https://app.secoda.co/dictionary/0c4c5f3e-7ece-4d49-93ec-4c9f2fe6e6bc>

If you could just make table a chart type, I'd probably often just default to that :slightly_smiling_face:

<@U04AD9BUCM7> If you configure the y Axis to be Numerical instead of Categories it shows it correctly.

<@U01MHQR7DE2> Carries colleague here, is it possible to configure the x axis (categories) to show all intervals? Looking at the source code you use echarts?<https://stackoverflow.com/questions/44076557/echarts-how-to-show-all-axis-labels>

Thanks for the pointer <@U04C7Q1H022> I'll pass this along to the engineering team

Hey <@U04AD9BUCM7> this issue has been resolved now

Hey <@U032K7RJDH8> there's a bug fix available in the latest version of Secoda. I think that 7shifts updates their instance on the weekends, so you'll get the fix on Monday

<@U01MHQR7DE2> can you take a look at this?

fyi, I created it another DBT Cloud account (the first one has many projects inside)
With this new account where only 1 project exists, it seems working

Hey <@U04CE3N6XH7> I'm taking a look into the issue, I'll get back to you shortly

<@U04CE3N6XH7> I'm just going to send you a DM to run an API request that the integration is failing on

<@U04CE3N6XH7> we've pushed a fix for this issue, let me know if the dbt integration runs successfully now

Hey <@U032DLWHQNN> happy to answer each of your questions:
1. We're adding a dbt icon that will help you indicate if something has been decorated by dbt or not
2. This is a bug on our end that we're fixing this week
3. We pull in View SQL and Tests currently. See screenshots below
4. Everything that's pulled in from dbt will be automatically published
If any of these answers aren't clear please let me know and I can provide more details.

Hi Andrew, FYI we don't see the tests on our side
Config: BQ + DBT and tests on DBT + EU instance

Hey <@U03C85WHYSX> <@U032DLWHQNN> we've pushed a fix for the dbt descriptions and tests. Let me know if this is resolved on the next extraction

I think it didn't resolve any problem on my side unfortunately (test, table description or shutdown)

<https://www.loom.com/share/6e8a47a62dbe4ab8b453c32d370d8e7d>

Update - it seems to be working reliably if you hit "create invitation link," refresh the page, and then select the group. I'm on Chrome Version 107.0.5304.110 (Official Build) (arm64)

Thanks Julie - we'll take a look. Thanks for reporting

<@U041UMTNUV7> Possibly related bug - when members are joining with the group invitation links in the video above, they are not getting assigned to those groups. They appear as new members but not on the groups page. I've been manually adding them to their groups

Thanks for letting us know about that as well Julie, I'll add a ticket for engineering to take a look at!

Hi Julie - heads up that this has been resolved in our most recent update.

Hi <@U04C7Q1H022> - where are you trying to add images?

<https://app.secoda.co/docs/17105621-3c29-4099-93b1-aaf89172d378>

Thanks Andre, it's a bug on our end, I'm going to request our support engineer to take a look. Thanks for providing a link, that's helpful

Hey <@U04C7Q1H022> we've resolved this issue now

Hey <@U04C7Q1H022> how many changes are you publishing? It should be close to instant if it's working correctly

Happy to hop on a video share if that helps you

<@U04C7Q1H022> can you try publishing all when you get a chance, I want to see if it triggers an exception that I can see

Look at it it seems to happen in the frontend:

```Uncaught (in promise) Error: Request aborted
    l_e <https://app.secoda.co/assets/index.35d5431c.js:1>
    onabort <https://app.secoda.co/assets/index.35d5431c.js:2>
    v_e <https://app.secoda.co/assets/index.35d5431c.js:2>
    v_e <https://app.secoda.co/assets/index.35d5431c.js:2>
    bIr <https://app.secoda.co/assets/index.35d5431c.js:2>
    promise callback*hQc&lt;/ZU.prototype.request <https://app.secoda.co/assets/index.35d5431c.js:2>
    e <https://app.secoda.co/assets/index.35d5431c.js:2>
    eNt <https://app.secoda.co/assets/index.35d5431c.js:1>
    $n <https://app.secoda.co/assets/index.35d5431c.js:259>
    onConfirm <https://app.secoda.co/assets/index.35d5431c.js:3929>
    h <https://app.secoda.co/assets/index.35d5431c.js:125>
    c$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    d$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    f$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    Krt <https://app.secoda.co/assets/index.35d5431c.js:31>
    pLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    mLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    WNt <https://app.secoda.co/assets/index.35d5431c.js:35>
    l$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    mLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    XLe <https://app.secoda.co/assets/index.35d5431c.js:31>
    KLe <https://app.secoda.co/assets/index.35d5431c.js:31>
    unstable_runWithPriority <https://app.secoda.co/assets/index.35d5431c.js:20>
    xM <https://app.secoda.co/assets/index.35d5431c.js:31>
    HNt <https://app.secoda.co/assets/index.35d5431c.js:35>
    F$r <https://app.secoda.co/assets/index.35d5431c.js:31>```

```Uncaught (in promise) Error: Request aborted
    l_e <https://app.secoda.co/assets/index.35d5431c.js:1>
    onabort <https://app.secoda.co/assets/index.35d5431c.js:2>
    v_e <https://app.secoda.co/assets/index.35d5431c.js:2>
    v_e <https://app.secoda.co/assets/index.35d5431c.js:2>
    bIr <https://app.secoda.co/assets/index.35d5431c.js:2>
    promise callback*hQc&lt;/ZU.prototype.request <https://app.secoda.co/assets/index.35d5431c.js:2>
    e <https://app.secoda.co/assets/index.35d5431c.js:2>
    eNt <https://app.secoda.co/assets/index.35d5431c.js:1>
    publish <https://app.secoda.co/assets/index.35d5431c.js:224>
    onConfirm <https://app.secoda.co/assets/index.35d5431c.js:3929>
    h <https://app.secoda.co/assets/index.35d5431c.js:125>
    c$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    d$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    f$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    Krt <https://app.secoda.co/assets/index.35d5431c.js:31>
    pLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    mLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    WNt <https://app.secoda.co/assets/index.35d5431c.js:35>
    l$r <https://app.secoda.co/assets/index.35d5431c.js:31>
    mLt <https://app.secoda.co/assets/index.35d5431c.js:31>
    XLe <https://app.secoda.co/assets/index.35d5431c.js:31>
    KLe <https://app.secoda.co/assets/index.35d5431c.js:31>
    unstable_runWithPriority <https://app.secoda.co/assets/index.35d5431c.js:20>
    xM <https://app.secoda.co/assets/index.35d5431c.js:31>
    HNt <https://app.secoda.co/assets/index.35d5431c.js:35>
    F$r <https://app.secoda.co/assets/index.35d5431c.js:31>```

<@U04C7Q1H022> this appears to be a Firefox issue. We've published the changes on our end and will resolve the issue shortly

Ok, let me know when it’s fixed and I can test it