Secoda Community

Is there any reason why this could be the case. I also noticed that in the past, some extraction processes took about  30hours to be completed. Can  you please shed more light on this

Hey <@U02UKSDP67J> I’m just sending you a DM

Hey Kiara, can you pull the latest version and re-run the MySQL extraction. This should be fixed in the latest update. Let me know if that works for you

perfect, it worked! thank you <@U01MHQR7DE2>!

Ideally yes, we don’t have a native integration with matilion but you could use our API for this part of the stack. Everything else can be easily connected and monitored with Secoda. Happy to show you how over a call if you’d like? 

one last question before we proceed here
ARE you HIPAA complaint ?
Cuz these are medical inf

I’ll message you about this, it’s a little bit more complex than a simple Y/N

Hi, that repository is from an older version. The latest is in this *public* repo:
```<https://github.com/secoda/terraform-aws-secoda>```

Hey <@U02P9HZGUJH> if you go into the Redshift integration and the schema tab you’ll be able to select which schemas are visible. The dbt integration won’t have any effect on the tables that are imported because it just decorates Redshift tables with metadata like lineage, descriptions, etc

<@U031Z4XUNMD> thanks for reporting. While we’re fixing, you can just send them the viewer invite link in the members section and then change their role once they join the workspace.

Thanks <@U02F1A61UEQ>, this is a bug on our end that we’ll fix

Hey <@U03S8TT8UHY>, It looks like you’ve gone over the usage limits for the <https://www.secoda.co/pricing|Free tier>. You will have to upgrade to our Growth tier to continue to use Secoda. Would you like us to send you the link to upgrade?

But I just have one integration, one editor and two viewers.

So I don’t know why occurs this

I see, I believe it’s because you’ve _invited_ multiple people as editors, but it looks like they haven’t accepted the invite yet.

We’ve removed the paywall so you can remove the other editor by going to <http://app.secoda.co/setttings/members|app.secoda.co/setttings/members>

You can, but this is a feature for admins / editors. You may want to ask you workspace admin <@U02739P8464> about this

<@U01PM6182AX> This feature is actually broken, has been for some time now. I do think this option should be at the table/view level available to all users instead of at a project level.

Thanks for the feedback <@U02739P8464>. In the meantime while we’re fixing this issue, you can use the API and this notebook to get all of the columns. If you have any questions let me know <@U02H1V13V99>

<https://github.com/secoda/secoda-api-examples/blob/master/Get%20All%20Columns.ipynb>

Follow-up: is it possible to show Foreign Keys in the Columns view?

Hey <@U03PB4YJ7GW> we automatically pull the foreign/primary keys that are defined in the information schema in postgres. I can send you the query we run in a DM if you’d like to run it and see what’s returned.

Thanks I thought that was the case, but I do not see the FKs in the UI, although I do see them in our DDL

Here's an example, from the Secoda UI, Related is empty (is this the place it would be displayed?):

While when I look at the database I see a FK defined for this field:

Hey <@U03NJMGE4Q4> we did move the description to the main panel, but it’s a bug that it’s truncated. We’ll resolve that issue today

<@U03NJMGE4Q4> can you DM me the link of the dictionary term that’s cut off / a screenshot. That would be helpful for debugging the issue

Hey <@U042XHUDFDX> are you using an ad blocker? It looks like the requests are not able to be made

maybe we have a blocker, which is the image bank that is linked to the background?

<@U042XHUDFDX> if you select an image from this website I can set it manually for you <https://unsplash.com/s/photos/bank>

this one
<https://unsplash.com/es/fotos/5W8fDb8f89s>

can we use an image that is not from unsplash?

Sure, which integration do you want to remove?

thank you so much Etai! I can't really remember which one other than Redshift i had enabled, but essentially would like to focus on the Redshift one as my only one at the moment :slightly_smiling_face:

<@U01MHQR7DE2> are you able to help Daniel?

Hey <@U03DA7NBWEB> happy to help, what email did you sign up with?

Hey <@U02P9HZGUJH> this is the query that we run on Redshift to get the schemas. If you run it with the same user that’s connected to Secoda and you get different results than what you’re seeing in the UI let me know.
```        SELECT DISTINCT
            t.table_catalog       AS cluster,
            t.table_schema        AS schema
        FROM svv_tables t
        WHERE schema != 'pg_catalog'
            AND schema != 'information_schema'
            AND schema != 'pg_internal'
            AND schema NOT LIKE 'pg_temp_%%'
        ORDER BY cluster, schema;```

Hey <@U01MHQR7DE2> yeah, I get 25 schemas when I run that query from within my SQL client using the credentials that Secoda uses to connect to our RS cluster

<@U02P9HZGUJH> you are correct, it’s an issue on our end. We’ve pushed a fix that should be available in ~10 minutes to fix your issue. Please let us know if it’s resolved

<@U01MHQR7DE2> thank you sir! Happy to report I can see them all now. Y'all are lightning fast with those hotfixes. :zap::pray:

Hey <@U03C85WHYSX> there was an issue that lead to some downtime for the application, but it’s back up now. The issue was due to non-rate limited API calls which overloaded the DB. The API is now rate limited so that this issue won’t happen again in the future.

Thanks a lot Andrew :slightly_smiling_face:

In case it helps, the same behavior occurs when I try to add a Collection to another Collection (i.e. to create a `parent` relationship)

Hi Daniel - after adding a Collection as a parent to another Collection - did you try refreshing the page?

yep - the `Schema` never gets added to a `Collection` and a `Collection` never gets added to another `Collection` (when using the `(+)` functionality)

Hi <@U03DA7NBWEB> - is this on <http://app.secoda.co|app.secoda.co>? You are correct about #1 - while it is added, we don't display that anywhere in the app right now. For #2, unfortunately, I can't reproduce it. I am able to view it when I navigate to the parent collection.

I have made a ticket for #1 on our end. Let me know about #2.

Thanks so for #1 we are saying that in the backend a Schema is added to the collection, but in the Collection UI there is nothing to display that

Thanks for the additional context here <@U03DA7NBWEB>, we’ve added a ticket for this issue to show schemas in collections

<@U02N4NXTTJR> are you able to take a look at this?

Pushing a fix for it, should be live in 10-15 minutes

Thanks a lot Kash, let me know when it's live :slightly_smiling_face:

Hi Daniel - that issue has coincidentally been fixed. There is a release going out this moment to resolve it.

fyi also - whenever I try to update multiple columns at once using the copy/paste approach not all changes get registered: For example if I copy and paste 3 times, only the first 2 actually get `Staged for Publish` and the rest will remain stuck in `Draft` "forever"

<@U03DA7NBWEB> for the copy/paste issue, are you using CMD + C and CMD + V on your keyboard, or copying/pasting via another method? I’m just trying to reproduce the issue to understand how we can fix it

keyboard method (tbh it just seems like a "rate-limiting" issue)

To reproduce:
• Catalog View
• Columns
• Put a Description on the 1st column (the "Staged" icon should appear)
• Quickly copy and paste (CMD + C &amp; CMD + V) the description to column 2,3,4,5,etc
• You will see only the first columns get registered as "Staged"

That’s helpful, you’re correct that it’s likely a rate limiting issue. I’ve created a ticket

Hi <@U03DA7NBWEB>! Dropping by to give you an update that this rate limiting issue has been resolved by our engineering team :clap:

Hello Ben, viewers are not able to execute queries themselves but only view the results of executed queries. If you want the viewers to be able to see up-to-date data from the queries, you could schedule the queries to run automatically on a set schedule (i.e. every 24 hours)

Any thoughts about <@U02N4NXTTJR>’s response <@U03PB4YJ7GW>? Does this accomplish your goal?

<@U03PB4YJ7GW> we’ve created a ticket for this issue, you should see it resolved early this week

<@U03DA7NBWEB> this is an issue with our documentation, we do not currently extract that last updated at timestamp and I’ve updated the documentation to reflect that.

However, it <https://stackoverflow.com/a/38620381|looks like >you can retrieve the last updated timestamp, so I’ve added a ticket to our backlog to get this info.

<@U01MHQR7DE2> - thanks! yeah getting that "Last Updated" from Redshift tables is a bit tricky so that's why i was surprised we had that feature. Is there a way that temporarily we can hide that column? it just feels that `Not available` is an issue on "our side" but it really is because that Metadata is not pulled to Secoda (so it creates an experience where it suggests the Documentation is not really complete)

Yeah you can temporarily hide it by clicking the “Customize” button in the top right and toggling that column <@U03DA7NBWEB>

Thanks so much Andrew!  i might be missing something :disappointed: but on the `Collections &gt; Tables` there is really not a `Customize` button (i can see under the `Catalog` view)

So the current behavior would be:
• User clicks on `Collections`
• User clicks on `Tables`
• User sees `Not available`  on all Redshift tables. Specially since there are only 5 columns in this view the `Not available` really stands out

Ah, I thought you were talking about the catalog page, not the collections page. We’re releasing a new update to the collections page that will allow you to customize the columns for the table. That should be released by the end of the week

<@U01MHQR7DE2> qq was this released yet? i have been checking since last week but nothing yet

Not yet <@U03DA7NBWEB>, it will be available early next week. Our engineer that started the project is on vacation and will be back on Monday

Hi <@U01MHQR7DE2> nothing quite yet here right?

That’s correct <@U03DA7NBWEB>, the project was delayed but should be out by the end of today

<@U03DA7NBWEB> you’re correct, this is a snippet of the code that’s causing the issue.
```            for obj in self._client.list_objects_v2(
                Bucket=self._s3_bucket, Prefix="great_expectations.yml"
            ).get("Contents"):
                if obj.get("Key") == "great_expectations.yml":```
What does your current file structure in S3 look like?

<@U01MHQR7DE2> got it got it that's what i thought.. so we don't really have `great_expectations.yml` in our S3 bucket bc that's not really a requirement and we send our data context "on the fly" whenever we do our GE validations using an Airflow Operator. So essentially something like:
```from great_expectations.data_context import BaseDataContext

context = BaseDataContext(project_config={
    'datasources': {
        'my_datasource_name': {
            'module_name': 'great_expectations.datasource',
            'class_name': 'Datasource',
            'execution_engine': {
                'module_name': 'great_expectations.execution_engine',
                'class_name': 'PandasExecutionEngine',
            },
            'data_connectors': {
                'my_dataconnector_name': {
                    'module_name': 'great_expectations.datasource.data_connector',
                    'class_name': 'InferredAssetS3DataConnector',
                    'bucket': 'some_bucket',
                    'prefix': 'some_prefix',
                     'default_regex': {
                         "pattern": "(.*)",
                         "group_names": ["data_asset_name"],  # This name is important, not a placeholder
                     }
                },
            },
        }
    },
    ...
})```
but i was wondering what do you extract from the `great_expectations.yml`? because i can just put a dummy one in there to avoid the fail integration

<@U03DA7NBWEB> you’re correct, announcements should not be included in the publishing worflow. We’ve created a ticket to address this issue

Hi <@U02H1V13V99> - is this for all columns when you click on the magic wand?

I'm having trouble reproducing your issue. Can you tell me which column you get this on specifically.

Nope, it's happening for some of the columns in the view and after sometime, I can see the magic marker working but it gives ERR message first then after 2-3 times clicked on magic marker, I am able to see the columns.

Hi <@U02H1V13V99> - so it seemed you were hitting our rate limits on that endpoint. I am putting out a change to increase the rate limit.

Should be live in about 10 minutes. Thanks.

<@U0403DQSD0D> I’ll send you a DM regarding this

What’s the link(s) that you’re trying to embed <@U03DA7NBWEB>? It’s unfortunately out of our control because some websites block iframes. For example, if you put in <https://wikipedia.org> it should show up ok

<@U01MHQR7DE2> ah makes sense thanks a lot :ok_hand::skin-tone-4: i tried a bunch, but maybe just all the ones i tried "block" the iframe.

I was kind of expecting the same behavior as Notion:
1. <https://wikipedia.org> -&gt; bc it accepts iframe then the embed is an actual embed
2. <https://google.com> -&gt; doesn't accept iframe, so instead of "refused to connect" it shows like a "nice link"

That’s good feedback, I’ll add a ticket for this

FWIW, I've noticed issues this week as well with my embedded Looker frames, both with new attempts and things that were working previously.

Are those missing datasets in the same location as the ones that are being imported successfully <@U0454PR5754>? Sometimes that leads to issues

This seems to be the problem, thanks Andrew. Is it resolvable without the need to migrate the data?

Yes <@U0454PR5754>, if you put the data location in this screenshot in the *Location* parameter in the Secoda BQ integration connection details, you’ll should resolve the issue.

Hey <@U0454PR5754> are you able to share what the structure of your GE files are in GCS, and we’ll be able to determine what’s causing the issue

It might be the same with <@U03DA7NBWEB> in that we don't have great_expectations.yml on the GCS bucket and we only persist the configuration on memory every time a validation is run.

this might help:
```            datasources={
                datasource_name: {
                    "module_name": "great_expectations.datasource",
                    'execution_engine': {
                        'class_name': 'SqlAlchemyExecutionEngine',
                        'connection_string': f'bigquery://{gcp_project_name}/{self.dataset}?credentials_path={credentials_path}',
                    },
                    'data_connectors': {
                        'query_runtime_data_connector_name': {
                            'class_name': 'RuntimeDataConnector',
                            'module_name': 'great_expectations.datasource.data_connector',
                            'batch_identifiers': {
                                "default_identifier_name": "default_identifier"
                            },
                        }
                    },
                    "class_name": "Datasource",
                }
            },
            stores={
                "expectations_GCS_store": {
                    "class_name": "ExpectationsStore",
                    "store_backend": {
                        "project": gcp_project_name,
                        "class_name": "TupleGCSStoreBackend",
                        "bucket": self.ge_bucket,
                        "prefix": f"results/{self.dataset.lower()}/{self.table.lower()}/{self.ge_path}/expectations",
                    },
                },
                "validations_GCS_store": {
                    "class_name": "ValidationsStore",
                    "store_backend": {
                        "project": gcp_project_name,
                        "class_name": "TupleGCSStoreBackend",
                        "bucket": self.ge_bucket,
                        "prefix": f"results/{self.dataset.lower()}/{self.table.lower()}/{self.ge_path}/validations",
                    },
                },
                "evaluation_parameter_store": {"class_name": "EvaluationParameterStore"},
            },
            expectations_store_name="expectations_GCS_store",
            validations_store_name="validations_GCS_store",
            evaluation_parameter_store_name="evaluation_parameter_store",
            validation_operators={
                "action_list_operator": {
                    "class_name": "ActionListValidationOperator",
                    "action_list": [
                        {
                            "name": "store_validation_result",
                            "action": {"class_name": "StoreValidationResultAction"},
                        },
                        {
                            "name": "store_evaluation_params",
                            "action": {"class_name": "StoreEvaluationParametersAction"},
                        },
                    ],
                }
            }```