Hello, I am having a problem with not seeing the validations of a certain PostgreSQL table. Not long ago (this morning) I was able to send different validations to another table. The thing is that now I am unable to do so, although I am following the same procedure. I am not seeing any error logs and, as shown in the image, the validation succeed. Any idea why could it be?

hey Pablo, so you say you can't send validations to the same table that you were able to send it to this morning as well? or is it just a problem with this new table not receiving updates?

I can't send the validations to another table from the same database and that is also ingested in Datahub. I will try again to send the validations to the table where it worked this morning and report back

ah yup makes sense why you can't see them if we never get them into ES. do you see any logs out of the ordinary in ES?

No at all. These are all of the logs from yesterday. As you can see, I erased the index to eliminate the possibility that such index might influence. Unfortunately, before and after the deletion of the index I got the same result

{"type": "server", "timestamp": "2023-01-11T01:34:00,001+01:00", "level": "INFO", "component": "o.e.x.m.MlDailyMainten                                        anceService", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFa                                        dCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T01:34:00,007+01:00", "level": "INFO", "component": "o.e.x.m.a.TransportDel                                        eteExpiredDataAction", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "Deleting expired data", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUp                                        TfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T01:34:00,012+01:00", "level": "INFO", "component": "o.e.x.m.j.r.UnusedStat                                        sRemover", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "Successfully deleted [0] unused stats documents", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MF                                        adCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T01:34:00,012+01:00", "level": "INFO", "component": "o.e.x.m.a.TransportDel                                        eteExpiredDataAction", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "Completed deletion of expired ML data", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "                                        MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T01:34:00,013+01:00", "level": "INFO", "component": "o.e.x.m.MlDailyMainten                                        anceService", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask", "cluster.uuid": "0Wl_IwRZT                                        S2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T02:30:00,000+01:00", "level": "INFO", "component": "o.e.x.s.SnapshotRetent                                        ionTask", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "starting SLM retention snapshot cleanup task", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFadCi                                        l_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T02:30:00,001+01:00", "level": "INFO", "component": "o.e.x.s.SnapshotRetent                                        ionTask", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "there are no repositories to fetch, SLM retention snapshot cleanup task complete", "cluster.uuid": "0Wl_IwR                                        ZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T14:09:40,171+01:00", "level": "INFO", "component": "o.e.c.m.MetadataDelete                                        IndexService", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "[assertion_assertionruneventaspect_v1/C05FOxjXTJGXZuV43SJxpg] deleting index", "cluster.uuid": "0Wl_Iw                                        RZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T14:11:52,087+01:00", "level": "INFO", "component": "o.e.c.m.MetadataCreate                                        IndexService", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "[assertion_assertionruneventaspect_v1] creating index, cause [api], templates [], shards [1]/[1]", "cl                                        uster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-11T14:11:58,232+01:00", "level": "INFO", "component": "o.e.c.r.a.AllocationSe                                        rvice", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[assertion_assertionrunevent                                        aspect_v1][0]]]).", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "deprecation.elasticsearch", "timestamp": "2023-01-11T16:48:10,106+01:00", "level": "CRITICAL", "component":                                         "o.e.d.r.a.d.RestGetAction", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "[types removal] Specifying types in document get requests is deprecated, use the /{index                                        }/_doc/{id} endpoint instead.", "key": "get_with_types", "category": "types", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFadCil_RkeqeiUpTfZmWA"  }
 {"type": "server", "timestamp": "2023-01-12T01:34:00,000+01:00", "level": "INFO", "component": "o.e.x.m.MlDailyMainten                                        anceService", "cluster.name": "audit-cluster", "node.name": "elasticsearch1", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "0Wl_IwRZTS2LGWwbW8FP4A", "node.id": "MFa                                        dCil_RkeqeiUpTfZmWA"  }

wait so you deleted your assertion index? i'm a bit confused on that. also you said you were previously receiving assertions in your datahub instance right? do you know if anything has changed in your config file since then?

Yes I did, I wanted to eliminate the possibility of such index having to do with it. Yes, datahub received validations at first. Theorically the files shouldn't have varied that much, it is true that I haven't used the same files and I have created each time a different set of files (but they were based on the same files). By "they were based on the same files" I mean that I used another set of files that worked (and have been working) to take the important lines out of them.

okay I think i understand. have you restored that index? because we need that index up and running in order to see assertions in the UI. are you seeing assertions appear in your document store in mysql? just trying to figure out where the issue is originating - like is it not getting to datahub at all (we don't see it in mysql) or is it getting to datahub but not getting set in elastic search (where you don't see it in that index)

No I haven't. How could such index be restored? And another thing, how are the assertions saved in the database?

okay yeah if that index no longer exists, then we won't be able to search for them and the UI won't display them. tbh I would just tear down your elasticsearch instance and start fresh, as I'm not really sure how to restore an individual index. you're going to have to run the restore indices job which will scan your database and populate relevant indices https://datahubproject.io/docs/how/restore-indices/#through-api

I am trying to look for them, but I can't seem to find any assertion information:

datahubdb=>select * from metadata_aspect_v2 where aspect LIKE '%assertion%';
(0 rows)

I just erased the assertions from ES and the UI (each individual assertion on my dataset, as they were storing one over another), but not from PostgreSQL, should they appear?? At least the assertion uid created for dataset assertions??

so if you delete the assertions from the UI, that will also delete them from postgres

Seems like it yeah

sounds good!

It must be some temporal error or something like it because I have tried creating expectations in other occasions and have been successful

ahh gotcha. well I'm glad you've got it up and working again!