Hi all. We recently upgraded our stage environment...
# troubleshoot
e
Hi all. We recently upgraded our stage environment from 0.9.6.1 to 10.1 and after that it seems like entities that have been soft-deleted are appearing again as if they've never been deleted. Any idea what might have caused that, and how can we get them back to being soft-deleted? We're using kubernetes and datahub helm chart, and restore-indices job has run successfully
🔍 1
📖 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
this likely happened due to the reindex that ran on upgrade
you may be able to restore from a backup- do you have a list of the entities? it could maybe be used with an sdk
e
but the problem is not the missing entities that could be restored, but the old-new "dead" ones appearing... what exactly should be restored here?
a
those entities were soft deleted, so a restore would re-delete them by applying that quality to the entities
you could also run a job to tag them and then delete with filters on the new tag
e
but while migrating from 0.9.6.1 to 10.1 we alreasy did a full data restore to get all our data in... do you mean doing it again, or some other kind of restore?
a
Hmm, the restore may have been the issue if the backup was from before the soft deletes
so in that case you wouldn’t be served by that- it may unfortunately come down to soft deleting again- @bulky-soccer-26729 are you familiar with any ways we might be able to fix this?
b
yeah if data was restored from before soft deleting it would make sense that the soft deletes are no longer in there. was the data restore necessary after the upgrade? i'm curious whether the upgrade caused this soft delete issue or if it's from a data restoration where the data saying which entity was soft deleted wasn't there
e
What do you mean was the data restore necessary? As far as I undertake migrating from one release to another is like a big restore from backup, at least that's how our DevOps engineer explained it to me
ohh I think I found an aspect that might have to do with re-appearing of soft-deleted entities. It's the
negated
aspect. Context: I used to have 9 Domains in my stage Datahub, and after upgrade to 10.1 I see 11 Domains, 2 of which seem to be old deleted ones The usual search for soft-deleted entities in GraphQL would be:
Copy code
searchAcrossEntities (
  input: { types: [DOMAIN]
    query: "*", count: 50,
    orFilters: [{and: [{field: "removed", condition: EQUAL, values: ["true"], negated: false}]}]
  } )
With this query I get 0 results, 0 soft-deleted Domains. But if i change the
negated
aspect to
true
, I get 11 Domains: my re-appering deleted Domains are there as well. Now I also have a full copy of my stage Datahub BEFORE the upgrade, a copy is on 0.9.6.1 release. And if I run the query with
negated
aspect set to
true
, I get only 9 domains - the re-appering deleted Domains are not there. The query with
negated: false
on pre-upgrade copy also gives my 0 domains, the the re-appering deleted are not there.
Another example that might make it more clear: on pre-upgraded copy of datahub (on 0.9.6.1) this query returns a total of 184 containers:
Copy code
{
  searchAcrossEntities (
  input: { types: [CONTAINER]
    query: "*", count: 50,
    orFilters: [{and: [{field: "removed", condition: EQUAL, values: ["false"], negated: false}]}]
  } )
But if I change
negated
to
true
- it gives me a total of 1105 containers. But on afer-upgrade version of Datahub (on 10.1) both versions of the query, with
negated: false
and
negated: true
give me a total of 1105 containers. Looks like bottomline is that
negated
aspect for soft-deleted entitied was somehow messed up in 10.1 or 10.0 relesase. I can't tell for sure cause we skipped 10.0 and went straight to 10.1. @bulky-soccer-26729 could you maybe explain to me wthat exactly does
negated
mean?
a
yeah negated here just means things that don't match this query. so in your example, if you have
removed = true
as the condition and
negated = true
it would basically say give me everything where
removed = false
(or more explicitly, everything where
removed
is not
true
)
e
okay, then in terms of this explanation, on 10.1 the queries for "everything where
removed
is
true
)" and for "everything where
removed
is not
true
)" give me the same amount of entities🤔 and on 9.6.1. it used to be different amount of entities