https://datahubproject.io logo
Join Slack
Powered by
For some reason, Datahub front displays as if it s...
# troubleshoot
m
For some reason, Datahub front displays as if it still has Tableau entities that were hard deleted from backend. It seems to concern only visual navigation (count of platform entities, filters, folders) and such — once you actually open a list, there’s no entities there. It also doesn’t display recent ingest request (most recent one was at least feb 8th)
Proof that there’s no tableau data on backend side. We tried hard deleting them again and got this message
Copy code
[2023-02-08 18:27:13,752] INFO     {datahub.cli.delete_cli:321} - Filter matched  dashboard entities of tableau. Sample: []
No urns to delete. Maybe you want to change entity_type=dashboard or platform=tableau to be something different?
e
Hey @magnificent-lock-58916 that seems weird, let me look into this for you
m
Sure, what else do you need?
e
I’m able to reproduce the issue so I don’t think I need anything else right now!
Hey @magnificent-lock-58916 did you issue a CLI delete against the Tableau data platform?
m
@crooked-optician-38646 can you help to answer from here?
c
@echoing-airport-49548 Hi, yes we did that. It worked fine with no errors. Retrying that action (just to check if removing worked correctly) brought the message: No urns to delete. Maybe you want to change entity_type=dashboard or platform=tableau to be something different? However, there are no changes on the frontend. Currently, we found this error message in our frontend pod:
Copy code
2023-02-13 07:06:29,012 [application-akka.actor.default-dispatcher-26] ERROR p.api.http.DefaultHttpErrorHandler - 

! @80i7pmbba - Internal server error, for (POST) [/api/v2/graphql] ->
 
play.api.UnexpectedException: Unexpected exception[CompletionException: akka.http.scaladsl.model.EntityStreamException: Entity stream truncation. The HTTP parser was receiving an entity when the underlying connection was closed unexpectedly.]
Here is our elasticsearch version:
Copy code
{
  "name": "elk-elasticsearch-master-1",
  "cluster_name": "elastic",
  "cluster_uuid": "gXc4z-hgTRC9gFyS50mkRQ",
  "version": {
    "number": "7.17.2",
    "build_flavor": "default",
    "build_type": "tar",
    "build_hash": "de7261de50d90919ae53b0eff9413fd7e5307301",
    "build_date": "2022-03-28T15:12:21.446567561Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}
Copy code
2023-02-13 10:44:22.133 GMT [1] LOG:  database system is ready to accept connections
2023-02-13 10:45:12.838 GMT [116] ERROR:  could not serialize access due to concurrent update
2023-02-13 10:45:12.838 GMT [116] STATEMENT:  update metadata_aspect_v2 set metadata=$1, createdOn=$2, createdBy=$3, createdFor=$4, systemmetadata=$5 where urn=$6 and aspect=$7 and version=$8
2023-02-13 10:45:14.467 GMT [117] ERROR:  could not serialize access due to concurrent update
2023-02-13 10:45:14.467 GMT [117] STATEMENT:  update metadata_aspect_v2 set metadata=$1, createdOn=$2, createdBy=$3, createdFor=$4, systemmetadata=$5 where urn=$6 and aspect=$7 and version=$8
2023-02-13 10:47:20.507 GMT [117] ERROR:  could not serialize access due to concurrent update
2023-02-13 10:47:20.507 GMT [117] STATEMENT:  update metadata_aspect_v2 set metadata=$1, createdOn=$2, createdBy=$3, createdFor=$4, systemmetadata=$5 where urn=$6 and aspect=$7 and version=$8
That is from our PostgresDB
p
Actual problem and for us.
m
@echoing-airport-49548 Hi, can you tell if someone already looking into this? This issue somewhat paralyses our data governance processes 😢
@astonishing-answer-96712 Hello, do you know anyone who potentially could help us? I see Aditya is on vacation, but have trouble understanding how to relay the issue to someone else
a
CC: @bulky-soccer-26729
p
up this
a
Hi, so just to double check- the issue here is that you’re still seeing old state /deleted objects in tableau?
so DataHub is out of sync?
m
There’s github issue linked above and yeah, Datahub just stop being out of sync. Especially most likely to happen after ingesting Tableau
especially if there’s a lot of metadata to ingest
a
@gray-shoe-75895 - any ideas here?
g
So it seems like there’s multiple things going on 1. clicking run ingestion -> nothing happening: this probably means that the datahub-actions pod wasn’t running or was otherwise unhealthy 2. nothing showing up in search + datahub delete not working: seems like an indexing issue. it sounds like datahub-gms / elasticsearch got overwhelmed and started dropping stuff. could you try running the datahub reindexing job to see if that fixes the issue? another possible variable is that this is in helm and appears to be using neo4j
p
reindex solves the problem with a 50% chance (sometimes yes sometimes no), the problem is that even if it solves it when you restart the index on the scoreboard. We were also inclined to think that the elk didn't respond in time(??) increasing resources for the elk didn't have a positive effect. The main problem is most likely that gms has no time(?) to read/receive data from the elk and goes to the database with a crooked request (we will try to show an example in the near future). Yes we deployed from helm and use neo4j, how to look for botlneck in it?
a
We’re working on some changes to GMS to hopefully fix timeout issues- I’ll forward this to the team as an example. Thank you!
6 Views
Open in Slack
PreviousNext
Powered by