https://datahubproject.io logo
Join Slack
Powered by
Team, Anyone faced an issue where the ingestions a...
# troubleshoot
b
Team, Anyone faced an issue where the ingestions are happening but the status of the executions do not reflect on the UI ? We are also trying to create the new Domains. They are also reflecting in UI. Any pointers ? Existing thread. https://datahubspace.slack.com/archives/CV2UXSE9L/p1675345635830569 Pasting here to get wider audience. appreciate your help.
a
Hey @bland-appointment-45659, looks like Chris is helping you in that thread!
b
I was not sure if we are to post in the UI or troubleshoot channel. So, added here mentioning it as an existing thread. We are still unable to resolve the issue. Appreciate if we can get help.
e
Hi @bland-appointment-45659 let me check what the status is here and get back to you
b
@echoing-airport-49548 We wiped off the entire data and recycled all the pods since it was a dev instance. After that it worked for a while and it got stuck again. We saw the error below and coincidentally the UI got stuck after this point. Do you know if they are related ?
Copy code
"failures": {"Stateful Ingestion": ["Fail safe mode triggered, entity difference percent:64.15525114155251 > '
                      'fail_safe_threshold:{self.stateful_ingestion_config.fail_safe_threshold}"]}
Cc @kind-sunset-55628
e
Got it, thanks for the error log, that’s helpful
Have you tried clicking “run” on the ingestion recipes? It’s a known bug that we’re fixing that sometimes it says “pending” when it just needs to be run, and it’s not actually stuck
b
the UI is kind of frozen. If I click on Run, it submits a request and we see the updates in action pod logs but the UI wont change its state. It will only show its last success or failed status.
b
@bland-appointment-45659 Are you able to cancel? Also which version of DataHub are you running? I know that this fail safe threshold was recently removed after it caused a lot of trouble
b
We dont see a running status itself. The ingestion just shows the last succeeded or failed (thats days old). We are using v0.9.2
b
Okay got it I'll have to check when but in new version of the CLI we've removed that check which will essentially get rid of this problem
b
We are using v0.9.2 with 0.9.1 as the CLI. What CLI can we use to avoid this issue ?
Copy code
- name: UI_INGESTION_DEFAULT_CLI_VERSION
              value: "0.9.1"
appreciate help !!
a
Hi @bland-appointment-45659, have you tried updating to 0.10.0, which is our most recent CLI release?
b
@astonishing-answer-96712 We had to use 0.9.2 because of other issues we are trying to resolve for later versions. Is there any code fix done for this issue in 0.10.0 ?
a
I believe a fix was merged yes!
b
We found that whenever GMS has a backlog of events to process UI does not refresh the job status . We could see that when all ingestions were complete, UI shows the latest status of the job runs. Whenever we ingest new data, GMS becomes blind. Is this something heard from others ? Any workarounds to make UI useful ? Any thoughts @astonishing-answer-96712 Thanks
e
It sounds like this might be a scaling issue
Have you tried devoting more resources to GMS?
b
We noticed that GMS reads from kafka queue when there are graph changes and at times its slow (one reason that there are multiple connections Removing and Adding from neo4J). We are trying standalone MAE and MCE now. I am still wondering why UI has to be dependent on data processing.
a
@bland-appointment-45659, how’s it going here- would you be interested in submitting some thoughts on detaching the ui from the processing in an RFC?
Open in Slack
PreviousNext
Powered by