Using Tableau ingestion and having an interesting ...
# troubleshootm
magnificent-lock-58916
12/15/2022, 8:46 AMUsing Tableau ingestion and having an interesting trouble…
Before I explain it in detail, worth noting that we automatically delete everything related to Tableau from Datahub before each ingest execution. We do this since currently Datahub doesn’t support stateful ingestion for Tableau and we need our meta-data be up to date.
So, the problem is that scheduled automatic ingestion only ingest one single Tableau folder called “Draft”. And executing it manually somehow ingest everything we need. What can be the reason behind this problem, how to fix it? Manually ingesting everyday is kinda troublesome
Another problem we experience only with Tableau ingestion is that after ingestion it takes DataHub some time before it displays every ingested entity. For example, after execution it shows only 20 Tableau entities, after some time 150 entities, and after another time 300. In total we have 1.7k Tableau entities, could this quantity be somehow related to this problem?
d
dazzling-judge-80093
12/15/2022, 9:57 AMWith the latest release we support stateful ingestion with Tableau connector.
dazzling-judge-80093
12/15/2022, 9:58 AMHow much time does it take to show up the entities? Small latency is expected.
m
magnificent-lock-58916
12/15/2022, 10:21 AMOh, it is? In that case, you can probably mark my feature request as done then https://feature-requests.datahubproject.io/p/tableau-stateful-ingestion
magnificent-lock-58916
12/15/2022, 10:22 AM@crooked-optician-38646 FYI, we have slightly more reasons to upgrade DataHub after all. Maybe you could ask your questions regarding that
Also, could you please answer the question about latency above?
c
crooked-optician-38646
12/15/2022, 10:30 AMHi Guys!
Approximately, It takes 10 minutes
m
modern-artist-55754
12/16/2022, 2:31 AM@magnificent-lock-58916 the reason you see the number of entities growing is that it takes sometime for ElasticSearch to index all the new entities. For large number of entities it can take some number of minutes (my tableau ingestion take 15mins to have frontend show up correctly with 13k entities)
m
magnificent-lock-58916
12/16/2022, 4:05 AMOk, thank you!
It’s the least bothersome of two problems though. Automatic ingestion still ingests only one tableau folder, while manual ingests everything…very strange
m
modern-artist-55754
12/16/2022, 4:05 AMcan you show the config?
modern-artist-55754
12/16/2022, 4:06 AMand what do you mean by automatic ingestion (UI Ingestion)?
m
magnificent-lock-58916
12/16/2022, 4:35 AMBy automatic I mean scheduled ingestion. And by manual I mean, well, manual, when I manually click “Run” in “Ingestion” tab.
Will this screenshot be enough? Sorry for all the blacks, I’m simply not sure that I should and shouldn’t display publicly
m
modern-artist-55754
12/16/2022, 4:37 AMyup that’s fine… Hm that’s a bit strange that the scheduled run only get one project
modern-artist-55754
12/16/2022, 4:37 AMdo you have the log of the run?
m
magnificent-lock-58916
12/16/2022, 4:43 AMSure
That’s the log of a scheduled ingestion which ingested only two entities
magnificent-lock-58916
12/16/2022, 4:45 AMAnd that’s the log of manual ingestion which….in the log UI it says it too only ingested two entities, but the strange thing is, it ingested everything that it should’ve
m
modern-artist-55754
12/16/2022, 4:47 AMhow long did you wait for the schedule run?
m
magnificent-lock-58916
12/16/2022, 4:48 AMLast one was 85.674s
magnificent-lock-58916
12/16/2022, 4:49 AMI did manual execution again just now and yup, log UI tells me it only ingested like 2-3 entities, while the actual counter is 718 and it keeps going up
d
dazzling-judge-80093
12/16/2022, 5:50 PM@bulky-soccer-26729 do you happen to know if this latency is normal?
2 Views