https://datahubproject.io logo
Join Slack
Powered by
HI guys, i think i found a bug on ui ingestion. I ...
# troubleshoot
f
HI guys, i think i found a bug on ui ingestion. I start an ingestion at
Copy code
Runs at 02:20 am (America/New_York)
now, after 2 hours, in ui its look like its still running so i check the logs and its says that its done
Copy code
tables_scanned': 323,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'views_scanned': 337,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'entities_profiled': 327,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'filtered': [],
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'soft_deleted_stale_entities': [],
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'query_combiner': {'total_queries': 10107,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'uncombined_queries_issued': 4765,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'combined_queries_issued': 603,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'queries_combined': 6298,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'query_exceptions': 11}}
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
Sink (datahub-rest) report:
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
{'records_written': 775,
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'warnings': [],
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'failures': [],
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'downstream_start_time': datetime.datetime(2022, 2, 18, 7, 22, 37, 552032),
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'downstream_end_time': datetime.datetime(2022, 2, 18, 7, 35, 23, 470652),
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
 'downstream_total_latency_in_seconds': 765.91862}
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
{}
Info
2022-02-18 09:35:26.496 EET
acryl-datahub-actions
Pipeline finished with failures
also i run 2/3 igestion in paralele, and for some ingestion i see in ui status Succeeded/Failed
b
Hmm
It might be due to the parallelism - If you run these individually do you see this bug?
It could also be bottlenecked on CPU / Memory during the multiple runs - Are you able to check the docker container stats
f
the ingestion its done ( i can see on logs and also in datahaub data) but in ui the status running.
These are the stats. I dont see anything abnormal 😞
But i see a strange status on datahub-acryl-actions
and i also got this
Pod The node had condition: [MemoryPressure].
s
Evicted status happens when there are no requests and limits specified for a pod in kubernetes and it breaches default memory limits. Please see this great post from google about the details of what is causing this https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits Please add resources block in your helm values under 
datahub-actions
section
f
thx @square-activity-64562, do you have any memory/cpu values base on your experience?
i try this
Copy code
acryl-datahub-actions:
  resources:
    requestes:
      memory: "4GB"
      cpu: "400m"
    limits:
      memory: "10GB"
      cpu: "500m"
and helm upgrade --install datahub datahub/datahub -f valuea.yaml but dosent seens like it works
s
Do you get specific error in the actions pod? Do you have nodes with that much RAM in your cluster?
The exact requirements would depend on how many jobs are running in your managed ingestion and how much data is there
Did you try changing your managed ingestion to only run one ingestion at a time?
Can you post the charts for CPU memory now for actions?
f
I ramake the ingsetions ( to be one in the same time)
last 2 days
s
Not last 2 days. Since you made the change
Can you run an execution after you made the change and share if there are errors
f
i did the change today, soo , i think i will need to w8 for 1/2 days to see how its works now
right now i have 5 project, but in the future i will need to add over 20 😂😅
another options is to create a cronjob for every file and use linkedin/datahub-ingestion image
h
@square-activity-64562 - We have the application deployed on EKS via Helm charts. We are trying to upgrade to 0.8.32 version, when trying to upgrade pods are getting evicted. Cam across this thread. What is the suggested memory and cpu for each pod ?
s
Is it all pods or just actions pod that are getting evicted? Have you checked with your infra team why they are being evicted? Usually that happens when the cluster is running out of resources
h
Yes, looks like we had to provide more nodes and then were able to deploy the application without issue.
Open in Slack
PreviousNext
Powered by