https://datahubproject.io logo
Join Slack
Powered by
Hi guys, Tried to ingest some metadata from postgr...
# troubleshoot
b
Hi guys, Tried to ingest some metadata from postgres (+ table profiling, not column). the db has 7 schemas, and 72 tables overall. After running 28h~ it failed on 'no disk space'. Can I estimate how much disk space an ingestion should take? how much of it is the profiling part? I know we don't have exact numbers, but approximations will be fine so i'll be able to continue. Thank you 🙂
✅ 1
d
Where did you see this no disk space? On server side or ingestion side?
b
On ingestion side, it failed on:
Copy code
packages/acryl/executor/execution/sub_process_ingestion_task.py", line 212, in execute\n'
           '    full_log_file.close()\n'
           'OSError: [Errno 28] No space left on device\n']}
and there isn't actually space left on disk, just checked
It's the only ingestion i've done so far (except 8 elastic indices)
So I either need much more space, or it's something else 😞
When ingesting the data, does it store all data from the ingestion on a volume, and then proceeds to ingest from volume to the dbs? or does it do it table-by-table
d
I think it fails on writing logs to disk.
How much space do you have for action pod?
b
action pod? 🤔
d
Did you run ingestion from the UI, right?
b
Yes, manually
I can paste the whole stack trace if needed
d
If you run it from k8s then there is a pod called action pod or from docker, the action container, and that is the place where the ingestion actually runs and it tried to write logs to disk.
b
I don't run on k8s (yet). running on a ec2 server with the quickstart build
And pasting here the full stack trace:
Copy code
RUN_INGEST - {'errors': [],
 'exec_id': 'd70fc0e1-9aa1-43a5-8ed1-4cda127df3d0',
 'infos': ['2022-12-12 14:06:20.795429 [exec_id=d70fc0e1-9aa1-43a5-8ed1-4cda127df3d0] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-12-13 20:16:54.619671 [exec_id=d70fc0e1-9aa1-43a5-8ed1-4cda127df3d0] INFO: Caught exception EXECUTING '
           'task_id=d70fc0e1-9aa1-43a5-8ed1-4cda127df3d0, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 187, in execute\n'
           '    await tasks.gather(read_output_task, report_progress_task, process_waiter_task)\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 175, in _report_progress\n'
           '    full_log_file.flush()\n'
           'OSError: [Errno 28] No space left on device\n'
           '\n'
           'The above exception was the direct cause of the following exception:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 210, in execute\n'
           "    raise RuntimeError(f'Something went wrong in the subprocess executor: {e}') from e\n"
           'RuntimeError: Something went wrong in the subprocess executor: [Errno 28] No space left on device\n'
           '\n'
           'During handling of the above exception, another exception occurred:\n'
           '\n'
           'OSError: [Errno 28] No space left on device\n'
           '\n'
           'During handling of the above exception, another exception occurred:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 123, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 212, in execute\n'
           '    full_log_file.close()\n'
           'OSError: [Errno 28] No space left on device\n']}
Execution finished with errors.
d
How much disk space your container have?
b
50gb
d
hmm, that’s a lot
b
I checked and looks like one of the volumes takes 32gb 😮
but it's name is not datahub_, but a guild
guid*
50gb is for the whole ec2 server
Checked volume/docker disc space with du-h: elastic - 7.2M mysql - 224M zk - 16K
d
can you check this for 
datahub-datahub-actions-1
as well?
b
it doesn't have a volume
these are the only volumes with 'datahub' on their name
and i don't have a docker with it's name
oh wait i do have
sry
Checked with some google. tried to do 'docker inspect -f "{{ .Mounts }} --docker action id--" it returned me an emplty list. When did it on elastic container it worked fine
Ok, looks like the volume with 32gb is the one mounted to the kafka broker container
seems legit as a lot of the folders in it are 'consumer offsets'
Why does it use kafka? shouldn't it use 'REST API' as the sink to my ingestions? (didn't put any sink config, but docs said REST is the default)
Tamas - do you have any idea what could be the problem? 🙏
d
can you check on the ec2 node how much disk space docker has?
Copy code
docker system df
b
message has been deleted
I expanded my disc space so i'll be able to load the site properly. Looks like new things i'm trying to add won't work (for example - creating glossary term / domain). I don't get an error (it says it created everything), but nothings shows, not even if I switch user / reload the page
d
Please, can you start a new thread about this issue?
b
Sure, will do. But what about the previous problem? i'm afraid it will take a lot of space next time i'll try to ingest something from postgrs/a different source
d
Please try it and let me know if it fails. I think the ingestion’s logfile was the one which caused your disk to run out of disk space. Hopefully now you will be fine.
b
So should I clear the volume? cause it still takes the same space
5 Views
Open in Slack
PreviousNext
Powered by