Wonder if <this> is related to my question about l...
# advice-data-ingestion
r
Wonder if this is related to my question about logs above?
z
Yeah tl;dr if you're using the tool at volume although they give you suggested instance size really you need to off-board log storage to an external service. Check out how to configure logging to S3 - this then takes a lot of the disk space it's eating up on the compute instance and allows for placement/retrieval in S3. I will note this S3 logging documentation is in the Kubernetes deployment section but I believe same principals apply.
r
Can we simply not delete the logs periodically?
z
I mean yes but who wants to do that lol But real talk - why opt for a setup that increases your manual maintenance requirements and intermittent failures? For long running instances generating high log volume would expect to see external log storage and archiving/retention policy.
r
We don’t need logs after 30 days, so following the KISS principle here
Where are the logs stored and how do we go about deleting them from a self-hosted instance (on ec2 for example)
So for now I’m running this cronjob
0 0 * * * find /var/lib/docker/volumes/airbyte_workspace/_data/ -maxdepth 1 -regex '/var/lib/docker/volumes/airbyte_workspace/_data/[0-9]+' -mtime +10 | xargs rm -rf