This message was deleted.
# troubleshootings
Slackbot
09/23/2022, 3:50 PMThis message was deleted.
👍 1
g
Gian Merlino
09/23/2022, 4:28 PMthe MSQ task runtime fetches input files one at a time and clears them when done
we're planning to move compaction to this runtime which will improve the behavior here (and improve the behavior in other ways too — it's faster/more reliable than the native batch ingest runtime)
if you want, right now you can use the MSQ runtime to do a form of "manual" compaction. for a table with no rollup it'd be like:
Copy code
REPLACE INTO <tbl>
OVERWRITE <some time range>
SELECT * FROM <tbl>
WHERE <same time range>
PARTITIONED BY <segment granularity>
CLUSTERED BY <any secondary partitioning dims>Gian Merlino
09/23/2022, 4:31 PMnote it still requires a good amount of local disk to store the output prior to generating output segments: we need this in order to do a resort of the data based on your the PARTITIONED BY CLUSTERED BY. so it does reduce the temp storage needs (no need to store both input and output) but is not a silver bullet as the output is still stored.
in future we will have an ability to use deep storage for temporary storage of the output data for the MSQ task runtime. that would be the silver bullet 🙂
d
Didip Kerabat
09/23/2022, 5:02 PMCan I configure MSQ to run on which MM tier?
Because our general pool MM and compaction MM have wildly different spec.
g
Gian Merlino
09/23/2022, 5:38 PMthe MSQ task engine is regular tasks with certain types, so you could use a worker select strategy
❤️ 1