be_cloud_native_compactions - Blank be_compactions...
# questions-and-troubleshooting
g
be_cloud_native_compactions - Blank be_compactions - Blank SHOW PROC *"/transactions/events/running"*; -- <-- lists our 16 INSERT_STREAMING and 80 LAKE_COMPACTION jobs that are all COMMITTED, have a prep and commit time but no Publish or Finish times. What is the process to roll these back or delete them? We are in emergency mode and must have a fix whether we roll forward or back does not matter but we need these cleared as no further updates are working.
k
the quickest way is to drop the partitions or tables that associate to the txns, it will invalidate the committed txns.
g
We will try that. Going forward, what is the best way to do a bulk update on a column and avoid all of this pain?
k
it depends on what's wrong with the committed txns, may need to find why from fe log and cn logs.
g
Ok we deleted all of the bad partitions and the hung running transactions went away - good! Then we tried the same update on a single partition and watched the transaction go from
SHOW PROC *"/transactions/events/running"
to*
SHOW PROC *"/transactions/events/finished"*
- also good! BUT now the
MAX_CS
score on that partition is 57 while all the others on that table are < 10. We tried running
ALTER TABLE <table_name> COMPACT <partition_name>;
but after running we didn't see anything in
SHOW PROC '/compactions'
and the MAX_CS score is unchanged. What gives?
compactions on
_statistics_.loads_history
and
_statistics_.predicate_columns
are being run frequently
but that's all i see in
SHOW PROC '/compactions'
Good news - transactions and compactions ran! Bad news - it took forever and I'm not sure why. I'm going to try and ship compaction related logs to Log Analytics (all CN.info would be too much) and try to collect the compaction related prometheus metrics as well.