Has anyone ever experienced what seems like a bad ...
# questions-and-troubleshooting
g
Has anyone ever experienced what seems like a bad partition? I have an ETL process that I run on partitions of a table and every once in a while I hit a partition that will make one of the CN nodes pop from OOM, it will do this on every rerun. Visually there appears to be no difference (
SHOW TABLETS FROM <table> PARTITION <partition>
below: Day before 'bad partition' - processed fine
Copy code
TabletId|BackendId|DataSize|RowCount|MinVersion|
--------+---------+--------+--------+----------+
52911   |[11225]  |1.7GB   |6435521 |0         |
52912   |[10001]  |1.7GB   |6432706 |0         |
52913   |[11208]  |1.7GB   |6430817 |0         |
52914   |[128726] |1.7GB   |6431317 |0         |
52915   |[133045] |1.7GB   |6429125 |0         |
52916   |[128405] |1.7GB   |6436122 |0         |
52917   |[128526] |1.7GB   |6433915 |0         |
52918   |[128556] |1.7GB   |6431447 |0         |
52919   |[11209]  |1.7GB   |6433770 |0         |
52920   |[11210]  |1.7GB   |6433959 |0         |
'bad partition' - OOMs 1/10 of the nodes every time it's run
Copy code
TabletId|BackendId|DataSize|RowCount|MinVersion|
--------+---------+--------+--------+----------+
52875   |[11225]  |1.7GB   |6418270 |0         |
52876   |[10001]  |1.7GB   |6418430 |0         |
52877   |[128526] |1.7GB   |6420669 |0         |
52878   |[128405] |1.7GB   |6423310 |0         |
52879   |[11208]  |1.7GB   |6427917 |0         |
52880   |[128726] |1.7GB   |6423061 |0         |
52881   |[133045] |1.7GB   |6422284 |0         |
52882   |[128556] |1.7GB   |6420007 |0         |
52883   |[11209]  |1.7GB   |6420614 |0         |
52884   |[11210]  |1.7GB   |6426465 |0         |
what's more curious is I've hit this in the past and gotten past it just by: 1. creating a backup table 2. copying bad partition to backup table 3. deleting bad partition in real table 4. copying partition back from backup table