For historical reasons, restbase100[7-9] have different hardware than either restbase100[1-6]. Without special efforts to adjust for the differing capacity, they are thus running out of disk space before other nodes do. They also perform significantly worse, with metrics like iowait consistently around 4-5% compared to about 0.5% on restbase100[1-6].
I think it is not worth spending the effort to work around these limitations. We should upgrade the hardware to match that of restbase100[1-6].
We did plan for this possibility when we purchased those boxes. They are dual CPU Dell machines with only one socket populated.
Upgrading them would entail:
- Adding a third SSD to match the disk space. This is the most urgent issue, and already tracked in T119659.
- Adding a second CPU + RAM. We did consider this when we purchased the hardware, but I'm not sure how expensive & labor intensive this would be. Of the two, upgrading RAM would probably be the more urgent one.