Page MenuHomePhabricator

change labstore1001/1002 to cfq io scheduler
Closed, ResolvedPublic

Description

This has been staged on both labstore1001 and labstore1002. It did result in a reduction in load under normal circumstances, though it wasn't stark. The main benefit was that [deadline] did not honor the ionice framework which was being referred for rsync jobs but could actually take effect.

Another thing changed today is the I/O scheduler. An interesting note is if you look at /usr/local/sbin/storage-replicate for the replication backup logic. It is using ionice. It is not using nice otherwise. The storage layer has been set to [deadline] up to this point. deadline by design does not honor ionice at all, so this was part of our load spikes during replication. Since replication can go for 11+ hours it was an issue. It was supposed to be using idle cycles only but there was no way to honor that.

This can be seen like:

for d in ls /sys/block/ | grep s; do echo $d; cat /sys/block/sda/queue/scheduler; done

I changed it like:

for d in ls /sys/block/ | grep s; do echo $d; echo cfq > /sys/block/$d/queue/scheduler; echo cat /sys/block/$d/queue/scheduler; done

You could change it back like:

for d in ls /sys/block/ | grep s; do echo $d; echo deadline > /sys/block/$d/queue/scheduler; echo cat /sys/block/$d/queue/scheduler; done

Related Objects

Event Timeline

chasemp created this task.Feb 5 2016, 11:25 PM
chasemp updated the task description. (Show Details)
chasemp raised the priority of this task from to Normal.
chasemp added a project: Operations.
chasemp added subscribers: Aklapper, chasemp.

Change 270421 had a related patch set uploaded (by Rush):
grub: allow setting ioscheduler

https://gerrit.wikimedia.org/r/270421

Change 270421 merged by Rush:
grub: allow setting ioscheduler

https://gerrit.wikimedia.org/r/270421

Change 270431 had a related patch set uploaded (by Rush):
labstore1001: persist cfq ioscheduler

https://gerrit.wikimedia.org/r/270431

Change 270431 abandoned by Rush:
labstore1001: persist cfq ioscheduler

https://gerrit.wikimedia.org/r/270431

Change 270432 had a related patch set uploaded (by Rush):
labstore1001: persist cfq ioscheduler

https://gerrit.wikimedia.org/r/270432

Change 270432 merged by Rush:
labstore1001: persist cfq ioscheduler

https://gerrit.wikimedia.org/r/270432