Page MenuHomePhabricator
Create Task
Maniphest T150532

Upgrade qemu on ganeti clusters to 2.8
Closed, ResolvedPublic
Actions

Description

qemu on the ganeti clusters was upgraded to qemu 2.5 from jessie-backports to mitigate some AIO-related deadlocks/stalls. qemu 2.7 is now available in jessie-backports, we should investigate an update. Changelogs at http://wiki.qemu.org/ChangeLog/2.6 and http://wiki.qemu.org/ChangeLog/2.7

Details

SubjectRepoBranchLines +/-
Revert "Depool poolcounter1001"operations/mediawiki-configmaster+1 -1
Depool poolcounter1001operations/mediawiki-configmaster+1 -1
Customize query in gerrit

Event Timeline

• MoritzMuehlenhoff created this task.Nov 11 2016, 1:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2016, 1:15 PM
akosiaris added a comment.Nov 11 2016, 1:41 PM

From a quick look into the Changelogs, 2.7 has nothing backwards incompatible that should worry us, 2.6 does however. Specifically

The aio=native option to "-drive" now requires the cache=none option, instead of silently disabling itself for other cache modes. The newly invalid combination had been warning since QEMU 2.3.

We 've set this in the past both as a workaround for QEMU deadlocking issues as well as for performance reasons. Unfortunately it seems like 2.12 (our ganeti version) does not pass cache=none, neither do future versions.

Possible solutions:

  • Patch ganeti to adhere to QEMU 2.6 (this should happen anyway probably and upstreamed but maybe we don't want to wait)
  • Set disk_cache=none in all of our clusters. That would be a change from the default of cache=writeback but we are implicitly there anyway as the QEMU 2.6 changelog says.
  • Unset disk_aio=native and use the default of threads. That is an option for sure, but might incur some performance penalties

Of the the above, the disk_cache=none seems like the easiest path forward

fgiunchedi triaged this task as Medium priority.Nov 30 2016, 1:15 AM
akosiaris changed the task status from Open to Stalled.May 16 2017, 1:44 PM

I 've tested this and indeed it breaks running VMs as expected. I 've patched up ganeti and awaiting review. PR for this is up at https://github.com/ganeti/ganeti/pull/43. In the meantime I am gonna set the task as stalled

• MoritzMuehlenhoff added a comment.Nov 9 2017, 11:26 AM

Your patch is also missing in the Ganeti version in stretch, let's report it to the Debian BTS so that it can possibly be backported to a stretch point release?

akosiaris added a comment.Nov 9 2017, 12:54 PM

Yes there hasn't been any release since the time of that patch, so let's do that. Filed it in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881255

akosiaris changed the task status from Stalled to Open.Apr 19 2018, 12:21 PM

With cache=none being set in all cluster for unrelated reasons, this is now unblocked. In the meantime jessie-backports has upgrade to 2.8. Fortunately the changelog[1] does not have any worrying items in it. The upgrade will require a round of VM reboots, but otherwise looks ok. I 'll empty an eqiad ganeti host, upgrade to 2.8 and move a few VMs to it for testing.

[1] https://wiki.qemu.org/ChangeLog/2.8

• MoritzMuehlenhoff added a comment.Apr 19 2018, 1:22 PM

Or we could upgrade the Ganeti cluster to stretch? It provides qemu 2.8 out of the box.

akosiaris added a comment.Apr 19 2018, 3:32 PM
In T150532#4142647, @MoritzMuehlenhoff wrote:

Or we could upgrade the Ganeti cluster to stretch? It provides qemu 2.8 out of the box.

I 'd rather not couple the 2 upgrades. Both need to be done of course, but I was thinking first make sure we are ok with 2.8 and then schedule the stretch upgrade

Stashbot subscribed.Apr 20 2018, 7:37 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-20T07:37:56Z] <akosiaris> upgrade qemu on ganeti2006 to 1:2.8+dfsg-3~bpo8+1 and migrate mwdebug2001 to it T150532

akosiaris renamed this task from Upgrade qemu on ganeti clusters to 2.7 to Upgrade qemu on ganeti clusters to 2.8.Apr 24 2018, 10:37 AM

mwdebug2001 showed no problems, I 'll proceed with upgrading the entire codfw cluster. A full cluster VM reboot is to follow

Stashbot added a comment.Apr 24 2018, 10:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-24T10:39:14Z] <akosiaris> upgrade to qemu 2.8 on codfw ganeti cluster. T150532

Stashbot added a comment.Apr 24 2018, 10:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-24T10:39:29Z] <akosiaris> starting a very slow rolling reboot of all VMs on codfw ganeti cluster T150532

Stashbot added a comment.Apr 25 2018, 7:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T07:05:32Z] <akosiaris> starting a very slow rolling reboot of all VMs on codfw ganeti cluster, row_C nodegroup, excluding poolcounter1001 and puppetdb1001. T150532

gerritbot subscribed.Apr 25 2018, 9:52 AM

Change 428894 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/mediawiki-config@master] Depool poolcounter1001

https://gerrit.wikimedia.org/r/428894

gerritbot added a project: Patch-For-Review.Apr 25 2018, 9:52 AM

Change 428896 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/mediawiki-config@master] Revert "Depool poolcounter1001"

https://gerrit.wikimedia.org/r/428896

gerritbot added a comment.Apr 25 2018, 9:54 AM

Change 428897 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/mediawiki-config@master] Add poolcounter1003 to $wmfAllServices

https://gerrit.wikimedia.org/r/428897

Stashbot added a comment.Apr 25 2018, 12:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T12:47:09Z] <akosiaris> reboot puppetdb1001 for T150532

gerritbot added a comment.Apr 25 2018, 1:16 PM

Change 428894 merged by Alexandros Kosiaris:
[operations/mediawiki-config@master] Depool poolcounter1001

https://gerrit.wikimedia.org/r/428894

Stashbot added a comment.Apr 25 2018, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T13:19:14Z] <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: depool poolcounter1001 T150532 (duration: 01m 17s)

Stashbot added a comment.Apr 25 2018, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T13:40:46Z] <akosiaris> reboot poolcounter1001 for T150532

gerritbot added a comment.Apr 25 2018, 1:47 PM

Change 428896 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Depool poolcounter1001"

https://gerrit.wikimedia.org/r/428896

Stashbot added a comment.Apr 25 2018, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T13:49:48Z] <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: repool poolcounter1001 T150532 (duration: 01m 16s)

Stashbot added a comment.Apr 25 2018, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-25T14:34:12Z] <akosiaris> reboot bohrium T150532

akosiaris closed this task as Resolved.Apr 25 2018, 2:34 PM
akosiaris claimed this task.

And we are at qemu 2.8 and this can finally be closed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits