Chase has identified that Nodepool causes an anomalous rate of queries to the OpenStack API. A low hanging fruit is requesting the list of floating IP for the tenant which we do not need at all T143943.
I have made a patch that shortcircuit the code.
We will want to upgrade Nodepool to 0.1.1-wmf5 which includes it.
Packages build and published at:
https://people.wikimedia.org/~hashar/debs/nodepool_0.1.1-wmf5/
Will want to do the upgrade during CI low traffic hours (eg when SF is not awake yet).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T142952 2016-08-10 CI incident follow-ups | |||
| Resolved | hashar | T143943 Investigate use of Nodepool ListFloatingIPsTask | |||
| Resolved | Dzahn | T145142 Upgrade Nodepool to 0.1.1-wmf5 to reduce requests made to OpenStack API |
I have refreshed the package on https://people.wikimedia.org/~hashar/debs/nodepool_0.1.1-wmf5/ . The .changes file missed the original tarball which is annoying when using reprepro (was T145797).
Mentioned in SAL (#wikimedia-operations) [2016-09-22T21:02:19Z] <mutante> imported nodepool_0.1.1-wmf5_amd64 into jessie-wikimedia (T145142)
root@carbon:~/nodepool# diff -ru 4/ 5/ diff -ru 4/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py 5/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py --- 4/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py 2015-05-09 14:44:05.000000000 +0000 +++ 5/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py 2016-09-20 11:41:20.000000000 +0000 @@ -602,7 +602,15 @@ # This will either get the server or raise an exception server = self.getServerFromList(server_id) - if self.hasExtension('os-floating-ips'): + # WMF HACK: stop listing floating IPs + # + # Since Wikimedia setup does not use floating/public IPs, we never need + # to clean or list them. + # Short circuit to prevent generating useless ListFloatingIPsTask + # + # https://phabricator.wikimedia.org/T143943 + # + if False and self.hasExtension('os-floating-ips'): for ip in self.listFloatingIPs(): if ip['instance_id'] == server_id: self.log.debug('Deleting floating ip for server %s' % Binary files 4/usr/share/doc/nodepool/changelog.Debian.gz and 5/usr/share/doc/nodepool/changelog.Debian.gz differ
root@carbon:/srv/wikimedia/incoming# reprepro ls nodepool nodepool | 0.1.1-wmf4 | jessie-wikimedia | amd64 root@carbon:/srv/wikimedia/incoming# reprepro -C thirdparty include jessie-wikimedia nodepool_0.1.1-wmf5_amd64.changes Exporting indices... Deleting files no longer referenced... root@carbon:/srv/wikimedia/incoming# reprepro ls nodepool nodepool | 0.1.1-wmf5 | jessie-wikimedia | amd64, source
I stopped Nodepool. Daniel apt-get installed. Done.
Monitoried the service, it is running all as expected.
From now on, Nodepool should no more queries for a list of floating IP whenever it has to delete a server \O/