Page MenuHomePhabricator

Upgrade Nodepool to 0.1.1-wmf5 to reduce requests made to OpenStack API
Closed, ResolvedPublic

Description

Chase has identified that Nodepool causes an anomalous rate of queries to the OpenStack API. A low hanging fruit is requesting the list of floating IP for the tenant which we do not need at all T143943.

I have made a patch that shortcircuit the code.

We will want to upgrade Nodepool to 0.1.1-wmf5 which includes it.

Packages build and published at:

https://people.wikimedia.org/~hashar/debs/nodepool_0.1.1-wmf5/

Will want to do the upgrade during CI low traffic hours (eg when SF is not awake yet).

Event Timeline

I have refreshed the package on https://people.wikimedia.org/~hashar/debs/nodepool_0.1.1-wmf5/ . The .changes file missed the original tarball which is annoying when using reprepro (was T145797).

Mentioned in SAL (#wikimedia-operations) [2016-09-22T21:02:19Z] <mutante> imported nodepool_0.1.1-wmf5_amd64 into jessie-wikimedia (T145142)

Dzahn added a subscriber: Dzahn.Sep 22 2016, 9:03 PM
root@carbon:~/nodepool# diff -ru 4/ 5/
diff -ru 4/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py 5/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py
--- 4/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py	2015-05-09 14:44:05.000000000 +0000
+++ 5/usr/lib/python2.7/dist-packages/nodepool/provider_manager.py	2016-09-20 11:41:20.000000000 +0000
@@ -602,7 +602,15 @@
         # This will either get the server or raise an exception
         server = self.getServerFromList(server_id)
 
-        if self.hasExtension('os-floating-ips'):
+        # WMF HACK: stop listing floating IPs
+        #
+        # Since Wikimedia setup does not use floating/public IPs, we never need
+        # to clean or list them.
+        # Short circuit to prevent generating useless ListFloatingIPsTask
+        #
+        # https://phabricator.wikimedia.org/T143943
+        #
+        if False and self.hasExtension('os-floating-ips'):
             for ip in self.listFloatingIPs():
                 if ip['instance_id'] == server_id:
                     self.log.debug('Deleting floating ip for server %s' %
Binary files 4/usr/share/doc/nodepool/changelog.Debian.gz and 5/usr/share/doc/nodepool/changelog.Debian.gz differ
root@carbon:/srv/wikimedia/incoming# reprepro ls nodepool
nodepool | 0.1.1-wmf4 |  jessie-wikimedia | amd64
root@carbon:/srv/wikimedia/incoming# reprepro -C thirdparty include jessie-wikimedia nodepool_0.1.1-wmf5_amd64.changes
Exporting indices...
Deleting files no longer referenced...
root@carbon:/srv/wikimedia/incoming# reprepro ls nodepool
nodepool | 0.1.1-wmf5 |  jessie-wikimedia | amd64, source
hashar closed this task as Resolved.Sep 22 2016, 9:10 PM
hashar reassigned this task from hashar to Dzahn.

I stopped Nodepool. Daniel apt-get installed. Done.

Monitoried the service, it is running all as expected.
From now on, Nodepool should no more queries for a list of floating IP whenever it has to delete a server \O/