People aborrero

aborrero (arturo)
Operations Engineer at Wikimedia Cloud Services Team

Projects

Calendar

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2017, 12:19 PM (26 w, 1 d)
Availability
Available
IRC Nick
arturo
LDAP User
Arturo Borrero Gonzalez
MediaWiki User
ABorrero (WMF)

I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Operations Engineer as part of the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.

You may found me in some FLOSS projects, like Netfilter and Debian.

Recent Activity
View All

Today

Addshore awarded T192892: Request creation of wikibase-registry VPS project a Love token.
Wed, Apr 25, 8:09 AM · Cloud-VPS (Project-requests)

Yesterday

Daniel_Mietchen awarded T192892: Request creation of wikibase-registry VPS project a Like token.
Tue, Apr 24, 4:35 PM · Cloud-VPS (Project-requests)
aborrero closed T192892: Request creation of wikibase-registry VPS project as Resolved.

Project created with default quota.

Tue, Apr 24, 4:27 PM · Cloud-VPS (Project-requests)
aborrero closed T192892: Request creation of wikibase-registry VPS project, a subtask of T192785: Create a discussion around finding different connected wikibase instances, as Resolved.
Tue, Apr 24, 4:27 PM · Wikidata, Federated-Wikibase-Workshops
aborrero updated subscribers of T181523: labtest puppetmaster is not working for clients.

We could be doing something wrong. From an IRC conversation with @jcrespo @akosiaris and @Vgutierrez we reached these conclusions:

Tue, Apr 24, 10:25 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.
In T181523#4151521, @Andrew wrote:

/var/lib/puppet/server/ssl/certs/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_pubkey.pem

/var/lib/puppet/server/ssl/private_keys/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_privkey.pem

Tue, Apr 24, 9:17 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic

Mon, Apr 23

aborrero added a comment to T192162: Set up proper apt pinning for Mitaka on Jessie, upgrade Jessie hosts to Mitaka.

To be fair, having more openstack releases in jessie-backports seems unlikely at this point. But in the other hand, having specific version pinning seems more robust (at the cost of having to update version by hand in case of changes)

Mon, Apr 23, 9:05 AM · Patch-For-Review, Cloud-Services

Fri, Apr 20

aborrero added a comment to T180462: puppet-compiler issue with CloudVPS instances.

Ideally we would get https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ to be able to compile catalog for VM instances in Cloud VPS (for example, toolforge nodes).

Fri, Apr 20, 1:05 PM · Cloud-VPS, cloud-services-team

Wed, Apr 18

aborrero added a comment to T192162: Set up proper apt pinning for Mitaka on Jessie, upgrade Jessie hosts to Mitaka.
In T192162#4137302, @Andrew wrote:

My other question is -- does pinning to backports mean that we will always get the mitaka packages? Or will they update the version available in backports at some point in the future?

Wed, Apr 18, 3:41 PM · Patch-For-Review, Cloud-Services
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

By copying the /var/lib/puppet/ssl/crl.pem file from a VM in main I can run puppet. Now the issue is the catalog itself. The VM in labtest has no name, so there is no catalog for it:

Wed, Apr 18, 10:09 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

I was able to get a puppet run almost working by sudo mv /var/lib/puppet/ssl/certs/Puppet_Internal_CA.pem /var/lib/puppet/ssl/certs/ca.pem.

Wed, Apr 18, 9:58 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic

Fri, Apr 13

aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

I turned on debug log in apache2 in labtestpuppetmaster. My testing client is 10.196.16.177 (file /var/log/apache2/error.log):

Fri, Apr 13, 1:13 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

Ok, more tests. These are from a VM in labtest.

Fri, Apr 13, 11:46 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic

Thu, Apr 12

aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

Weird stuff. I'm trying to understand what is happening here, comparing VM<-->puppetmaster in labtest and main deployments.

Thu, Apr 12, 1:12 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

I was finally able to jump by SSH into a test machine by means of:

Thu, Apr 12, 9:36 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero renamed T192059: CloudVPS: VMs created with non-allowed characters in the hostname fail to be autosigned by puppet from Labs VMs created with non-allowed characters in the hostname fail to be autosigned by puppet to CloudVPS: VMs created with non-allowed characters in the hostname fail to be autosigned by puppet.
Thu, Apr 12, 9:01 AM · Horizon, cloud-services-team

Wed, Apr 11

aborrero merged task T191814: labtest: instance creation via horizon into T181523: labtest puppetmaster is not working for clients.
Wed, Apr 11, 11:35 AM · cloud-services-team (Kanban)
aborrero merged T191814: labtest: instance creation via horizon into T181523: labtest puppetmaster is not working for clients.
Wed, Apr 11, 11:35 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.
In T181523#4120747, @Andrew wrote:

That said, it might be worth trying to duplicate the server certs between labpuppetmaster1001 and labtestpuppetmaster2001, if they aren't already.

Wed, Apr 11, 11:25 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic

Tue, Apr 10

aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

After replacing the puppet.conf path to other certificates, I create some random VM with no luck:

Tue, Apr 10, 10:31 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T181523: labtest puppetmaster is not working for clients.

the labtestpuppetmaster1001.wikimedia.org server might not be using the right certificate:

Tue, Apr 10, 9:41 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
aborrero added a comment to T191814: labtest: instance creation via horizon.

The issue with SSH might be that puppet fails the first run at instance creation, so SSH keys are not properly configured.

Tue, Apr 10, 8:49 AM · cloud-services-team (Kanban)
aborrero triaged T191814: labtest: instance creation via horizon as Normal priority.
Tue, Apr 10, 8:39 AM · cloud-services-team (Kanban)

Mon, Apr 9

aborrero added a comment to T191814: labtest: instance creation via horizon.

BTW created https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/testing_deployment

Mon, Apr 9, 4:57 PM · cloud-services-team (Kanban)
aborrero added a comment to T191814: labtest: instance creation via horizon.

I'm now part of a test project in labtest and can login to horizon after creating a wikitech account and setting up 2FA.

Mon, Apr 9, 4:42 PM · cloud-services-team (Kanban)
aborrero created T191814: labtest: instance creation via horizon.
Mon, Apr 9, 4:41 PM · cloud-services-team (Kanban)
aborrero added a comment to T188266: labtestn to Mitaka on Jessie.
In T188266#4116554, @chasemp wrote:

@aborrero can you reimage labtestservices2002.wikimedia.org as jessie as part of this? There is a lot of work to be done but I think post that this can be closed.

Mon, Apr 9, 12:55 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Fri, Apr 6

aborrero added a comment to T159254: Blacklist apache from unattended-upgrades on tools puppetmaster.

We just got hit by this again.

Fri, Apr 6, 11:35 AM · Patch-For-Review, Cloud-Services

Thu, Apr 5

aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.
In T190512#4108528, @aborrero wrote:

The /srv/carbon/whisper/archived_metrics directory is 1452250952 Bin size, 1.4 TB. Perhaps some purging mechanism should exists.

Thu, Apr 5, 1:02 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

The /srv/carbon/whisper/archived_metrics directory is 1452250952 Bin size, 1.4 TB. Perhaps some purging mechanism should exists.

Thu, Apr 5, 12:59 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Wed, Apr 4

aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

The rsync has run several times already, but something weird is happening, since it reports complete, but both servers don't have the same data?

Wed, Apr 4, 5:08 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Tue, Apr 3

aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

I added ssh forced-command to the SSH key per @fgiunchedi suggestion.

Tue, Apr 3, 3:16 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

This should be ready to go. I tested manually the rsync, and it works. Let's wait a couple of hours in case cron reports any misbehavior.

Tue, Apr 3, 12:08 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

I'm sure the relay path is the way to go, but perhaps a bit over complicated for what we are looking for right now. Let's stick to rsync for whisper files and we can follow-up eventually with a proper clusterization of this deployment, including perhaps a shared service URL for clients.

Tue, Apr 3, 10:30 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Mon, Apr 2

aborrero removed a watcher for cloud-services-team: aborrero.
Mon, Apr 2, 2:47 PM
aborrero removed a member for PAWS: aborrero.
Mon, Apr 2, 2:46 PM

Wed, Mar 28

aborrero updated subscribers of T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

@fgiunchedi suggests we better switch to use carbon relays rather than rsync.

Wed, Mar 28, 4:28 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Mar 26 2018

aborrero added a comment to T189871: labmon1002 as cold standby for labmon1001.

Another cronspam:

Mar 26 2018, 10:26 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Mar 23 2018

aborrero added a comment to T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.

That last patch caused a change in labmon1001 catalog:

Mar 23 2018, 5:50 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero updated the task description for T190512: labmon: syncronize whisper files between labmon1001 and labmon1002.
Mar 23 2018, 5:33 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero closed T190515: labmon: persist apache/httpd configuration as Resolved.
Mar 23 2018, 5:27 PM · Patch-For-Review, cloud-services-team, Cloud-VPS
aborrero closed T190515: labmon: persist apache/httpd configuration, a subtask of T189871: labmon1002 as cold standby for labmon1001, as Resolved.
Mar 23 2018, 5:27 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero triaged T190515: labmon: persist apache/httpd configuration as Normal priority.
Mar 23 2018, 1:31 PM · Patch-For-Review, cloud-services-team, Cloud-VPS
aborrero closed T190312: labmon: missing puppet configuration for keystone integration as Resolved.
Mar 23 2018, 1:16 PM · Patch-For-Review, cloud-services-team, Cloud-VPS
aborrero closed T190312: labmon: missing puppet configuration for keystone integration, a subtask of T189871: labmon1002 as cold standby for labmon1001, as Resolved.
Mar 23 2018, 1:16 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero triaged T190512: labmon: syncronize whisper files between labmon1001 and labmon1002 as Normal priority.
Mar 23 2018, 12:52 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero added a comment to T190312: labmon: missing puppet configuration for keystone integration.

I tried adding profile::openstack::base::clientlib to role::labs::monitoring, but it pulls too much unrelated packages:

Mar 23 2018, 12:50 PM · Patch-For-Review, cloud-services-team, Cloud-VPS

Mar 21 2018

aborrero triaged T190312: labmon: missing puppet configuration for keystone integration as Normal priority.
Mar 21 2018, 5:26 PM · Patch-For-Review, cloud-services-team, Cloud-VPS
madhuvishy awarded T190182: implement renewed *.tools.wmflabs.org cert/key pair a Mountain of Wealth token.
Mar 21 2018, 5:19 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge, Operations
aborrero closed T190182: implement renewed *.tools.wmflabs.org cert/key pair as Resolved.
Mar 21 2018, 5:19 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge, Operations

Mar 20 2018

aborrero added a comment to T190135: toolforge: puppet broken in the fleet due to a font package.

@MoritzMuehlenhoff reverted the chages and the cluster is back to normal puppet state.

Mar 20 2018, 12:03 PM · Patch-For-Review, cloud-services-team, Toolforge
aborrero added a comment to T190135: toolforge: puppet broken in the fleet due to a font package.

The patch solver the first issue, but now other appeared:

Mar 20 2018, 11:23 AM · Patch-For-Review, cloud-services-team, Toolforge
aborrero created T190135: toolforge: puppet broken in the fleet due to a font package.
Mar 20 2018, 11:10 AM · Patch-For-Review, cloud-services-team, Toolforge

Mar 19 2018

aborrero updated subscribers of T190014: https://recommend.wmflabs.org is down.

A reboot didn't solve the issue.

Mar 19 2018, 12:08 PM · VPS-Projects, Services (watching), Recommendation-API, ContentTranslation, Research

Mar 16 2018

aborrero added a comment to T189584: wikitech static monitoring changes.

Hey @Andrew sorry for not handling this in a timely manner. Is this solved now? I don't see anything weird in icinga.

Mar 16 2018, 6:15 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero triaged T184259: labspuppetmaster1001: have consistency in owner of git repos as Low priority.
Mar 16 2018, 6:07 PM · cloud-services-team (Kanban), Patch-For-Review
aborrero triaged T184261: labspuppetmaster1001: alert if git sync fails as Low priority.
Mar 16 2018, 6:07 PM · cloud-services-team (Kanban)
aborrero added a comment to T183249: Reimage and upgrade to stretch all dbproxies.

I just update wikireplica DNS records:

Mar 16 2018, 2:16 PM · Patch-For-Review, Operations, DBA
aborrero added a comment to T189871: labmon1002 as cold standby for labmon1001.

There are several possible approaches for this:

  • sync data between the two hosts by means of a manual process (rsync, cron, etc)
  • having some shared filesystem between the 2 servers for the datasets
  • collect data in both at the same time (by means of a virtual IP address or something?)
Mar 16 2018, 1:06 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Mar 15 2018

aborrero added a comment to T189165: Request creation of WCDO VPS project.

Project has been created.

Mar 15 2018, 4:55 PM · Cloud-VPS (Project-requests)
aborrero closed T189682: reimage labtestmetal2001 with virt compatible partman as Jessie as Resolved.
aborrero@compiler02:~$ grep lsbdistcodename /var/lib/catalog-differ/puppet/yaml/facts/labtestmetal2001.codfw.wmnet.yaml 
lsbdistcodename: jessie
Mar 15 2018, 4:34 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
aborrero closed T189682: reimage labtestmetal2001 with virt compatible partman as Jessie, a subtask of T188266: labtestn to Mitaka on Jessie, as Resolved.
Mar 15 2018, 4:34 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Mar 14 2018

aborrero added a comment to T189682: reimage labtestmetal2001 with virt compatible partman as Jessie.
In T189682#4050747, @aborrero wrote:

the current partitioning layout of labtestmetal2001 has a /var/lib/nova/instances partition with xfs over lvm and software raid. Perhaps we can just keep the same layout.

Mar 14 2018, 5:48 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
aborrero added a comment to T189682: reimage labtestmetal2001 with virt compatible partman as Jessie.

the current partitioning layout of labtestmetal2001 has a /var/lib/nova/instances partition with xfs over lvm and software raid. Perhaps we can just keep the same layout.

Mar 14 2018, 5:39 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
aborrero added a comment to T189584: wikitech static monitoring changes.

icinga error message is 140555763406480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:827: which isn't very useful.

Mar 14 2018, 5:27 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T189682: reimage labtestmetal2001 with virt compatible partman as Jessie.

Related patch https://gerrit.wikimedia.org/r/c/419258

Mar 14 2018, 1:40 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
aborrero claimed T189682: reimage labtestmetal2001 with virt compatible partman as Jessie.
Mar 14 2018, 1:40 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
aborrero updated the task description for T184259: labspuppetmaster1001: have consistency in owner of git repos.
Mar 14 2018, 10:18 AM · cloud-services-team (Kanban), Patch-For-Review

Mar 13 2018

aborrero added a comment to T189584: wikitech static monitoring changes.

Icinga seems to accept our new config:

Mar 13 2018, 6:41 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero closed T187189: toolforge: cronspam by phpsessionclean as Resolved.
Mar 13 2018, 5:51 PM · cloud-services-team, Cloud-Services
aborrero added a comment to T187189: toolforge: cronspam by phpsessionclean.
In T187189#4046551, @chasemp wrote:

@aborrero is this task still actionable?

Mar 13 2018, 5:44 PM · cloud-services-team, Cloud-Services
aborrero claimed T184259: labspuppetmaster1001: have consistency in owner of git repos.

I would simply put somewhere in puppet that this should be gitpuppet:gitpuppet, so each puppet run will ensure these owners. Does this sound crazy?

Mar 13 2018, 4:22 PM · cloud-services-team (Kanban), Patch-For-Review
aborrero claimed T184261: labspuppetmaster1001: alert if git sync fails.

Cron send by email all the stdout/stderr by default, even if the return code is 0. So there is no simple way to send by email only stderr if the git-sync-upstream script failed.

Mar 13 2018, 1:12 PM · cloud-services-team (Kanban)
aborrero added a comment to P6836 (An Untitled Masterwork).
aborrero@tools-clushmaster-01:~$ clush -w @all "grep -q ubuntu /etc/os-release || exit 0; egrep mount\|nfs /var/log/dpkg.log | grep upgrade || true"
tools-checker-02.tools.eqiad.wmflabs: 2018-03-12 12:51:28 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-checker-02.tools.eqiad.wmflabs: 2018-03-12 12:51:35 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-checker-02.tools.eqiad.wmflabs: 2018-03-12 12:52:10 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-exec-1411.tools.eqiad.wmflabs: 2018-03-12 12:51:55 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1411.tools.eqiad.wmflabs: 2018-03-12 12:51:59 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-grid-shadow.tools.eqiad.wmflabs: 2018-03-12 12:52:59 upgrade mount:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-grid-shadow.tools.eqiad.wmflabs: 2018-03-12 12:53:06 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-grid-shadow.tools.eqiad.wmflabs: 2018-03-12 12:53:14 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-exec-1412.tools.eqiad.wmflabs: 2018-03-12 12:52:14 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1412.tools.eqiad.wmflabs: 2018-03-12 12:52:20 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1413.tools.eqiad.wmflabs: 2018-03-12 12:52:56 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1413.tools.eqiad.wmflabs: 2018-03-12 12:53:06 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1414.tools.eqiad.wmflabs: 2018-03-12 12:53:19 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1414.tools.eqiad.wmflabs: 2018-03-12 12:53:36 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1415.tools.eqiad.wmflabs: 2018-03-12 12:52:35 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1415.tools.eqiad.wmflabs: 2018-03-12 12:52:41 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1416.tools.eqiad.wmflabs: 2018-03-12 12:52:36 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1416.tools.eqiad.wmflabs: 2018-03-12 12:52:42 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1417.tools.eqiad.wmflabs: 2018-03-12 12:52:56 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1417.tools.eqiad.wmflabs: 2018-03-12 12:53:03 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1418.tools.eqiad.wmflabs: 2018-03-12 12:52:52 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1418.tools.eqiad.wmflabs: 2018-03-12 12:52:57 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1420.tools.eqiad.wmflabs: 2018-03-12 12:53:00 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1420.tools.eqiad.wmflabs: 2018-03-12 12:53:05 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1422.tools.eqiad.wmflabs: 2018-03-12 12:53:40 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1422.tools.eqiad.wmflabs: 2018-03-12 12:53:49 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1421.tools.eqiad.wmflabs: 2018-03-12 12:53:11 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1421.tools.eqiad.wmflabs: 2018-03-12 12:53:16 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1423.tools.eqiad.wmflabs: 2018-03-12 12:53:29 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1423.tools.eqiad.wmflabs: 2018-03-12 12:53:35 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1424.tools.eqiad.wmflabs: 2018-03-12 12:53:39 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1424.tools.eqiad.wmflabs: 2018-03-12 12:53:44 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1426.tools.eqiad.wmflabs: 2018-03-12 12:53:54 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1426.tools.eqiad.wmflabs: 2018-03-12 12:54:00 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1425.tools.eqiad.wmflabs: 2018-03-12 12:54:05 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1425.tools.eqiad.wmflabs: 2018-03-12 12:54:14 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1427.tools.eqiad.wmflabs: 2018-03-12 12:54:22 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1427.tools.eqiad.wmflabs: 2018-03-12 12:54:31 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1429.tools.eqiad.wmflabs: 2018-03-12 12:54:13 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1429.tools.eqiad.wmflabs: 2018-03-12 12:54:20 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1428.tools.eqiad.wmflabs: 2018-03-12 12:54:38 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1428.tools.eqiad.wmflabs: 2018-03-12 12:54:51 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1432.tools.eqiad.wmflabs: 2018-03-12 12:54:33 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1432.tools.eqiad.wmflabs: 2018-03-12 12:54:42 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1431.tools.eqiad.wmflabs: 2018-03-12 12:54:32 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1431.tools.eqiad.wmflabs: 2018-03-12 12:54:43 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1433.tools.eqiad.wmflabs: 2018-03-12 12:54:28 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1433.tools.eqiad.wmflabs: 2018-03-12 12:54:37 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1430.tools.eqiad.wmflabs: 2018-03-12 12:54:29 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1430.tools.eqiad.wmflabs: 2018-03-12 12:54:38 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1436.tools.eqiad.wmflabs: 2018-03-12 12:54:35 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1436.tools.eqiad.wmflabs: 2018-03-12 12:54:39 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1437.tools.eqiad.wmflabs: 2018-03-12 12:54:44 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1437.tools.eqiad.wmflabs: 2018-03-12 12:54:49 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1435.tools.eqiad.wmflabs: 2018-03-12 12:55:05 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1435.tools.eqiad.wmflabs: 2018-03-12 12:55:20 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1438.tools.eqiad.wmflabs: 2018-03-12 12:55:42 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1438.tools.eqiad.wmflabs: 2018-03-12 12:55:53 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1440.tools.eqiad.wmflabs: 2018-03-12 12:55:12 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1440.tools.eqiad.wmflabs: 2018-03-12 12:55:17 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-services-02.tools.eqiad.wmflabs: 2018-03-12 12:57:58 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-services-02.tools.eqiad.wmflabs: 2018-03-12 12:58:04 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-services-02.tools.eqiad.wmflabs: 2018-03-12 12:58:28 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-exec-1442.tools.eqiad.wmflabs: 2018-03-12 12:55:50 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1442.tools.eqiad.wmflabs: 2018-03-12 12:55:57 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1439.tools.eqiad.wmflabs: 2018-03-12 12:55:43 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1439.tools.eqiad.wmflabs: 2018-03-12 12:55:53 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1434.tools.eqiad.wmflabs: 2018-03-12 12:54:45 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1434.tools.eqiad.wmflabs: 2018-03-12 12:54:56 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: 2018-03-12 12:56:04 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: 2018-03-12 12:56:10 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1441.tools.eqiad.wmflabs: 2018-03-12 12:56:09 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-exec-1441.tools.eqiad.wmflabs: 2018-03-12 12:56:23 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1404.tools.eqiad.wmflabs: 2018-03-12 12:49:36 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1404.tools.eqiad.wmflabs: 2018-03-12 12:49:43 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1404.tools.eqiad.wmflabs: 2018-03-12 12:50:02 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: 2018-03-12 12:51:33 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: 2018-03-12 12:51:45 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: 2018-03-12 12:52:11 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-generic-1403.tools.eqiad.wmflabs: 2018-03-12 12:58:34 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1403.tools.eqiad.wmflabs: 2018-03-12 12:58:45 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1403.tools.eqiad.wmflabs: 2018-03-12 12:59:12 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: 2018-03-12 12:51:06 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: 2018-03-12 12:51:15 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: 2018-03-12 12:51:41 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: 2018-03-12 12:49:43 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: 2018-03-12 12:49:52 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: 2018-03-12 12:50:12 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: 2018-03-12 12:51:57 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: 2018-03-12 12:52:10 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: 2018-03-12 12:52:35 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: 2018-03-12 12:49:34 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: 2018-03-12 12:49:40 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: 2018-03-12 12:49:58 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: 2018-03-12 12:51:03 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: 2018-03-12 12:51:11 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: 2018-03-12 12:51:33 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: 2018-03-12 12:48:53 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: 2018-03-12 12:48:59 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: 2018-03-12 12:49:47 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: 2018-03-12 12:49:55 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: 2018-03-12 12:50:16 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: 2018-03-12 12:49:33 upgrade mount:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: 2018-03-12 12:49:42 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: 2018-03-12 12:49:57 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: 2018-03-12 12:51:39 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: 2018-03-12 12:51:50 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: 2018-03-12 12:52:15 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: 2018-03-12 12:50:00 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: 2018-03-12 12:50:06 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: 2018-03-12 12:50:30 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: 2018-03-12 12:50:35 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1402.tools.eqiad.wmflabs: 2018-03-12 12:58:25 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1402.tools.eqiad.wmflabs: 2018-03-12 12:58:36 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-generic-1402.tools.eqiad.wmflabs: 2018-03-12 12:59:05 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: 2018-03-12 12:50:42 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: 2018-03-12 12:50:47 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: 2018-03-12 12:49:54 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: 2018-03-12 12:49:59 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: 2018-03-12 12:50:51 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: 2018-03-12 12:50:54 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: 2018-03-12 12:50:54 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: 2018-03-12 12:50:59 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: 2018-03-12 12:51:48 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: 2018-03-12 12:51:55 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: 2018-03-12 12:51:55 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: 2018-03-12 12:52:03 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: 2018-03-12 12:51:06 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: 2018-03-12 12:51:11 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: 2018-03-12 12:51:28 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: 2018-03-12 12:51:33 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: 2018-03-12 12:51:19 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: 2018-03-12 12:51:24 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: 2018-03-12 12:51:24 upgrade mount:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: 2018-03-12 12:51:30 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.7 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: 2018-03-12 12:50:29 upgrade mount:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: 2018-03-12 12:50:42 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.6 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: 2018-03-12 12:51:03 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: 2018-03-12 12:51:06 upgrade mount:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: 2018-03-12 12:51:15 upgrade libmount1:amd64 2.20.1-5.1ubuntu20.3 2.20.1-5.1ubuntu20.9
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: 2018-03-12 12:51:37 upgrade nfs-common:amd64 1:1.2.8-6ubuntu1.1 1:1.2.8-6ubuntu1.2
Mar 13 2018, 11:40 AM

Mar 12 2018

aborrero closed T189018: Toolforge Iinstances (maybe only Jessie?) are having issues with NFS/LDAP as Resolved.

I think we can close this.

Mar 12 2018, 5:37 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending upgrades from stable:

Mar 12 2018, 5:01 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending upgrades from stretch-wikimedia:

Mar 12 2018, 4:53 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending upgrades from jessie-wikimedia:

Mar 12 2018, 4:32 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending upgrades from jessie-backports:

Mar 12 2018, 1:18 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Packages upgraded from trusty-upgrades:

Mar 12 2018, 1:08 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending package upgrades from trusty-wikimedia:

Mar 12 2018, 12:32 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge

Mar 8 2018

aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Pending *-tools package upgrades:

Mar 8 2018, 2:00 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero closed T187193: toolforge: pin key packages by version, a subtask of T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case, as Resolved.
Mar 8 2018, 10:56 AM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero closed T187193: toolforge: pin key packages by version as Resolved.
Mar 8 2018, 10:56 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge

Mar 7 2018

aborrero reopened T187193: toolforge: pin key packages by version as "Open".

We need to pin more nss/ldap/pam packages as seen in T189018

Mar 7 2018, 12:33 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero reopened T187193: toolforge: pin key packages by version, a subtask of T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case, as Open.
Mar 7 2018, 12:33 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge

Mar 6 2018

aborrero added a comment to T189018: Toolforge Iinstances (maybe only Jessie?) are having issues with NFS/LDAP.

Tested again, after madhu's operations:

Mar 6 2018, 4:21 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T189018: Toolforge Iinstances (maybe only Jessie?) are having issues with NFS/LDAP.

Testing to which jessie machines I can SSH and see my home:

Mar 6 2018, 4:08 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T189018: Toolforge Iinstances (maybe only Jessie?) are having issues with NFS/LDAP.

Today I was doing package upgrades on jessie nodes in toolforge, all recorded in SAL and in T188994.

Mar 6 2018, 3:45 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T189001: Global User Contributions complains about replica conf file.

Possible offending packages:

Mar 6 2018, 2:26 PM · Tool-Global-user-contributions
aborrero added a comment to T189001: Global User Contributions complains about replica conf file.

There are no pending puppet changes right now in this host:

Mar 6 2018, 2:05 PM · Tool-Global-user-contributions
aborrero added a comment to T189001: Global User Contributions complains about replica conf file.

By the time this issue started, I was doing package upgrades in toolforge (T188994), so this could be related to some package upgrade.

Mar 6 2018, 1:58 PM · Tool-Global-user-contributions
aborrero renamed T188994: toolforge: package upgrades as part of the new workflow from toolforge: upgrade to toolforge: package upgrades as part of the new workflow.
Mar 6 2018, 12:59 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188994: toolforge: package upgrades as part of the new workflow.

Upgrading packages in jessie oldstable nodes:

Mar 6 2018, 12:58 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero triaged T188994: toolforge: package upgrades as part of the new workflow as High priority.
Mar 6 2018, 12:50 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero closed T188911: toolforge: cleanup kernel packages, a subtask of T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case, as Resolved.
Mar 6 2018, 12:09 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero closed T188911: toolforge: cleanup kernel packages as Resolved.

We are good to go:

Mar 6 2018, 12:09 PM · cloud-services-team (Kanban), Toolforge

Mar 5 2018

aborrero added a comment to T188911: toolforge: cleanup kernel packages.

Now with ubuntu,

Mar 5 2018, 6:41 PM · cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188911: toolforge: cleanup kernel packages.

After the operations in the stretch machines, there are no pending kernel upgrades:

Mar 5 2018, 2:38 PM · cloud-services-team (Kanban), Toolforge
aborrero added a comment to T188911: toolforge: cleanup kernel packages.

For stretch machines, kernel package installed:

Mar 5 2018, 2:32 PM · cloud-services-team (Kanban), Toolforge
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL