aborrero (arturo)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2017, 12:19 PM (12 w, 5 d)
Availability
Available
LDAP User
Arturo Borrero Gonzalez
MediaWiki User
ABorrero (WMF)

Recent Activity

Yesterday

aborrero added a comment to T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.

Related, as mentioned by @chasemp : https://gerrit.wikimedia.org/r/#/c/334203/

Fri, Jan 19, 2:15 PM · Toolforge, cloud-services-team
aborrero added a comment to T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.

In tools-cron-01 I can see a lot of failed logins in /var/log/auth.log, such as:

Fri, Jan 19, 2:04 PM · Toolforge, cloud-services-team
aborrero added a comment to T185314: Toolforge crontab command needs to validate output.

keywords enconding, UTF8 and zhuyifei all pointed me to this patch: https://gerrit.wikimedia.org/r/#/c/383770/

Fri, Jan 19, 1:57 PM · Patch-For-Review, Toolforge, cloud-services-team
aborrero updated subscribers of T185314: Toolforge crontab command needs to validate output.

I can't find this file in our puppet repo, I'm not sure where it is generated from, probably somewhere in /data/project/ato. Hey @bd808 any idea?

Fri, Jan 19, 1:44 PM · Patch-For-Review, Toolforge, cloud-services-team
aborrero added a comment to T185314: Toolforge crontab command needs to validate output.

The file seems to be /var/spool/cron/crontabs/tools.ato in tools-cron-01.tools.eqiad.wmflabs

Fri, Jan 19, 1:39 PM · Patch-For-Review, Toolforge, cloud-services-team
aborrero created T185314: Toolforge crontab command needs to validate output.
Fri, Jan 19, 12:49 PM · Patch-For-Review, Toolforge, cloud-services-team
aborrero triaged T179388: tools-docker-builder-05.tools.eqiad.wmflabs firewall rules don't allow ssh from tools-clushmaster-01.eqiad.wmflabs as Normal priority.
Fri, Jan 19, 12:46 PM · Toolforge, cloud-services-team
aborrero added a comment to T179388: tools-docker-builder-05.tools.eqiad.wmflabs firewall rules don't allow ssh from tools-clushmaster-01.eqiad.wmflabs.

Today I did a puppet run and saw the same for another instance:

Fri, Jan 19, 12:46 PM · Toolforge, cloud-services-team
aborrero added a comment to T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.

This seems unresolved as of today. I'll try to take a look as soon as possible.

Fri, Jan 19, 12:45 PM · Toolforge, cloud-services-team
aborrero claimed T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.
Fri, Jan 19, 12:43 PM · Toolforge, cloud-services-team
aborrero claimed T179388: tools-docker-builder-05.tools.eqiad.wmflabs firewall rules don't allow ssh from tools-clushmaster-01.eqiad.wmflabs.
Fri, Jan 19, 12:43 PM · Toolforge, cloud-services-team

Thu, Jan 18

chasemp awarded T178717: Upgrade wmcs instances and masters to puppet 4.8 a Like token.
Thu, Jan 18, 9:27 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero updated the task description for T178717: Upgrade wmcs instances and masters to puppet 4.8.
Thu, Jan 18, 4:29 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero closed T178717: Upgrade wmcs instances and masters to puppet 4.8 as Resolved.

All instances should now be running puppet 4.

Thu, Jan 18, 4:29 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero added a comment to T178717: Upgrade wmcs instances and masters to puppet 4.8.

I'm trying to accomplish this following the workflow we are developing in T181647

Thu, Jan 18, 11:30 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services

Wed, Jan 17

aborrero edited P6606 (An Untitled Masterwork).
Wed, Jan 17, 6:49 PM
aborrero added a comment to P6606 (An Untitled Masterwork).
tools-bastion-02.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-bastion-02.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-bastion-02.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-bastion-03.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-bastion-03.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-bastion-05.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-bastion-05.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-checker-01.tools.eqiad.wmflabs: hhvm:amd64/trusty-wikimedia 3.6.5+dfsg1-1+wm6 upgradeable to 3.12.7+dfsg-1+wmf1~trusty1
tools-checker-01.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-checker-01.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-checker-01.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-checker-02.tools.eqiad.wmflabs: hhvm:amd64/trusty-wikimedia 3.6.5+dfsg1-1+wm6 upgradeable to 3.12.7+dfsg-1+wmf1~trusty1
tools-checker-02.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-checker-02.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-checker-02.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-cron-01.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-cron-01.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1401.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1401.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1401.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1402.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1402.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1402.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1403.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1403.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1403.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1404.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1404.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1404.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1405.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1405.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1405.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1406.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1406.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1406.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1407.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1407.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1407.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1408.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1408.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1408.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1409.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1409.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1409.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1410.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1410.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1410.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-exec-1411.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1411.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1412.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1412.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1413.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1413.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1414.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1414.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1415.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1415.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1416.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1416.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1417.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1417.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1418.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1418.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1419.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1419.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1420.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1420.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1421.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1421.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1422.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1422.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1423.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1423.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1424.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1424.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1425.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1425.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1426.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1426.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1427.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1427.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1428.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1428.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1429.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1429.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1430.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1430.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1431.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1431.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1432.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1432.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1433.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1433.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1434.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1434.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1435.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1435.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1436.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1436.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1437.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1437.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1438.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1438.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1439.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1439.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1440.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1440.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1441.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1441.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1442.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-1442.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-grid-master.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-grid-master.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-grid-shadow.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-grid-shadow.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-mail.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-mail.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-mail.tools.eqiad.wmflabs: python-mwclient:all/trusty-wikimedia 0.8.1-1 upgradeable to 0.8.4-1
tools-services-02.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-services-02.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-services-02.tools.eqiad.wmflabs: python-mwclient:all/trusty-wikimedia 0.8.1-1 upgradeable to 0.8.4-1
tools-webgrid-generic-1401.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1401.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1401.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-generic-1402.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1402.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1402.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-generic-1403.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1403.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1403.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-generic-1404.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1404.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-generic-1404.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: python-paramiko:all/trusty-wikimedia 1.10.1-1git1build1 upgradeable to 1.15.2-1
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: puppet:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: puppet-common:all/trusty-wikimedia 3.8.5-2~bpo8trusty+2 upgradeable to 4.8.2-5~trusty1
Wed, Jan 17, 6:48 PM
aborrero added a comment to P6606 (An Untitled Masterwork).

Not good:

Wed, Jan 17, 6:42 PM
aborrero created P6606 (An Untitled Masterwork).
Wed, Jan 17, 6:31 PM
aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

So, this task seems complete. Perhaps we wait before closing it to know if the docs are in shape and the workflow really fulfill our use case.

Wed, Jan 17, 2:14 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Wed, Jan 17, 2:13 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge

Mon, Jan 15

aborrero added a comment to T184189: Cloud: Labvirt and instance reboots for Meltdown.

@MoritzMuehlenhoff reports in T184910 that there are servers just pending the reboot. Should that ticket be merged into this one?

Mon, Jan 15, 2:15 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)
aborrero triaged T180254: unattended-upgrades: upgrade packages from -updates suites as well as Low priority.
Mon, Jan 15, 2:12 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero triaged T184909: labvirt1021-1022 spam the dhcp server with requests as Normal priority.
Mon, Jan 15, 2:12 PM · Operations, cloud-services-team, Cloud-Services
aborrero closed T184413: CloudVPS: quota increase request for 'aborrero-test' project as Resolved.

Thanks @bd808, I did it:

Mon, Jan 15, 11:49 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Fri, Jan 12

aborrero triaged T184676: Increase the limit of VCPUs for the recommendation-api project as Normal priority.

We usually discuss quota modification request during our team meetings, the next one being 2018-01-16.

Fri, Jan 12, 5:21 PM · Cloud-VPS (Quota-requests)

Thu, Jan 11

aborrero closed T184604: tools: out of disk: tools-worker-1020 as Resolved.

After checking again the grafana board, all seems fine:

Thu, Jan 11, 12:18 PM · cloud-services-team (Kanban), Tools

Wed, Jan 10

aborrero added a comment to T184604: tools: out of disk: tools-worker-1020.

BTW logrotate seems to be not working as supposed to work:

Wed, Jan 10, 1:56 PM · cloud-services-team (Kanban), Tools
aborrero claimed T184604: tools: out of disk: tools-worker-1020.

After cleaning by hand those files (logrotate won't handle them) I tried with cordon and drain of tools-worker-1020 and restarting kubelet. Continuous log errors are being produced again.

Wed, Jan 10, 1:48 PM · cloud-services-team (Kanban), Tools
aborrero updated the task description for T184604: tools: out of disk: tools-worker-1020.
Wed, Jan 10, 1:06 PM · cloud-services-team (Kanban), Tools
aborrero created T184604: tools: out of disk: tools-worker-1020.
Wed, Jan 10, 1:05 PM · cloud-services-team (Kanban), Tools

Mon, Jan 8

aborrero added a comment to T182722: Ferm changes on the host node break networking for Kubernetes pods.

I was hoping that the ferm @preverse option may make sense for us here but there are a few problems with the idea:

  • seems only available in 2.4 and jessie has 2.2 available and I don't see anything in backports
tools-worker-1018:~$ dpkg -s ferm | grep Version

Version: 2.2-3

  • preserve does not ignore a table/chain combo here -- it dumps and restores it. This is very racy for our setup as anything scheduled in that window would in theory be orphaned. We want an option that doesn't exist as "ignore table/chain". Ferm does not play nice with others.
Mon, Jan 8, 5:28 PM · Patch-For-Review, PAWS, cloud-services-team (Kanban), Kubernetes, Toolforge
aborrero added a comment to T184433: Deletion of als and mo wikis causes halting errors in https://tools.wmflabs.org/meta/userpages.

To do a clean sweep and rebuild of meta_p

/usr/local/sbin/maintain-meta_p --all-databases [--debug]

Mon, Jan 8, 2:22 PM · Tools, Data-Services
aborrero created T184433: Deletion of als and mo wikis causes halting errors in https://tools.wmflabs.org/meta/userpages.
Mon, Jan 8, 2:03 PM · Tools, Data-Services
aborrero created T184413: CloudVPS: quota increase request for 'aborrero-test' project.
Mon, Jan 8, 11:17 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
aborrero added a comment to T184244: Puppet broken on deployment-mx due to systemd on trusty.

Added @Krenair (Alex Monk) to the 'projectadmin' role in the 'deployment-prep' project at https://wikitech.wikimedia.org/wiki/Special:NovaProject

Mon, Jan 8, 10:26 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure

Fri, Jan 5

aborrero created T184261: labspuppetmaster1001: alert if git sync fails.
Fri, Jan 5, 10:01 AM · cloud-services-team
aborrero created T184259: labspuppetmaster1001: have consistency in owner of git repos.
Fri, Jan 5, 9:52 AM · cloud-services-team

Thu, Jan 4

aborrero updated the task description for T184189: Cloud: Labvirt and instance reboots for Meltdown.
Thu, Jan 4, 4:07 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)
aborrero updated the task description for T184189: Cloud: Labvirt and instance reboots for Meltdown.
Thu, Jan 4, 4:06 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)
aborrero added a comment to T184189: Cloud: Labvirt and instance reboots for Meltdown.

Regarding toolsforge instances, we should take into account that some of them have pending kernel upgrades already: T180809

Thu, Jan 4, 4:00 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)
aborrero updated the task description for T184189: Cloud: Labvirt and instance reboots for Meltdown.
Thu, Jan 4, 3:52 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)
aborrero updated the task description for T184189: Cloud: Labvirt and instance reboots for Meltdown.
Thu, Jan 4, 3:47 PM · Patch-For-Review, Operations, Toolforge, Cloud-VPS, cloud-services-team (Kanban)

Wed, Jan 3

aborrero added a comment to T183235: tools: failed to connect to apt repository.

Some issues with the merged patch: P6517 reported by @bd808

Wed, Jan 3, 3:45 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero closed T154150: role::puppetmaster::standalone has no firewall rule for port 8140 as Resolved.

Merged both:

Wed, Jan 3, 11:36 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Puppet

Sat, Dec 23

Framawiki awarded T183142: paws: 502 bad gateway a The World Burns token.
Sat, Dec 23, 10:49 AM · PAWS, Cloud-Services

Dec 21 2017

aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
  • It feels a little odd to have specific apt sources.list here. It's not very DRY, could potentially conflict with the system's configuration and ultimately with what's in configuration management (puppet).

Well, since all is in the same repo (puppet), one simply should take care to use consistent repos when doing commits?

Well there are two things broadly that could be done better here: a) decouple code from configuration, b) integrate configuration with configuration management, possibly centralizing them in the same place. The fact that the duplicated information is on the same repository doesn't make it less duplicated or less prone to drift over time, unfortunately.

Dec 21 2017, 7:04 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero added a comment to T183235: tools: failed to connect to apt repository.

How do we persist this?

Dec 21 2017, 5:45 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero updated subscribers of T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

About the patch for the wrapper tool to do targeted upgrades, @faidon added some comments in gerrit (https://gerrit.wikimedia.org/r/#/c/398079/) and I would like to discuss this here better than there.

Dec 21 2017, 5:12 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
Addshore awarded T183142: paws: 502 bad gateway a Heartbreak token.
Dec 21 2017, 3:36 PM · PAWS, Cloud-Services

Dec 20 2017

aborrero added a comment to T183235: tools: failed to connect to apt repository.

I discussed with @chasemp this, and we will be adding support for stretch.

Dec 20 2017, 2:01 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
aborrero added a comment to T183142: paws: 502 bad gateway.

The website no longer shows the 502, but the 500:

Dec 20 2017, 11:21 AM · PAWS, Cloud-Services

Dec 19 2017

aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 19 2017, 3:34 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero created T183235: tools: failed to connect to apt repository.
Dec 19 2017, 12:07 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge

Dec 18 2017

aborrero closed T183091: 502 Bad Gateway on the whole server as Invalid.

Opened T183142 with more details. Sorry for the duplicate.

Dec 18 2017, 2:01 PM · PAWS
aborrero created T183142: paws: 502 bad gateway.
Dec 18 2017, 1:57 PM · PAWS, Cloud-Services

Dec 15 2017

aborrero created P6473 (An Untitled Masterwork).
Dec 15 2017, 1:26 PM
aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

I'm not happy with the reporter script, it returns loads of data if running with clush in the tools project. Lots of nodes, Ubuntu with lots of pending upgrades, lots of output. It doesn't fit in this environment with such a scale

Dec 15 2017, 12:47 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 15 2017, 12:41 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge

Dec 14 2017

aborrero updated the language for P6466 apt-upgrade.py from autodetect to python.
Dec 14 2017, 7:01 PM
aborrero created P6466 apt-upgrade.py .
Dec 14 2017, 7:01 PM

Dec 13 2017

aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 13 2017, 5:33 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge

Dec 12 2017

aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 12 2017, 6:54 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 12 2017, 6:52 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero updated the task description for T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.
Dec 12 2017, 6:48 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

Created https://wikitech.wikimedia.org/wiki/Attended_package_upgrades to dump documentation

Dec 12 2017, 4:22 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge

Dec 1 2017

aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

In https://gerrit.wikimedia.org/r/394572 there is a script that reports every upgradeable package, per source repository, and what could be upgraded with the current unattended-ugprade configuration in the local node.

Dec 1 2017, 2:38 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero added a comment to P6365 report-pending-upgrades.sh.

Updated script, which now shows upgradeable packages by repository

Dec 1 2017, 2:14 PM

Nov 30 2017

aborrero closed T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually as Resolved.
Nov 30 2017, 6:31 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero added a comment to T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case.

Upgraded patch by @chasemp to implement only 2 hiera keys: wmf and updates. Security upgrades cannot be disabled since they come enabled by default with the unattended-upgrades package installation, and we want them always enabled anyway.

Nov 30 2017, 6:29 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
aborrero added a comment to T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.

All seems fine now.

Nov 30 2017, 6:22 PM · cloud-services-team (Kanban), Patch-For-Review, Tools

Nov 28 2017

aborrero added a comment to T180513: Document wiki-replicas architecture for future automation.

Pending to meet with DBAs, for them to provide additional information and discuss improvements in our side.

Nov 28 2017, 5:21 PM · Data-Services, cloud-services-team (Kanban), Cloud-VPS

Nov 24 2017

aborrero added a comment to T154150: role::puppetmaster::standalone has no firewall rule for port 8140.

More testing. I see that a patch like this just works, but the reporter @scfc seems to suggest this doesn't work:

Nov 24 2017, 7:00 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Puppet
aborrero closed T181304: ferm package installation fails because asks debconf question as Resolved.
Nov 24 2017, 4:37 PM · cloud-services-team
aborrero added a comment to T181304: ferm package installation fails because asks debconf question.

This was a mistake, dpkg was apparently failing because a wrong configuration in /etc/apt/. Nothing to do with the debconf question.

Nov 24 2017, 4:37 PM · cloud-services-team
aborrero added a project to T181304: ferm package installation fails because asks debconf question: cloud-services-team.
Nov 24 2017, 4:11 PM · cloud-services-team
aborrero created T181304: ferm package installation fails because asks debconf question.
Nov 24 2017, 1:53 PM · cloud-services-team

Nov 23 2017

aborrero added a comment to T177920: unattended-upgrades not upgrading "-wikimedia" packages automatically in wmcs.

Just to don't forget, I created this script: P6365

Nov 23 2017, 6:03 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Nov 22 2017

aborrero added a comment to P6365 report-pending-upgrades.sh.

Example output in debian stable stretch:

Nov 22 2017, 3:53 PM
aborrero added a comment to P6365 report-pending-upgrades.sh.

Example output in tools-exec-1403 (ubuntu trusty)

Nov 22 2017, 3:53 PM
aborrero added a comment to P6365 report-pending-upgrades.sh.

Example output in debian buster testing:

Nov 22 2017, 3:52 PM
aborrero edited P6365 report-pending-upgrades.sh.
Nov 22 2017, 2:07 PM
aborrero created P6365 report-pending-upgrades.sh.
Nov 22 2017, 1:54 PM

Nov 20 2017

aborrero added a comment to T154150: role::puppetmaster::standalone has no firewall rule for port 8140.

I would need more investigation/information. These are my findings until now:

Nov 20 2017, 7:00 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Puppet
aborrero claimed T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.
Nov 20 2017, 6:24 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero added a comment to T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.

I was wrong, we can indeed force to use old config by deploying a new file /etc/apt/apt.conf.d/00dpkg-conf:

Nov 20 2017, 12:44 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero added a comment to T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.

So, we can forget about nginx-common which has been manually upgraded by @bd808 the other day.

Nov 20 2017, 12:24 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero updated the task description for T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.
Nov 20 2017, 12:10 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero updated subscribers of T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.

Today I found a lot of emails from servers in the tools cluster with this content:

Nov 20 2017, 11:55 AM · cloud-services-team (Kanban), Patch-For-Review, Tools

Nov 17 2017

aborrero renamed T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually from tools cluster: package 'puppet-common' has conffile prompt and needs to be upgraded manually to tools cluster: packages have conffile prompts and needs to be upgraded manually.
Nov 17 2017, 6:18 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero renamed T180809: tools cluster: pending linux kernel upgrades from Pending linux kernel upgrades in tools cluster to tools cluster: pending linux kernel upgrades.
Nov 17 2017, 6:07 PM · Tools, cloud-services-team
aborrero created T180811: tools cluster: packages have conffile prompts and needs to be upgraded manually.
Nov 17 2017, 5:12 PM · cloud-services-team (Kanban), Patch-For-Review, Tools
aborrero created T180809: tools cluster: pending linux kernel upgrades.
Nov 17 2017, 5:02 PM · Tools, cloud-services-team
aborrero added a comment to T175885: Toolforge's static webserver broken by Puppet changes and stale nginx packages.

A side note.

Nov 17 2017, 4:52 PM · cloud-services-team (Kanban), Operations, Cloud-VPS, Toolforge

Nov 16 2017

aborrero updated subscribers of T177920: unattended-upgrades not upgrading "-wikimedia" packages automatically in wmcs.

@chasemp and me merged this patch and did a controlled deployment into the tools cluster.

Nov 16 2017, 5:47 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Nov 15 2017

aborrero committed rICMVdfa6c0c02de7: commit_message_validator: homogeinize case in footers (authored by aborrero).
commit_message_validator: homogeinize case in footers
Nov 15 2017, 6:40 PM
aborrero committed rICMVaf14a6d5a03b: commit_message_validator: homogeinize case in footers (authored by aborrero).
commit_message_validator: homogeinize case in footers
Nov 15 2017, 6:21 PM
aborrero committed rICMV58d503dccada: commit_message_validator: homogeinize case in footers (authored by aborrero).
commit_message_validator: homogeinize case in footers
Nov 15 2017, 6:21 PM
aborrero committed rICMV731d133f6db1: commit_message_validator: homogeinize case in footers (authored by aborrero).
commit_message_validator: homogeinize case in footers
Nov 15 2017, 6:21 PM
aborrero committed rICMVeed40be4e3f0: commit_message_validator: homogeinize case in footers (authored by aborrero).
commit_message_validator: homogeinize case in footers
Nov 15 2017, 6:21 PM