All Ubuntu Trusty VMs need to be replaced with VMs running a supported operating system by the April 2019 upstream end of life date. On Nov 20, 2017 we disabled self-service creation of new Trusty VMs by most projects. We now need to begin the process of identifying the Trusty VMs which remain, notifying their owners of the need to replace these VMs with Debian Stretch VMs (or Debian Jessie in exceptional circumstances), and monitoring the results.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T186029 Remove support for Ubuntu Trusty prior to upstream End Of Life for release | |||
| Resolved | aborrero | T199003 Develop timeline for Cloud VPS wide deprecation of Trusty |
Event Timeline
Comment Actions
Generating a list of physical servers by dist:
aborrero@neodymium:~ 3s 130 $ sudo cumin --force -m async -b 10 'P{lab* or cloud*}' "lsb_release -c" -o txt | awk 'x==1 { print $0 } /_____FORMATTED_OUTPUT_____/ { x=1 }' | grep trusty
78 hosts will be targeted:
cloudcontrol[1003-1004].wikimedia.org,cloudelastic[1001-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1019-1022].eqiad.wmnet,labcontrol[1001-1002].wikimedia.org,labmon[1001-1002].eqiad.wmnet,labnet[1001-1002].eqiad.wmnet,labnodepool1001.eqiad.wmnet,labpuppetmaster[1001-1002].wikimedia.org,labsdb[1004-1007,1009-1011].eqiad.wmnet,labservices[1001-1002].wikimedia.org,labstore[2001-2004].codfw.wmnet,labstore[1001-1005].eqiad.wmnet,labstore[1006-1007].wikimedia.org,labtestcontrol[2001,2003].wikimedia.org,labtestmetal2001.codfw.wmnet,labtestnet[2002-2003].codfw.wmnet,labtestneutron[2001-2002].codfw.wmnet,labtestpuppetmaster2001.wikimedia.org,labtestservices[2001-2003].wikimedia.org,labtestvirt[2001-2003].codfw.wmnet,labtestweb2001.wikimedia.org,labvirt[1001-1018].eqiad.wmnet,labweb[1001-1002].wikimedia.org
[...]
labcontrol1001.wikimedia.org: Codename: trusty
labcontrol1002.wikimedia.org: Codename: trusty
labnet1001.eqiad.wmnet: Codename: trusty
labnet1002.eqiad.wmnet: Codename: trusty
labservices1001.wikimedia.org: Codename: trusty
labservices1002.wikimedia.org: Codename: trusty
labstore1003.eqiad.wmnet: Codename: trusty
labtestcontrol2001.wikimedia.org: Codename: trusty
labtestnet2002.codfw.wmnet: Codename: trusty
labtestnet2003.codfw.wmnet: Codename: trusty
labtestservices2001.wikimedia.org: Codename: trusty
labtestvirt2001.codfw.wmnet: Codename: trusty
labtestvirt2002.codfw.wmnet: Codename: trusty
labvirt1001.eqiad.wmnet: Codename: trusty
labvirt1002.eqiad.wmnet: Codename: trusty
labvirt1003.eqiad.wmnet: Codename: trusty
labvirt1004.eqiad.wmnet: Codename: trusty
labvirt1005.eqiad.wmnet: Codename: trusty
labvirt1006.eqiad.wmnet: Codename: trusty
labvirt1007.eqiad.wmnet: Codename: trusty
labvirt1008.eqiad.wmnet: Codename: trusty
labvirt1009.eqiad.wmnet: Codename: trusty
labvirt1010.eqiad.wmnet: Codename: trusty
labvirt1011.eqiad.wmnet: Codename: trusty
labvirt1012.eqiad.wmnet: Codename: trusty
labvirt1013.eqiad.wmnet: Codename: trusty
labvirt1014.eqiad.wmnet: Codename: trusty
labvirt1015.eqiad.wmnet: Codename: trusty
labvirt1017.eqiad.wmnet: Codename: trustyComment Actions
Generating a list of VM by dist:
aborrero@labpuppetmaster1001:~$ sudo cumin --force -m async -b 10 -p 0 'O{power_state:1}' "lsb_release -c" -o txt | awk 'x==1 { print $0 } /_____FORMATTED_OUTPUT_____/ { x=1 }' | grep trusty
[...]
api-alpha.mw-api-testing.eqiad.wmflabs: Codename: trusty
bf-wmpageview.butterfly.eqiad.wmflabs: Codename: trusty
botbot.bots.eqiad.wmflabs: Codename: trusty
bugzilla.dumps.eqiad.wmflabs: Codename: trusty
butterfly-m4m.butterfly.eqiad.wmflabs: Codename: trusty
client.nonfreewiki.eqiad.wmflabs: Codename: trusty
commonsarchive-backup.commonsarchive.eqiad.wmflabs: Codename: trusty
commonsarchive-production.commonsarchive.eqiad.wmflabs: Codename: trusty
commonsarchive-varnish.commonsarchive.eqiad.wmflabs: Codename: trusty
danny-b.wildcat.eqiad.wmflabs: Codename: trusty
db.mwfileimport.eqiad.wmflabs: Codename: trusty
deployment-zotero01.deployment-prep.eqiad.wmflabs: Codename: trusty
design-lsg3.design.eqiad.wmflabs: Codename: trusty
df2.wikidata-page-banner.eqiad.wmflabs: Codename: trusty
discourse-wam.discourse-wam.eqiad.wmflabs: Codename: trusty
discovery-production.shiny-r.eqiad.wmflabs: Codename: trusty
discovery-testing.shiny-r.eqiad.wmflabs: Codename: trusty
drmf-beta.math.eqiad.wmflabs: Codename: trusty
drmf.math.eqiad.wmflabs: Codename: trusty
dumps-stats.dumps.eqiad.wmflabs: Codename: trusty
fastcci-master.fastcci.eqiad.wmflabs: Codename: trusty
fawp-tts.fa-wp.eqiad.wmflabs: Codename: trusty
fawp.fa-wp.eqiad.wmflabs: Codename: trusty
federation-wikis.wikidata-federation.eqiad.wmflabs: Codename: trusty
gadgets.social-tools.eqiad.wmflabs: Codename: trusty
huggle-pg.huggle.eqiad.wmflabs: Codename: trusty
ids-testing.mw-extension-ids.eqiad.wmflabs: Codename: trusty
images.nonfreewiki.eqiad.wmflabs: Codename: trusty
jupyter-mw-02.jupyter.eqiad.wmflabs: Codename: trusty
ldfclient.wikidata-query.eqiad.wmflabs: Codename: trusty
living-style-guide.design.eqiad.wmflabs: Codename: trusty
lsg-01.design.eqiad.wmflabs: Codename: trusty
maps-tiles2.maps.eqiad.wmflabs: Codename: trusty
maps-tiles3.maps.eqiad.wmflabs: Codename: trusty
maps-warper2.maps.eqiad.wmflabs: Codename: trusty
maps-wma1.maps.eqiad.wmflabs: Codename: trusty
math-ru.math.eqiad.wmflabs: Codename: trusty
mathoid2.math.eqiad.wmflabs: Codename: trusty
mediaviewer.multimedia.eqiad.wmflabs: Codename: trusty
multiwiki-wikibase.wikidata-federation.eqiad.wmflabs: Codename: trusty
mw-base.wikitextexp.eqiad.wmflabs: Codename: trusty
mw-expt.wikitextexp.eqiad.wmflabs: Codename: trusty
mwfuzz1.mwfuzz.eqiad.wmflabs: Codename: trusty
mwoffliner1.mwoffliner.eqiad.wmflabs: Codename: trusty
mwoffliner2.mwoffliner.eqiad.wmflabs: Codename: trusty
mwoffliner3.mwoffliner.eqiad.wmflabs: Codename: trusty
mwoffliner4.mwoffliner.eqiad.wmflabs: Codename: trusty
mwoffliner5.mwoffliner.eqiad.wmflabs: Codename: trusty
nomad.reading-web-staging.eqiad.wmflabs: Codename: trusty
osm-serv.osmit.eqiad.wmflabs: Codename: trusty
osmit-due.osmit.eqiad.wmflabs: Codename: trusty
osmit-tre.osmit.eqiad.wmflabs: Codename: trusty
pagemigration-vagrant.pagemigration.eqiad.wmflabs: Codename: trusty
petscan-dev2.petscan.eqiad.wmflabs: Codename: trusty
phlogiston-2.phlogiston.eqiad.wmflabs: Codename: trusty
phlogiston-3.phlogiston.eqiad.wmflabs: Codename: trusty
pole.wikidata-query.eqiad.wmflabs: Codename: trusty
puppet-mailman.puppet.eqiad.wmflabs: Codename: trusty
queryr-api-01.queryrapi.eqiad.wmflabs: Codename: trusty
queryr-db-01.queryrapi.eqiad.wmflabs: Codename: trusty
recommendations.reportcard.eqiad.wmflabs: Codename: trusty
rel.search.eqiad.wmflabs: Codename: trusty
relforge-search.search.eqiad.wmflabs: Codename: trusty
relic.toolserver-legacy.eqiad.wmflabs: Codename: trusty
sentry-testwiki.sentry.eqiad.wmflabs: Codename: trusty
shinken-01.shinken.eqiad.wmflabs: Codename: trusty
shinyserv-01.discovery-stats.eqiad.wmflabs: Codename: trusty
shinyserv-lb.discovery-stats.eqiad.wmflabs: Codename: trusty
signwriting-server.signwriting.eqiad.wmflabs: Codename: trusty
social-tools2.social-tools.eqiad.wmflabs: Codename: trusty
striker-deploy03.striker.eqiad.wmflabs: Codename: trusty
suggestbot-prod.suggestbot.eqiad.wmflabs: Codename: trusty
telnet2.telnet.eqiad.wmflabs: Codename: trusty
timeless1.social-tools.eqiad.wmflabs: Codename: trusty
tin.rcm.eqiad.wmflabs: Codename: trusty
tools-bastion-02.tools.eqiad.wmflabs: Codename: trusty
tools-bastion-03.tools.eqiad.wmflabs: Codename: trusty
tools-bastion-05.tools.eqiad.wmflabs: Codename: trusty
tools-checker-01.tools.eqiad.wmflabs: Codename: trusty
tools-checker-02.tools.eqiad.wmflabs: Codename: trusty
tools-cron-01.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1401.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1402.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1403.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1404.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1405.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1406.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1407.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1408.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1409.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1410.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1411.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1412.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1413.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1414.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1415.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1416.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1417.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1418.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1419.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1420.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1421.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1422.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1423.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1424.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1425.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1426.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1427.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1428.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1429.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1430.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1431.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1432.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1433.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1434.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1435.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1436.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1437.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1438.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1439.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1440.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1441.tools.eqiad.wmflabs: Codename: trusty
tools-exec-1442.tools.eqiad.wmflabs: Codename: trusty
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: Codename: trusty
tools-grid-master.tools.eqiad.wmflabs: Codename: trusty
tools-grid-shadow.tools.eqiad.wmflabs: Codename: trusty
tools-mail.tools.eqiad.wmflabs: Codename: trusty
tools-services-01.tools.eqiad.wmflabs: Codename: trusty
tools-services-02.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-generic-1401.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-generic-1402.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-generic-1403.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-generic-1404.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: Codename: trusty
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: Codename: trusty
toolsbeta-bastion-01.toolsbeta.eqiad.wmflabs: Codename: trusty
toolsbeta-exec-1401.toolsbeta.eqiad.wmflabs: Codename: trusty
toolsbeta-grid-master.toolsbeta.eqiad.wmflabs: Codename: trusty
toolsbeta-webgrid-generic-1401.toolsbeta.eqiad.wmflabs: Codename: trusty
toolsbeta-webgrid-lighttpd-1401.toolsbeta.eqiad.wmflabs: Codename: trusty
trusty-update.puppet.eqiad.wmflabs: Codename: trusty
utrs-database.utrs.eqiad.wmflabs: Codename: trusty
utrs-production.utrs.eqiad.wmflabs: Codename: trusty
vmbuilder-trusty.openstack.eqiad.wmflabs: Codename: trusty
vrt.getstarted.eqiad.wmflabs: Codename: trusty
wdsearch.search.eqiad.wmflabs: Codename: trusty
webservices.getstarted.eqiad.wmflabs: Codename: trusty
wigi.wikidumpparse.eqiad.wmflabs: Codename: trusty
wikibase-vue.wikidata-dev.eqiad.wmflabs: Codename: trusty
wikidataconcepts.wikidataconcepts.eqiad.wmflabs: Codename: trusty
xenon.rcm.eqiad.wmflabs: Codename: trustyComment Actions
In the following days I will digest these lists and develop possible alternatives for them.
Comment Actions
Physical hardware:
labstore1003.eqiad.wmnet --- needs specific plan labtestcontrol2001.wikimedia.org --- to be discontinued once we shutdown main deployment labtestnet2002.codfw.wmnet --- to be discontinued once we shutdown main deployment labtestnet2003.codfw.wmnet --- to be discontinued once we shutdown main deployment labtestservices2001.wikimedia.org --- to be discontinued once we shutdown main deployment labtestvirt2001.codfw.wmnet --- to be discontinued once we shutdown main deployment labtestvirt2002.codfw.wmnet --- to be discontinued once we shutdown main deployment labcontrol1001.wikimedia.org --- replaced by eqiad1 deployment labcontrol1002.wikimedia.org --- replaced by eqiad1 deployment labnet1001.eqiad.wmnet --- replaced by eqiad1 deployment labnet1002.eqiad.wmnet --- replaced by eqiad1 deployment labservices1001.wikimedia.org --- replaced by eqiad1 deployment labservices1002.wikimedia.org --- replaced by eqiad1 deployment labvirt1001.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1002.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1003.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1004.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1005.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1006.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1007.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1008.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1009.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1010.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1011.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1012.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1013.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1014.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1015.eqiad.wmnet --- replaced by eqiad1 deployment labvirt1017.eqiad.wmnet --- replaced by eqiad1 deployment
Comment Actions
For VMs, this is my proposed timeline:
- Now: open phabricator tickets per affected project asking their administrators to move to Stretch (or jessie if there is a strong justification)
- In 1 month: monitor results of open tickets, ping administrators
- In 2 months: monitor result of open tickers, ping administrators
- In 3 months: evaluate if trusty VMs not migrated are actually in use. If not, just delete them? Help administrators with migration issues?
I guess the first step is mandatory anyway while other steps can be discussed, so I will start doing it now.
Also, I will add a warning here https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge to projects that contain trusty VMs.
Comment Actions
@Krenair asked me to look for instances running something other than Trusty, Jessie, or Stretch. Here's what I found:
outreachdashboard.globaleducation.eqiad.wmflabs: Codename: xenial wikidata-lexeme.wikidata-dev.eqiad.wmflabs: Codename: xenial
Additionally, these instances failed to allow the cumin user to connect which probably indicates broken Puppet config:
- cloudservices.getstarted.eqiad.wmflabs
- compiler.puppet.eqiad.wmflabs
Comment Actions
If we are happy with the text, we should sent that by email to cloud-announce. @Harej would you like to do that? If not, I can send the email myself :-P
Comment Actions
I will defer to you on the email since you're probably more knowledgeable in the particulars of the transition.
Comment Actions
According to the timeline described here https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects we should be sending pings to the deprecation subtasks starting today.
Comment Actions
Needs discussion: the final deadline is approaching: 2018-12-18. How to handle remaining Trusty VMs?
Comment Actions
We need to evaluate them VM by VM (or at least project by project). Anything that seems to be actively wiki facing, for example the maps project, deserves at least outreach to on-wiki communities that are likely to be effected by a shutdown give them a reasonable amount of time to find a replacement or at least remove the dependency gracefully. We know that we will be continuing to support Trusty for Toolforge for at least 2 more months, so a few more weeks for the other stragglers is not the end of the world.
Comment Actions
Note that on T204695, the wikidataconcepts admin has asked for more time. It seems reasonable to give it to them but we should figure out what the adjusted deadline is and communicate it to them.
Comment Actions
We agreed on giving a bit more time before shutting down instances (2019-01-18) from unresponsive projects.
Comment Actions
The timeline was developed long time ago. We are in fact very close to the final cleanup stage. Closing task.