Page MenuHomePhabricator

cloudvps: wikidataconcepts project trusty deprecation
Closed, ResolvedPublic

Description

Ubuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date.

All instances in the wikidataconcepts project needs to upgrade as soon as possible.

The list of affected VMs is:

  • wikidataconcepts.wikidataconcepts.eqiad.wmflabs

Listed administrator are:

More info in openstack browser: https://tools.wmflabs.org/openstack-browser/project/wikidataconcepts

Event Timeline

aborrero triaged this task as Medium priority.Sep 18 2018, 11:58 AM
aborrero created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2018, 11:58 AM

@aborrero

All instances in the wikidataconcepts project needs to upgrade as soon as possible.

How "ASAP" is this? There are some important developments that will be taking place there and I would not like to interrupt that work before it is done. Please provide a deadline. Thank you.

Hey @GoranSMilovanovic! Just a friendly reminder that you should get rid of your Trusty instances as described in https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects. The deadline is 2018-12-18. Please get in contact if you need help.

Another ping. Deadline is approaching (2018-12-18).

Ping. Today is Friday and the deadline is Tuesday.

@GoranSMilovanovic: Today is the deadline.

@Krenair For long will the Trusty instances be available? I mean: you are deleting them, or not? The wikidataconcepts instance is fully backed-up, just in case.

I suspect it'll go through a regularly-powered-off-but-not-deleted phase and then ultimately they'll delete the old trusty instances. Projects with active migration efforts will probably be left be for now, e.g. the tools project which is the biggest and most complicated project still has trusty stuff and is expected to for a few more months, and that one is directly administered by the same people that also administer the Cloud VPS environment in general. I wouldn't count on stuff working out for projects where there is no serious migration happening though.

@Krenair Could you please get me in touch with the CloudVPS people who would eventually delete this project?
It is currently running some of the essential Wikidata stats (http://wdcm.wmflabs.org/). I will migrate it to a Debian Stretch instance, but I really need more time for this.
Thanks.

I can ask them to post here but I think they will want to know:

  • why is more time needed? the task was filed 18th september, the deadline (18th december) given on 22nd october
  • how much more time is needed?
  • how is it essential exactly? it runs inside wmflabs rather than production
  • More time is needed because I have confused this Phab ticket with another operation (see: https://phabricator.wikimedia.org/phame/post/view/120/neutron_is_here/)
  • More time is also needed because I run all WMDE analytics - from back-end to front-end operations - as an external contractor, and I simply did not have enough time to migrate what is on wikidataconcepts to another (Debian Stretch) instance;
  • Everything from the current wikidataconcepts instance is backed-up so if they really have to delete it, let them delete it, but the outcome will be the absence of many Wikidata statistics services for a while;
  • It is rather essential (see: https://wikitech.wikimedia.org/wiki/Wikidata_Concepts_Monitor).

Thanks. I hope the CloudVPS people will have mercy.

Andrew added a subscriber: Andrew.

@GoranSMilovanovic Thank you for the explanation. We're not likely to take any immediate action with that VM; we'll discuss this and try to get back to you with an adjusted timeline. If you're able to rebuild things on Stretch before we get back to you, all the better :)

@Andrew Thank you very much. I will do my best to migrate everything to a fresh new Strech instance ASAP.

Hi! Since the deadline already passed, we agreed on shutting down remaining Trusty instances on 2019-01-18. More info at https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects

GoranSMilovanovic added a comment.EditedJan 16 2019, 7:28 PM
  • Launched new m1.xlarge instance wmde-dashboards from the wmde-dashboards project to replace the existing Ubuntu Trusty instance which is set for deletion on 01/18/2019.
  • Deployed image on the new instance: debian-9.6-stretch
  • Final wikidataconcepts.eqiad.wmflabs instance back-up is now completed.
  • The instance will be deleted as soon as everything is set up on the new wmde-dashboards instance under Strech;
  • There will be a short period when all running WMDE dashboards will be unavailable (due to a migration of the http://wdcm.wmflabs.org/ domain);
  • @Addshore is currently inspecting why we can't SSH to the new wmde-dashboards instance (maybe the first Puppet runs are not completed yet).
  • New instance wmde-dashboards.eqiad.wmflabs up and running;
  • installations ongoing.
  • RStudio Server up and running;
  • installing R packages now;
  • next step: Shiny Server.

@Krenair @bd808 @Andrew

Now this is really silly: I cannot ssh to wmde-dashboards.eqiad.wmflabs anymore (it was possible until 10 minutes ago or so - and no settings were changed in the meantime):

ssh -o ServerAliveInterval=120 wmde-dashboards.eqiad.wmflabs
Enter passphrase for key 'blah-blah-blah': 
channel 0: open failed: connect failed: Connection timed out
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

Please advise.

  • Security issue recognized during the process: T213977
  • Waiting for instructions on how to proceed from the CloudVPS @Krenair @bd808 @Andrew
GoranSMilovanovic changed the task status from Open to Stalled.Jan 16 2019, 11:28 PM
  • Stalled until T213977 (critical) is resolved.
GoranSMilovanovic changed the task status from Stalled to Open.Jan 17 2019, 1:20 AM

In the meantime I have installed Shiny Server on wmde-dashboards.eqiad.wmflabs:

  • no problems in installation;
  • security group created and applied to the instance, port 3838 opened;
  • new web proxy created wmdeanalytics.wmflabs.org and mapped onto port 3838;
  • nginx.conf on the new instances is *exactly* the same as on wikidataconcepts.eqiad.wmflabs which runs Shiny Server successfully;
  • curl localhost:3838 does return the html content of the Shiny Server welcome page, and still
  • a visit to http://wmdeanalytics.wmflabs.org/ returns (Chromium): This site can’t be reached wmdeanalytics.wmflabs.org’s server IP address could not be found. DNS_PROBE_FINISHED_NXDOMAIN

Reminder: previously, we were able to make Shiny Server running from a CloudVPS instance, see T167702

Please advise. Thanks.

  • new web proxy created wmdeanalytics.wmflabs.org and mapped onto port 3838;
  • a visit to http://wmdeanalytics.wmflabs.org/ returns (Chromium): This site can’t be reached wmdeanalytics.wmflabs.org’s server IP address could not be found. DNS_PROBE_FINISHED_NXDOMAIN

I think this is just the usual designate-changes-being-slow:

alex@alex-laptop:~$ dig wmdeanalytics.wmflabs.org @labs-ns0.wikimedia.org +short
185.15.56.49
alex@alex-laptop:~$ dig wmdeanalytics.wmflabs.org @labs-ns1.wikimedia.org +short
alex@alex-laptop:~$ dig wmdeanalytics.wmflabs.org @labs-ns2.wikimedia.org +short
185.15.56.49

labs-ns1 doesn't have it yet

@Krenair Thanks, I'll give it some time then.

Ok. I need to get some sleep, it's 02:30 CET. Let's see if there is a way to satisfy the constraints described in T213977#4886646. Good night!

FYI DNS seems to be okay now

@Krenair It is, I can see, thanks.

Ok now we need to take care of T213977 and especially T213977#4886646 - any ideas @Addshore

@Lydia_Pintscher If this ^^ is not resolved today, we will be missing WDCM and other WMDE analytical features as of tomorrow and until the ticket's resolution.

@Krenair @bd808 @aborrero

  • I think I found the solution how to keep the R development workflow from the new wmde-dashboards.eqiad.wmflabs instance;
  • Please ping here *before* you delete the wikidataconcepts.eqiad.wmflabs instance. Thank you.

Project wikidataconcepts.eqiad.wmflabs can be safely deleted.

Mentioned in SAL (#wikimedia-cloud) [2019-01-21T16:41:05Z] <gtirloni> T204695 shutdown ubuntu VM: wikidataconcepts

aborrero closed this task as Resolved.Feb 12 2019, 4:29 PM
aborrero claimed this task.