Page MenuHomePhabricator

Create a dockerized mobileapps instance in the Beta Cluster
Closed, ResolvedPublic

Description

The mobileapps service is currently running in the Beta Cluster on the deployment-mcs01 instance. We should create a new instance, using the role::beta::docker_services Puppet role, to host the Dockerized service using images from the deployment pipeline. The configuration will be similar to that used on deployment-push-notifications01.

To reduce confusion and match the configuration on k8s, I would suggest updating the naming convention to use the technical service name 'mobileapps' rather than the informal 'pcs', making this new instance deployment-mobileapps01 (or, for additional clarity, deployment-docker-mobileapps01).

AC

  • Create new instance deployment-mobileapps01 (or deployment-docker-mobileapps01) from the latest Debian Buster image
  • Update the Puppet SSL cert to get Puppet running successfully with the Beta Cluster puppetmaster (see P7162 for an example of the procedure)
  • Add required hiera config, including the service configuration
  • Apply the role::beta::docker_services and ensure Puppet still runs successfully
  • Verify that the service is correctly serving internal requests
  • Create a security group (if needed) to expose the service port (8888) to incoming traffic, and apply it to deployment-[docker-]mobileapps01
  • Migrate any existing references to deployment-mcs01 to deployment-[docker-]mobileapps01
  • Destroy deployment-mcs01

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2020, 6:43 PM
Mholloway updated the task description. (Show Details)Jun 30 2020, 6:55 PM
LGoto triaged this task as Medium priority.Jul 1 2020, 3:39 PM
LGoto moved this task from Needs triage to Backlog on the Product-Infrastructure-Team-Backlog board.
Mholloway updated the task description. (Show Details)Jul 10 2020, 2:23 PM
Jgiannelos updated the task description. (Show Details)Jul 14 2020, 10:36 AM
Jgiannelos updated the task description. (Show Details)Jul 14 2020, 11:11 AM
Jgiannelos updated the task description. (Show Details)Wed, Jul 15, 8:04 AM
Jgiannelos updated the task description. (Show Details)Wed, Jul 15, 8:24 AM
Jgiannelos updated the task description. (Show Details)Wed, Jul 15, 8:31 AM

Migrate any existing references to deployment-mcs01 to deployment-[docker-]mobileapps01

I tried to grep the cloud/instance-puppet repository but didn't return any references. Does this mean we should go through all the instances and check the configs ?

In this case, I would grep the following repos and call it good:

I've just checked the project config and the deployment-restbase prefix config in Horizon for the deployment-prep (aka Beta Cluster) project, and didn't see any references that needed updating.

That seems about right to me. Update that reference and we should be all set.

Change 613144 had a related patch set uploaded (by Jgiannelos; owner: Jgiannelos):
[operations/puppet@production] Change hostname of mobileapps to the dockerized instance.

https://gerrit.wikimedia.org/r/613144

Jgiannelos updated the task description. (Show Details)Thu, Jul 16, 1:39 PM

Currently there is no web proxy to expose deployment-mcs01. Do we only expose it using restbase?

Ah -- you know what, I think that's intentional. The goal for the Beta Cluster is to mirror production as closely as possible, and production mobileapps isn't exposed to the public but is only consumed by RESTBase. I'll remove that item from the AC list.

Mholloway updated the task description. (Show Details)Thu, Jul 16, 2:44 PM

Cool, I am also removing the web proxy I created.

Jgiannelos added a comment.EditedThu, Jul 16, 5:27 PM

Here is the current status:

  • We have a new CloudVPS instance deployment-docker-mobileapps01 that's running mobileapps using role::beta::docker_services
    • The profile::docker::engine:: hiera config is based on the deployment-push-notifications01 instance config
    • Its using the same security groups as deployment-mcs01
  • Here is the yaml diff with the required changes so the dockerized version is in a working state: diff
    • Removed the log stream to the file that wouldn't work because the path was not backed by a persistent volume
    • Added a stream to write logs to stdout so docker can pick it up
    • Changed the module path because file structure now is different
  • After GETing a couple of endpoints looks like service is working fine
    • curl 127.0.0.1:8888/en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/Dog --> returns the right results
    • curl 127.0.0.1:8888/en.wikipedia.org/v1/page/mobile-html/Dog --> returns an error
    • Browsing the swagger UI looks like its working fine

I think moving forward we need to:

  • We need to build images and deploy the current beta version. Currently for testing the deployment I used the latest production version I found in the docker registry.
  • Review and merge: https://gerrit.wikimedia.org/r/c/613144 to connect beta restbase to the new mobileapps instance
  • Adapt the code releasing pipeline so beta mobileapps is deployed to deployment-docker-mobileapps01

@Mholloway thoughts?

Here is the current status:
[...]

All of this looks good.

I think moving forward we need to:

  • We need to build images and deploy the current beta version. Currently for testing the deployment I used the latest production version I found in the docker registry.

Just to be clear, there is no separate set of beta images. The standard practice is simply to keep the Beta Cluster and production in sync, i.e., keep the Beta Cluster updated with the current production version. So it sounds like what you have now is correct.

Yep. I +1'd and added SRE reviewers for a +2.

  • Adapt the code releasing pipeline so beta mobileapps is deployed to deployment-docker-mobileapps01

I don't think there's anything more to do here, but I did create a new wiki page (https://www.mediawiki.org/wiki/Wikimedia_Product/Wikimedia_Product_Infrastructure_team/Service_deployment_process) to document the current deployment process.

I think moving forward we need to:

  • We need to build images and deploy the current beta version. Currently for testing the deployment I used the latest production version I found in the docker registry.

Just to be clear, there is no separate set of beta images. The standard practice is simply to keep the Beta Cluster and production in sync, i.e., keep the Beta Cluster updated with the current production version. So it sounds like what you have now is correct.

Cool, thanks for the clarification. I thought beta was one step before prod in the release process, not that they are in sync.

Yeah, it's a bit weird. It's probably true to say that the Beta Cluster is only really a beta environment for MediaWiki core and extension code; that's automatically updated after every patch is merged in those repos. Services, on the other hand, have always been deployed to the Beta Cluster in sync with production. I'm not sure if that's the best possible way of doing things, but it's what's been done for as long as I've been here.

It might be an interesting thing to chat about with the Core Platform Team during their office hours sometime.

Change 613144 merged by Alexandros Kosiaris:
[operations/puppet@production] Change hostname of mobileapps to the dockerized instance.

https://gerrit.wikimedia.org/r/613144

Change 615502 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] Use new naming convention for deployment-docker-mobileapps01

https://gerrit.wikimedia.org/r/615502

Change 615502 merged by Dzahn:
[operations/puppet@production] Use new naming convention for deployment-docker-mobileapps01

https://gerrit.wikimedia.org/r/615502

Mholloway closed this task as Resolved.Fri, Jul 31, 2:02 PM