Page MenuHomePhabricator
Create Task
Maniphest T218733

Migrate mobileapps to k8s and node 10
Closed, ResolvedPublic
Actions

Description

To migrate mobileapps to Node.js 10 we will need to do the following:

  • Update the blubberfile to use node 10 base images
  • Benchmark the service
  • Write a Helm chart
  • Create kubernetes tokens @akosiaris will do that
  • Create kubernetes namespace @akosiaris will do that
  • Deploy
  • Switch traffic to the new deployment
  • Switch LVS conftool to the kubernetes cluster
  • Remove mobileapps from scb conftool config
  • Remove mobileapps from scb
  • Remove mobileapps from puppet

Details

SubjectRepoBranchLines +/-
mobileapps: Remove the profile and the roleoperations/puppetproduction+0 -45
mobileapps: Remove mobileapps from scboperations/puppetproduction+3 -3
mobileapps: Remove from scb conftool configoperations/puppetproduction+42 -42
mobileapps: Switch conftool to kubernetes/kubesvcoperations/puppetproduction+2 -2
mobileapps: Lower replicas to 80 from 240operations/deployment-chartsmaster+2 -2
mobileapps: Bump memory limits another 20%operations/deployment-chartsmaster+304 -304
mobileapps: Bump quotasoperations/deployment-chartsmaster+18 -0
mobileapps: Bump 20x the number of replicasoperations/deployment-chartsmaster+2 -2
mobileapps: Bump memory limit by 25%operations/deployment-chartsmaster+323 -303
mobileapps: fix prometheus statsd exporter issue in 0.0.10operations/deployment-chartsmaster+322 -302
mobileapps: Amend statsd exporter bucketsoperations/deployment-chartsmaster+329 -326
mobileapps: Add kubernetes nodes next to scb nodesoperations/puppetproduction+35 -11
mobileapps: Add a temporary non-TLS releaseoperations/deployment-chartsmaster+36 -0
Mobileapps: Add initial helmfile stanzasoperations/deployment-chartsmaster+121 -0
Create namespaces/calico rules for new servicesoperations/deployment-chartsmaster+85 -3
Kubernetes: Create token stanzas for some new servicesoperations/puppetproduction+42 -0
Add k8s dummy tokens for 3 new services.labs/privatemaster+24 -0
Add chart for mobileappsoperations/deployment-chartsmaster+549 -0
Blubber: Include ca-certificates in the production imagemediawiki/services/mobileappsmaster+1 -0
Blubber: Use nodejs10 base imagesmediawiki/services/mobileappsmaster+2 -2
Add Blubber configmediawiki/services/mobileappsmaster+28 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224907 Drop php55 testing support
 ResolvedReedyT205039 Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205041 Tracking bug for 1.27.6/1.30.2/1.31.2/1.32.2 security release
 ResolvedBawolffT197279 Direct POST to Special:ChangeEmail will bypass reauth check
 ResolvedAnomieT204729 Potential enwiki DOS due to slow WatchedItemStore::countVisitingWatchersMultiple
 ResolvedBawolffT207603 CVE-2019-12471: Loading JS from user space where the username is not a registered account is dangerous and should be banned
 ResolvedBawolffT208881 CSS using var() to create exponential sized calc() on wiki page will crash visitor's browser
 ResolvedLegoktmT199540 Forbid blocking IP ranges as big as /1 and /2, as done on ruwikiquote using the API
 ResolvedLucas_Werkmeister_WMDET212118 API responses for unpatrolled or (not) autopatrolled recent changes require privileges but may be cached publicly
 ResolvedBawolffT209794 Need to make a limit of count of attempts to change email address
 ResolvedsbassettT222036 Exposed suppressed username or log in Special:EditTags
 ResolvedsbassettT222038 Exposed suppressed log in RevisionDelete page
 ResolvedJdforrester-WMFT221739 Patch jQuery due to CVE-2019-11358
 ResolvedsbassettT25227 Use token when logging out
 ResolvedsbassettT221868 Send out wikitech-l post for T25227 ("Use token when logging out")
 ResolvedReedyT205042 Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedReedyT205043 Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT205044 Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205046 Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205047 Tag MW 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT224499 RELEASE-NOTES for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT224912 Update MediaWiki.org links and versions
 ResolvedJdforrester-WMFT224913 EOL REL1_27 and REL1_30 onwiki
 ResolvedReedyT225201 Formally announce EOL of MW 1.27 and 1.30
 ResolvedReedyT225149 Update CVEs and publish them
 ResolvedReedyT205048 Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedhasharT223348 Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x
 ResolvedhasharT210287 Migrate wikimedia-fundraising-civicrm to a Docker container
 ResolvedJdforrester-WMFT224905 Move wikiba.se tests from php56 to php72
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia
 DuplicateDzahnT210008 upgrade krypton (webserver_misc_apps) to stretch
 DeclinedDzahnT224194 switch webserver_misc_apps to PHP 7.2 (7.1)
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 OpenNoneT210704 Migrate node-based services in production to node10
 OpenNoneT198901 Migrate production services to kubernetes using the pipeline
 ResolvedJdforrester-WMFT247012 Archive mediawiki/services/mobileapps/deploy once it's no longer used
 ResolvedakosiarisT218733 Migrate mobileapps to k8s and node 10
 DeclinedNoneT215539 Node.js 10 changes encoding for at least one Georgian character
 Resolved• MhollowayT258186 Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Mholloway updated the task description. (Show Details)Mar 3 2020, 3:21 PM
Jdforrester-WMF updated the task description. (Show Details)Mar 5 2020, 4:46 PM
gerritbot added a comment.Mar 5 2020, 4:46 PM

Change 570162 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add chart for mobileapps

https://gerrit.wikimedia.org/r/570162

akosiaris mentioned this in rDEPLOYCHARTSf709146afe2b: Add chart for mobileapps.Mar 5 2020, 4:49 PM
akosiaris added a comment.Mar 5 2020, 5:00 PM

Chart released under https://releases.wikimedia.org/charts/

\o/

akosiaris updated the task description. (Show Details)Mar 5 2020, 5:01 PM
• Mholloway added a comment.Mar 5 2020, 5:02 PM
In T218733#5945428, @akosiaris wrote:

Chart released under https://releases.wikimedia.org/charts/

\o/

\o/

Jdforrester-WMF added a parent task: T247012: Archive mediawiki/services/mobileapps/deploy once it's no longer used.Mar 5 2020, 6:07 PM
• bearND awarded a token.Mar 5 2020, 6:42 PM
• Mholloway mentioned this in T219924: Move mobileapps logging to new logging pipeline.Mar 5 2020, 8:45 PM
fgiunchedi subscribed.Mar 6 2020, 9:23 AM
• Mholloway moved this task from Code Review to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 10 2020, 1:53 PM
gerritbot added a comment.Mar 17 2020, 10:24 AM

Change 580294 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[labs/private@master] Add k8s dummy tokens for 3 new services.

https://gerrit.wikimedia.org/r/580294

gerritbot added a comment.Mar 17 2020, 10:30 AM

Change 580295 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Kubernetes: Create token stanzas for some new services

https://gerrit.wikimedia.org/r/580295

• Mholloway added a project: serviceops.May 5 2020, 4:16 PM
JMeybohm subscribed.May 6 2020, 1:16 PM
gerritbot added a comment.May 29 2020, 12:10 PM

Change 599812 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Create namespaces/calico rules for new services

https://gerrit.wikimedia.org/r/599812

gerritbot added a comment.May 29 2020, 1:59 PM

Change 580294 merged by Alexandros Kosiaris:
[labs/private@master] Add k8s dummy tokens for 3 new services.

https://gerrit.wikimedia.org/r/580294

akosiaris mentioned this in rLPRI2078705f3c40: Add k8s dummy tokens for 3 new services..May 29 2020, 2:01 PM
gerritbot added a comment.May 29 2020, 2:09 PM

Change 580295 merged by Alexandros Kosiaris:
[operations/puppet@production] Kubernetes: Create token stanzas for some new services

https://gerrit.wikimedia.org/r/580295

gerritbot added a comment.May 29 2020, 2:20 PM

Change 599812 merged by jenkins-bot:
[operations/deployment-charts@master] Create namespaces/calico rules for new services

https://gerrit.wikimedia.org/r/599812

akosiaris updated the task description. (Show Details)Jun 2 2020, 12:05 PM

@Mholloway, @bearND namespaces, rules, tokens have been created. Chart has been merged and publish. You are free to deploy. You will require a change like 968132909b4d24192b2f69a657c14bb30acd7a42 in order to instantiate the first deploy, feel free to add me as a reviewer.

• Mholloway added a comment.Jun 3 2020, 7:17 PM

Awesome! Thanks, @akosiaris! I'll get that change going shortly.

Maybe I'm getting ahead of myself, but thinking ahead to switching traffic over to the k8s mobileapps deployment, I'm thinking it would be best to switch traffic over in stages. Could we create a new service discovery domain, something like mobileapps-docker.discovery.wmnet, to use to point certain requests to the new k8s deployment during the switchover process?

gerritbot added a comment.Jun 3 2020, 8:37 PM

Change 602155 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] Mobileapps: Add initial helmfile stanzas

https://gerrit.wikimedia.org/r/602155

akosiaris added a comment.Jun 4 2020, 8:35 AM
In T218733#6190306, @Mholloway wrote:

Awesome! Thanks, @akosiaris! I'll get that change going shortly.

Maybe I'm getting ahead of myself, but thinking ahead to switching traffic over to the k8s mobileapps deployment, I'm thinking it would be best to switch traffic over in stages. Could we create a new service discovery domain, something like mobileapps-docker.discovery.wmnet, to use to point certain requests to the new k8s deployment during the switchover process?

Before answering the question let me paint a picture of the plan that SRE wants to follow for switching the traffic to the k8s mobileapps deployment, in case that helps. It looks more or less like this:

  • Get mobileapps deployed on k8s
  • Ensure all automated healthchecks pass in exactly the same manner as the current deployment (this is more or less automated and based on the exact same healthchecks that pass on the old scb cluster and is based on the openapi/swagger spec the software has).
  • Make sure whatever other manual checks (if any) succeed as well.
  • Add the k8s deployment to the traffic pool, albeit with inactive status.
  • Set very low weights for the k8s endpoints and enable them in the traffic pool
  • Pick the inactive datacenter, codfw in this case.
  • Slowly over multiple days increase the weight of the k8s endpoints so that traffic moves from scb to k8s. The exact increments can be subject to discussion, up to now (we 've done this multiple times) the usual steps are: ~1%, ~2%, ~5%, ~10%, ~25%, ~50%, ~75%, 100%. The ~ is because the weights are integers and good precision isn't possible, so you may see something like 78% instead of 75% but the ballpark goal remains the same. Also, depending on how well this goes, we may add steps or skip steps, but overall we tend to follow the approach above.
  • Once scb serves 0%, remove it from the pool.
  • Repeat with the above 2 bullet points with the active datacenter, eqiad in this case.
  • Wait out multiple days in case an emergency rollback is warranted.
  • Undeploy from scb.

Now, to answer your question:

As you can tell, this is a quantitative approach across the entirety of the HTTP endpoints. Which means that is you want to move over specific HTTP endpoints only it isn't possible. If that's your desire, we can probably create some temporary mobileapps-migration.svc.${dc}.wmnet DNS record, with the caveat that any callers' configuration (e.g. restbase) would have to be updated at least twice (one for the migration to the temporary DNS record and once more back to the normal one).

I would be wary of creating a service discovery record for the migration itself as discovery records are about pooling/depooling datacenters from the service pool, which adds complexity to the process I don't see a reason for yet (but please correct me, I might have missed something).

• Mholloway added a comment.Jun 4 2020, 2:47 PM

Thanks for that explanation, @akosiaris, that process sounds great. Of course with mobileapps the outlier in terms of traffic is the /page/summary endpoint, but given the cautious approach you've described in terms of increasing percentages, I don't think we need (or want) to try to do anything like migrate /page/summary over separately. Does the plan sound good to you, @bearND?

• bearND added a comment.Jun 4 2020, 3:18 PM

Yes, this plan sounds good to me. I think we should probably try to deploy both SCB and k8s instances at roughly the same times during this transition period.

JMeybohm added a comment.Jun 4 2020, 3:29 PM

If you want so start with TLS (via envoy) right away (which would be great!) you need to go through the extra steps of generating certificates (current document draft at https://wikitech.wikimedia.org/wiki/User:Giuseppe_Lavagetto/Add_Tls_On_Kubernetes) and "register" a TCP port at https://wikitech.wikimedia.org/wiki/Service_ports

• Mholloway added a comment.Jun 4 2020, 4:04 PM

Thanks, @JMeybohm. Using TLS right away sounds great! I've reserved port 4102 for mobileapps TLS (as well as 4103 for chromium-render TLS) but I don't believe either @bearND or I have the access to the puppet private repo to generate certificates.

JMeybohm added a comment.Jun 5 2020, 8:23 AM

Oh, my bad. Then we'll create them for you ofc.
Unfortunately starting with TLS right away would not permit the gradual traffic shift Alex was suggesting so it's probably better to start without and migrate to TLS in a second step. :-/

akosiaris added a comment.Jun 5 2020, 9:07 AM
In T218733#6195390, @JMeybohm wrote:

Oh, my bad. Then we'll create them for you ofc.
Unfortunately starting with TLS right away would not permit the gradual traffic shift Alex was suggesting so it's probably better to start without and migrate to TLS in a second step. :-/

+1. Yes, let's decouple those 2 steps.

gerritbot added a comment.Jun 9 2020, 4:42 PM

Change 602155 merged by jenkins-bot:
[operations/deployment-charts@master] Mobileapps: Add initial helmfile stanzas

https://gerrit.wikimedia.org/r/602155

• Mholloway updated the task description. (Show Details)Jun 11 2020, 6:44 PM
LGoto subscribed.Jul 9 2020, 4:22 PM
gerritbot added a comment.Jul 13 2020, 1:05 PM

Change 612273 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Add a temporary non-TLS release

https://gerrit.wikimedia.org/r/612273

gerritbot added a comment.Jul 14 2020, 12:41 PM

Change 612567 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] mobileapps: Add kubernetes nodes next to scb nodes

https://gerrit.wikimedia.org/r/612567

gerritbot added a comment.Jul 15 2020, 10:28 AM

Change 612273 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Add a temporary non-TLS release

https://gerrit.wikimedia.org/r/612273

gerritbot added a comment.Jul 15 2020, 10:44 AM

Change 612567 merged by Alexandros Kosiaris:
[operations/puppet@production] mobileapps: Add kubernetes nodes next to scb nodes

https://gerrit.wikimedia.org/r/612567

Stashbot added a comment.Jul 15 2020, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:12:00Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 2% T218733

gerritbot added a comment.Jul 15 2020, 9:52 PM

Change 612959 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Amend statsd exporter buckets

https://gerrit.wikimedia.org/r/612959

gerritbot added a comment.Jul 15 2020, 9:57 PM

Change 612959 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Amend statsd exporter buckets

https://gerrit.wikimedia.org/r/612959

gerritbot added a comment.Jul 15 2020, 10:19 PM

Change 612966 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: fix prometheus statsd exporter issue in 0.0.10

https://gerrit.wikimedia.org/r/612966

gerritbot added a comment.Jul 15 2020, 10:21 PM

Change 612966 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: fix prometheus statsd exporter issue in 0.0.10

https://gerrit.wikimedia.org/r/612966

Stashbot added a comment.Jul 16 2020, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T12:35:58Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 5% T218733

Stashbot added a comment.Jul 16 2020, 1:02 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T13:02:13Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 10% T218733

Stashbot added a comment.Jul 16 2020, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T13:36:06Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 25% T218733

Stashbot added a comment.Jul 16 2020, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T15:15:30Z] <akosiaris> lower codfw mobileapps kubernetes traffic to 10% T218733. Will open up task for it

• Mholloway closed subtask T258186: Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete as Resolved.Jul 20 2020, 3:14 PM
Stashbot added a comment.Jul 20 2020, 4:27 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-20T16:27:42Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 25% T218733. Take #2

Stashbot added a comment.Jul 21 2020, 8:37 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T08:37:36Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 47% T218733

Stashbot added a comment.Jul 21 2020, 9:58 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T09:58:01Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 72.727272% T218733

Stashbot added a comment.Jul 21 2020, 9:59 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T09:59:48Z] <akosiaris> move all codfw mobileapps nodes (kubernetes and scb) to weight 10. Traffic level remains at 72.727272% flowing to kubernetes, the rest to scb T218733

Stashbot added a comment.Jul 21 2020, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T13:41:57Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 96% T218733

Stashbot added a comment.Jul 21 2020, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T14:35:50Z] <akosiaris> decrease codfw mobileapps kubernetes traffic to 72% T218733. Weird latency patterns exhibited when 92% was reached. See https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=34&fullscreen&orgId=1&from=1595338489749&to=1595342071227&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All

Stashbot added a comment.Jul 22 2020, 8:16 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:16:01Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 96% T218733. Take #2. Let's see if I can reproduce the weird increases in p99 latencies and figure out their cause

gerritbot added a comment.Jul 22 2020, 9:20 AM

Change 615416 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Bump memory limit by 25%

https://gerrit.wikimedia.org/r/615416

gerritbot added a comment.Jul 22 2020, 9:22 AM

Change 615416 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Bump memory limit by 25%

https://gerrit.wikimedia.org/r/615416

Stashbot added a comment.Jul 22 2020, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T09:25:48Z] <akosiaris> bump memory limits for mobileapps by 25% T218733

Stashbot added a comment.Jul 22 2020, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T09:40:45Z] <akosiaris> increase codfw mobileapps kubernetes traffic to 100% T218733

Stashbot added a comment.Jul 22 2020, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T09:46:31Z] <akosiaris> codfw mobileapps kubernetes traffic back to 96% T218733 again. scb pooled again.

Stashbot added a comment.Jul 22 2020, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T09:55:37Z] <akosiaris> bump memory in codfw mobileapps another 20% T218733

gerritbot added a comment.Jul 22 2020, 1:26 PM

Change 615484 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Bump 20x the number of replicas

https://gerrit.wikimedia.org/r/615484

gerritbot added a comment.Jul 22 2020, 1:32 PM

Change 615484 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Bump 20x the number of replicas

https://gerrit.wikimedia.org/r/615484

gerritbot added a comment.Jul 22 2020, 1:49 PM

Change 615494 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Bump quotas

https://gerrit.wikimedia.org/r/615494

gerritbot added a comment.Jul 22 2020, 1:51 PM

Change 615494 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Bump quotas

https://gerrit.wikimedia.org/r/615494

gerritbot added a comment.Jul 23 2020, 8:45 AM

Change 615671 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Bump memory limits another 20%

https://gerrit.wikimedia.org/r/615671

gerritbot added a comment.Jul 23 2020, 8:45 AM

Change 615672 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] mobileapps: Lower replicas to 80 from 240

https://gerrit.wikimedia.org/r/615672

gerritbot added a comment.Jul 23 2020, 8:49 AM

Change 615671 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Bump memory limits another 20%

https://gerrit.wikimedia.org/r/615671

gerritbot added a comment.Jul 23 2020, 8:50 AM

Change 615672 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Lower replicas to 80 from 240

https://gerrit.wikimedia.org/r/615672

Stashbot added a comment.Jul 23 2020, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T09:19:56Z] <akosiaris> lower replica count back to 80 for mobileapps. T218733

Stashbot added a comment.Jul 23 2020, 9:51 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T09:51:53Z] <akosiaris> prepare for pooling kubernetes mobileapps capacity in eqiad. T218733

Stashbot added a comment.Jul 23 2020, 10:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T10:11:43Z] <akosiaris> poole kubernetes in mobileapps/eqiad. T218733

Stashbot added a comment.Jul 23 2020, 11:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T11:18:01Z] <akosiaris> depool scb in mobileapps/eqiad. T218733

akosiaris lowered the priority of this task from High to Low.Jul 23 2020, 11:32 AM
akosiaris updated the task description. (Show Details)

Traffic has been switched fully for the last 18hours in codfw and for the last 1.5 hours in eqiad. I 've added a few followup items on the task for SRE, but otherwise this is looking good.

MSantos awarded a token.Jul 23 2020, 2:48 PM
Stashbot mentioned this in T247012: Archive mediawiki/services/mobileapps/deploy once it's no longer used.Jul 27 2020, 10:08 AM

Mentioned in SAL (#wikimedia-releng) [2020-07-27T10:08:15Z] <James_F> Zuul: [mediawiki/services/mobileapps] Drop npm-docker job T247012 T218733

• Mholloway moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Aug 4 2020, 9:48 PM
• Mholloway edited projects, added Product-Infrastructure-Team-Backlog-Deprecated; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).

This is now tracking for Product Infrastructure.

• Mholloway removed • Mholloway as the assignee of this task.Aug 4 2020, 9:50 PM
• Mholloway moved this task from Upcoming to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.
gerritbot added a comment.Aug 5 2020, 8:08 AM

Change 618485 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] mobileapps: Switch conftool to kubernetes/kubesvc

https://gerrit.wikimedia.org/r/618485

gerritbot added a comment.Aug 5 2020, 8:08 AM

Change 618486 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] mobileapps: Remove from scb conftool config

https://gerrit.wikimedia.org/r/618486

gerritbot added a comment.Aug 5 2020, 8:08 AM

Change 618487 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] mobileapps: Remove mobileapps from scb

https://gerrit.wikimedia.org/r/618487

gerritbot added a comment.Aug 5 2020, 8:08 AM

Change 618488 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] mobileapps: Remove the profile and the role

https://gerrit.wikimedia.org/r/618488

gerritbot added a comment.Aug 5 2020, 8:11 AM

Change 618485 merged by Alexandros Kosiaris:
[operations/puppet@production] mobileapps: Switch conftool to kubernetes/kubesvc

https://gerrit.wikimedia.org/r/618485

gerritbot added a comment.Aug 5 2020, 9:25 AM

Change 618486 merged by Alexandros Kosiaris:
[operations/puppet@production] mobileapps: Remove from scb conftool config

https://gerrit.wikimedia.org/r/618486

gerritbot added a comment.Aug 5 2020, 9:31 AM

Change 618487 merged by Alexandros Kosiaris:
[operations/puppet@production] mobileapps: Remove mobileapps from scb

https://gerrit.wikimedia.org/r/618487

gerritbot added a comment.Aug 5 2020, 1:26 PM

Change 618488 merged by Alexandros Kosiaris:
[operations/puppet@production] mobileapps: Remove the profile and the role

https://gerrit.wikimedia.org/r/618488

akosiaris closed this task as Resolved.Aug 5 2020, 1:48 PM
akosiaris claimed this task.
akosiaris updated the task description. (Show Details)

Resolving. Final puppet related mobileapps pieces have been cleanedup from scb and the repo \o/

• Mholloway closed subtask T215539: Node.js 10 changes encoding for at least one Georgian character as Declined.Mar 5 2021, 9:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL