Page MenuHomePhabricator

kamila (Kamila Součková)
Site Reliability Engineer - ServiceOps

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Mar 16 2023, 2:18 PM (66 w, 2 h)
Availability
Available
IRC Nick
kamila_
LDAP User
Kamila Součková
MediaWiki User
KSoučková-WMF [ Global Accounts ]

Recent Activity

Yesterday

kamila moved T367681: Update terms and timeline of access already granted for AndyRussG from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Wed, Jun 19, 4:36 PM · Patch-For-Review, SRE, LDAP-Access-Requests
kamila updated the task description for T367872: Grant Access to analytics-privatedata-users for DMburugu.
Wed, Jun 19, 3:47 PM · SRE-Access-Requests, Data-Engineering, SRE
kamila added a comment to T367681: Update terms and timeline of access already granted for AndyRussG.

@AndyRussG since you're now with WMDE, we'd like to update your email, can you please let me know your @wikimedia.de email address?

Wed, Jun 19, 3:40 PM · Patch-For-Review, SRE, LDAP-Access-Requests
kamila added a comment to T367872: Grant Access to analytics-privatedata-users for DMburugu.

@DMburugu can you please confirm that you have read the Analytics Data Access User Responsibilities? Thanks!

Wed, Jun 19, 1:02 PM · SRE-Access-Requests, Data-Engineering, SRE
kamila updated the task description for T367872: Grant Access to analytics-privatedata-users for DMburugu.
Wed, Jun 19, 12:57 PM · SRE-Access-Requests, Data-Engineering, SRE
kamila closed T367914: Update "WMDE group" approvers on Wikitech as Resolved.
Wed, Jun 19, 12:37 PM · wikitech.wikimedia.org, SRE-Access-Requests, SRE
kamila created T367967: Requesting content administrator access for Kamila Součková.
Wed, Jun 19, 12:25 PM · wikitech.wikimedia.org
kamila closed T367184: Grant Access to ldap/wmde for Audrey Penven as Resolved.

Done, though it's my first time doing clinic duty, so let me know if it doesn't work :D

Wed, Jun 19, 10:31 AM · SRE, LDAP-Access-Requests
kamila added a member for WMF-NDA: AudreyPenven_WMDE.
Wed, Jun 19, 10:25 AM

Tue, Jun 18

kamila updated subscribers of T367681: Update terms and timeline of access already granted for AndyRussG.

As per the process at https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#WMDE_Group , we need confirmation from an engineering manager. @WMDE-leszek would you mind confirming this request please?

Tue, Jun 18, 6:53 PM · Patch-For-Review, SRE, LDAP-Access-Requests
kamila added a comment to T367681: Update terms and timeline of access already granted for AndyRussG.

@AndyRussG Can you please confirm the new end date? I assume you meant July 30, 2024, correct?

Tue, Jun 18, 5:33 PM · Patch-For-Review, SRE, LDAP-Access-Requests
kamila changed the status of T367184: Grant Access to ldap/wmde for Audrey Penven from In Progress to Stalled.
Tue, Jun 18, 2:58 PM · SRE, LDAP-Access-Requests

Mon, Jun 17

kamila added a comment to T366205: codfw:(3) wikikube-ctrl NIC upgrade to 10G.

wikikube-ctrl1001 looks happy, thanks for the help!

Mon, Jun 17, 3:33 PM · SRE-OnFire, Sustainability (Incident Followup), serviceops, ops-codfw, DC-Ops
kamila changed the status of T367295: Requesting access to private data-based dashboards for Jsn.sherman from Open to Stalled.

waiting for approval

Mon, Jun 17, 1:57 PM · SRE, SRE-Access-Requests
kamila changed the status of T365074: Requesting access to cassandra-staging-devs for milimetric from In Progress to Stalled.

Stalled on @Milimetric signing L3

Mon, Jun 17, 1:56 PM · SRE, SRE-Access-Requests
kamila claimed T365074: Requesting access to cassandra-staging-devs for milimetric.
Mon, Jun 17, 1:52 PM · SRE, SRE-Access-Requests
kamila updated the task description for T365074: Requesting access to cassandra-staging-devs for milimetric.
Mon, Jun 17, 1:51 PM · SRE, SRE-Access-Requests
kamila added a comment to T365074: Requesting access to cassandra-staging-devs for milimetric.

@Milimetric also, you provided an SSH key fingerprint, we need the public key. It should start with something like ssh-rsa AAAA...

There's an existing user already, this only needs to have the group added: L1920 in data.yaml

Mon, Jun 17, 12:39 PM · SRE, SRE-Access-Requests
kamila added a comment to T365074: Requesting access to cassandra-staging-devs for milimetric.

@Milimetric also, you provided an SSH key fingerprint, we need the public key. It should start with something like ssh-rsa AAAA...

Mon, Jun 17, 12:33 PM · SRE, SRE-Access-Requests
kamila added a comment to T365074: Requesting access to cassandra-staging-devs for milimetric.

@Milimetric I cannot find your signatures on L3 nor NDA, could you please ensure that you have signed those?

Mon, Jun 17, 12:27 PM · SRE, SRE-Access-Requests

Fri, Jun 14

kamila awarded T367466: sre.hosts.downtime, and any other maintenance processes, should use auto-extending silences a Like token.
Fri, Jun 14, 1:56 PM · SRE-tools, Infrastructure-Foundations, Spicerack, Observability-Alerting
kamila closed T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G as Resolved.

New NICs seem to be happy including overnight network testing out of sheer paranoia, so calling it done 🎉

Fri, Jun 14, 8:19 AM · SRE, Sustainability (Incident Followup), SRE-OnFire, DC-Ops, serviceops, ops-eqiad
kamila closed T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G, a subtask of T353464: Migrate wikikube control planes to hardware nodes, as Resolved.
Fri, Jun 14, 8:19 AM · Patch-For-Review, serviceops, Prod-Kubernetes, Kubernetes
kamila closed T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G, a subtask of T366094: k8s master capacity issues, as Resolved.
Fri, Jun 14, 8:19 AM · serviceops, SRE

Thu, Jun 13

kamila added a comment to T366205: codfw:(3) wikikube-ctrl NIC upgrade to 10G.

@kamila your plan works for us as well, just depool and power the first server you want to move in your time zone and let us know which one and when we are on site in our time zone we will put in the 10G nic, move it to the new rack and do all the Netbox changes and hang it back to you for re-images. Once you are happy with it we can move to the next one.

Thu, Jun 13, 10:07 AM · SRE-OnFire, Sustainability (Incident Followup), serviceops, ops-codfw, DC-Ops

Wed, Jun 12

kamila reopened T340935: Some apache access logs are invalid json as "Open".

I'm seeing more bad messages that look like an encoding problem:

Wed, Jun 12, 10:40 AM · Observability-Logging, serviceops, MW-on-K8s

Tue, Jun 11

kamila added a comment to T366205: codfw:(3) wikikube-ctrl NIC upgrade to 10G.

@Papaul Thanks for the additional details!

Tue, Jun 11, 1:46 PM · SRE-OnFire, Sustainability (Incident Followup), serviceops, ops-codfw, DC-Ops

Mon, Jun 10

kamila added a comment to T366205: codfw:(3) wikikube-ctrl NIC upgrade to 10G.

@Papaul could you please let me know when would be a good time for you to do this? We don't have any specific time requirements, just that earlier would be nice. I would like to do it in two steps (first just 1 server, then the other 2) for capacity reasons. I am in CEST, so US mornings would work well for me, but I can also decom in advance and reimage the next day if you'd prefer to do it asynchronously, just let me know.

Mon, Jun 10, 3:32 PM · SRE-OnFire, Sustainability (Incident Followup), serviceops, ops-codfw, DC-Ops

Thu, Jun 6

kamila added a comment to T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G.

@VRiley-WMF the reimage of wikikube-ctrl1001 was finally successful, I want to run a few more tests due to having had network(?) problems during the reimage, but I _think_ we should be good to proceed next week, if that works for you. Tuesday would be the best day for me, but I can make other days work too. I believe that capacity-wise, we can move the two remaining machines at the same time if that's easier for you.

Thu, Jun 6, 6:49 PM · SRE, Sustainability (Incident Followup), SRE-OnFire, DC-Ops, serviceops, ops-eqiad

Tue, Jun 4

kamila added a comment to T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G.

@VRiley-WMF Yes, that works, thank you!

Tue, Jun 4, 9:27 AM · SRE, Sustainability (Incident Followup), SRE-OnFire, DC-Ops, serviceops, ops-eqiad

Fri, May 31

kamila added a comment to T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G.

@VRiley-WMF I am in UTC+2, so US mornings are best for me. Would Tuesday work for you? Thank you!

Fri, May 31, 1:25 PM · SRE, Sustainability (Incident Followup), SRE-OnFire, DC-Ops, serviceops, ops-eqiad

Wed, May 29

kamila created T366205: codfw:(3) wikikube-ctrl NIC upgrade to 10G.
Wed, May 29, 4:41 PM · SRE-OnFire, Sustainability (Incident Followup), serviceops, ops-codfw, DC-Ops
kamila created T366204: eqiad:(3) wikikube-ctrl NIC upgrade to 10G.
Wed, May 29, 4:40 PM · SRE, Sustainability (Incident Followup), SRE-OnFire, DC-Ops, serviceops, ops-eqiad

Mar 27 2024

kamila updated the task description for T359423: Migrate charts to Calico Network Policies.
Mar 27 2024, 2:44 PM · Patch-For-Review, Data-Platform-SRE, Prod-Kubernetes, Kubernetes, serviceops

Mar 25 2024

TheDJ awarded T357296: Create new flavour of shellbox for video transcoding a Party Time token.
Mar 25 2024, 1:27 PM · Video, MW-on-K8s, serviceops
kamila closed T357296: Create new flavour of shellbox for video transcoding as Resolved.

Based on some quick tests the image seems to be working \o/

Mar 25 2024, 12:23 PM · Video, MW-on-K8s, serviceops
kamila closed T357296: Create new flavour of shellbox for video transcoding, a subtask of T356241: Move video transcoding to use Shellbox, as Resolved.
Mar 25 2024, 12:23 PM · Video, TimedMediaHandler, MW-on-K8s, serviceops

Mar 21 2024

kamila added a comment to T197470: find a way to systematically update the deployment server name across all repos.

For the record, I used sudo bash -c 'find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy1002.eqiad.wmnet/git_server: deploy2002.codfw.wmnet/"' (or the other way around) on all deployment servers to work around this.

Mar 21 2024, 11:53 AM · Release-Engineering-Team (Priority Backlog 📥), Scap
kamila added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

Note also the increase in RX (but not really TX) traffic that coincides with these:

Screenshot from 2024-03-21 11-59-27.png (1×1 px, 217 KB)

Mar 21 2024, 11:21 AM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team

Feb 26 2024

kamila added a comment to T312067: SRE needs a logo.

Inspired by some of the above:

Screenshot_20240226_161618_Sketchbook.png (780×769 px, 174 KB)
Screenshot_20240226_161654_Sketchbook.png (641×710 px, 104 KB)
Screenshot_20240226_161719_Sketchbook.png (625×754 px, 136 KB)

Feb 26 2024, 3:23 PM · SRE Program Management, Logos, SRE

Feb 21 2024

kamila changed the status of T357309: Create a deployment for `shellbox-timedmedia`, a subtask of T356241: Move video transcoding to use Shellbox, from Open to In Progress.
Feb 21 2024, 10:42 AM · Video, TimedMediaHandler, MW-on-K8s, serviceops
kamila changed the status of T357309: Create a deployment for `shellbox-timedmedia` from Open to In Progress.
Feb 21 2024, 10:42 AM · Patch-For-Review, Video, TimedMediaHandler, MW-on-K8s, serviceops

Feb 19 2024

kamila created T357907: Migrate remaining internal MW API traffic to k8s.
Feb 19 2024, 2:19 PM · MW-on-K8s, serviceops

Feb 12 2024

kamila claimed T357309: Create a deployment for `shellbox-timedmedia`.
Feb 12 2024, 6:15 PM · Patch-For-Review, Video, TimedMediaHandler, MW-on-K8s, serviceops

Feb 7 2024

kamila updated the task description for T356877: Increase visibility of kubernetes network status.
Feb 7 2024, 3:09 PM · observability, Prod-Kubernetes, serviceops, Kubernetes
kamila updated the task description for T356877: Increase visibility of kubernetes network status.
Feb 7 2024, 3:01 PM · observability, Prod-Kubernetes, serviceops, Kubernetes
kamila created T356877: Increase visibility of kubernetes network status.
Feb 7 2024, 2:59 PM · observability, Prod-Kubernetes, serviceops, Kubernetes

Feb 6 2024

kamila added a comment to T356709: Debian installer waits for input for network config during host reimage.

mw1386 seems to be going fine now, so yes, we can close this. Sorry and thanks for finding the cause @Volans <3

Feb 6 2024, 12:23 PM · Infrastructure-Foundations, serviceops

Feb 5 2024

kamila created T356709: Debian installer waits for input for network config during host reimage.
Feb 5 2024, 9:45 PM · Infrastructure-Foundations, serviceops

Feb 1 2024

kamila created P56055 POC benthos creating k8s jobs from kafka.
Feb 1 2024, 11:49 AM

Jan 23 2024

kamila closed T355187: Reimage cookbook fails to downtime hosts when run concurrently as Resolved.

I believe the above patch fixed it, so I'm closing this. I will reopen in case I see the race again.

Jan 23 2024, 12:29 PM · SRE-tools, Infrastructure-Foundations

Jan 17 2024

kamila closed T350846: Migrate mobileapps to k8s as Resolved.

All traffic is now going to k8s \o/

Jan 17 2024, 5:30 PM · Patch-For-Review, Release-Engineering-Team (Seen), SRE, Traffic, serviceops, MW-on-K8s
kamila closed T350846: Migrate mobileapps to k8s, a subtask of T333120: Migrate internal traffic to k8s, as Resolved.
Jan 17 2024, 5:30 PM · Patch-For-Review, Release-Engineering-Team (Seen), SRE, Traffic, serviceops, MW-on-K8s

Jan 16 2024

kamila created T355187: Reimage cookbook fails to downtime hosts when run concurrently.
Jan 16 2024, 8:20 PM · SRE-tools, Infrastructure-Foundations

Jan 15 2024

kamila awarded T353836: RequestTimeoutException when attempting to visit user page of non-existent user a Love token.
Jan 15 2024, 1:28 PM · SecTeam-Processed, Performance Issue, MediaWiki-Blocks, MediaWiki-Engineering, Security, Wikimedia-production-error

Jan 10 2024

kamila closed T354413: Reboot issues for mw13[77-83].eqiad.wmnet as Resolved.

With Alex's patches, I ran 7 reimages and 20 reboots without the issue reappearing. It might be worthwhile to understand the issue better to see if that workaround is adequate, but as it appears to not be blocking us anymore, I'm closing this.

Jan 10 2024, 6:49 PM · serviceops, MW-on-K8s
kamila closed T354413: Reboot issues for mw13[77-83].eqiad.wmnet, a subtask of T351074: Move servers from the appserver/api cluster to kubernetes, as Resolved.
Jan 10 2024, 6:48 PM · serviceops, MW-on-K8s

Jan 8 2024

kamila updated the task description for T354413: Reboot issues for mw13[77-83].eqiad.wmnet.
Jan 8 2024, 5:42 PM · serviceops, MW-on-K8s
kamila triaged T354413: Reboot issues for mw13[77-83].eqiad.wmnet as High priority.
Jan 8 2024, 5:41 PM · serviceops, MW-on-K8s
kamila added a comment to T354413: Reboot issues for mw13[77-83].eqiad.wmnet.

Note a few other things that have been tried:

Jan 8 2024, 5:38 PM · serviceops, MW-on-K8s

Jan 5 2024

kamila added a comment to T354413: Reboot issues for mw13[77-83].eqiad.wmnet.

Note that this is non-deterministic: the problem seems to happen more than half the time but far from always. So several reboots may be required to reproduce. Yay!

Jan 5 2024, 5:07 PM · serviceops, MW-on-K8s
kamila added a comment to T354413: Reboot issues for mw13[77-83].eqiad.wmnet.

Is this reproducible with every reboot or just some? One thing worth doing is to connect to the serial console an then issue a reboot over Cumin. Maybe we're seeing a kernel oops during the system tear down?

Jan 5 2024, 12:24 PM · serviceops, MW-on-K8s
kamila added a comment to T354413: Reboot issues for mw13[77-83].eqiad.wmnet.

Additional findings:

  • the watchdog: watchdog0: watchdog did not stop! message seems to be a red herring, it's always there
  • the problem only occurs when running nohup reboot (which is what the cookbooks do), not when I run just reboot
  • it cannot be reproduced on an insetup system with overlayfs in use
Jan 5 2024, 12:22 PM · serviceops, MW-on-K8s
kamila added a comment to T354413: Reboot issues for mw13[77-83].eqiad.wmnet.

Note that we have tried updating the firmware: mw1388 is on new UEFI and iDRAC and still exhibits this problem.

Jan 5 2024, 11:43 AM · serviceops, MW-on-K8s
kamila created T354413: Reboot issues for mw13[77-83].eqiad.wmnet.
Jan 5 2024, 11:06 AM · serviceops, MW-on-K8s

Dec 5 2023

kamila closed T264625: Deploy kube-state-metrics as Resolved.

KSM is deployed in other clusters and appears to work, so I'm closing this :-)

Dec 5 2023, 10:54 AM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes
kamila closed T264625: Deploy kube-state-metrics, a subtask of T266216: Increase visibility of container/pod ressource exhaustion , as Resolved.
Dec 5 2023, 10:54 AM · observability, serviceops, Prod-Kubernetes, Kubernetes

Nov 21 2023

kamila added a comment to T264625: Deploy kube-state-metrics.

KSM in staging-eqiad was in a half installed state (probably due to prematurely terminated helmfile/helm command): [...]

Nov 21 2023, 4:33 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes

Nov 13 2023

kamila added a comment to T264625: Deploy kube-state-metrics.

With @JMeybohm's suggestion (T264625#9324445) we are at around 60k timeseries.

Nov 13 2023, 4:55 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes

Nov 10 2023

kamila created P53291 kube-state-metrics sample.
Nov 10 2023, 10:23 AM

Nov 9 2023

kamila added a comment to T264625: Deploy kube-state-metrics.

@fgiunchedi is adding up to 100k timeseries per k8s cluster OK?

Not sure off the bat, do you have a dump or a sample of the metrics? Will they be collected by Prometheus from a single (per-cluster) endpoint?

Nov 9 2023, 2:25 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes
kamila updated subscribers of T264625: Deploy kube-state-metrics.

@fgiunchedi is adding up to 100k timeseries per k8s cluster OK?

Nov 9 2023, 1:40 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes
kamila added a comment to T264625: Deploy kube-state-metrics.

A prototype deployment in eqiad with a pretty vanilla config:

  • generates <80k timeseries; so let's call it up to ~100k timeseries per cluster
  • uses <50mCPU and <120MB memory with (curl-simulated) metrics scraping on, so I believe that deploying a single replica is fine
Nov 9 2023, 1:16 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes

Nov 3 2023

kamila changed the status of T350445: not possible to edit in the german language wikipedia from Open to In Progress.
Nov 3 2023, 12:37 AM · User-notice-archive, DBA

Oct 31 2023

kamila changed the status of T264625: Deploy kube-state-metrics, a subtask of T266216: Increase visibility of container/pod ressource exhaustion , from Stalled to In Progress.
Oct 31 2023, 5:02 PM · observability, serviceops, Prod-Kubernetes, Kubernetes
kamila changed the status of T264625: Deploy kube-state-metrics from Stalled to In Progress.
Oct 31 2023, 5:02 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes

Oct 16 2023

kamila created T348990: Simplify switchover of deployment server.
Oct 16 2023, 2:29 PM · serviceops, Datacenter-Switchover
kamila moved T348985: Automate the "make sure email works" step of the DC switchover from Backlog to Automation improvements on the Datacenter-Switchover board.
Oct 16 2023, 1:50 PM · serviceops, Datacenter-Switchover
kamila triaged T348985: Automate the "make sure email works" step of the DC switchover as Low priority.
Oct 16 2023, 1:49 PM · serviceops, Datacenter-Switchover
kamila created T348985: Automate the "make sure email works" step of the DC switchover.
Oct 16 2023, 1:49 PM · serviceops, Datacenter-Switchover

Oct 11 2023

Clement_Goubert awarded T343801: Create kube-state-metrics docker image a Love token.
Oct 11 2023, 2:46 PM · Prod-Kubernetes, serviceops, Kubernetes
kamila closed T343801: Create kube-state-metrics docker image, a subtask of T264625: Deploy kube-state-metrics, as Resolved.
Oct 11 2023, 2:36 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes
kamila closed T343801: Create kube-state-metrics docker image as Resolved.

Image built and published.

Oct 11 2023, 2:36 PM · Prod-Kubernetes, serviceops, Kubernetes

Oct 10 2023

kamila changed the status of T343801: Create kube-state-metrics docker image, a subtask of T264625: Deploy kube-state-metrics, from Open to In Progress.
Oct 10 2023, 2:51 PM · Prod-Kubernetes, serviceops, User-jijiki, Kubernetes
kamila changed the status of T343801: Create kube-state-metrics docker image from Open to In Progress.
Oct 10 2023, 2:51 PM · Prod-Kubernetes, serviceops, Kubernetes

Oct 2 2023

kamila added a comment to T345265: CommRel support for September 2023 Datacenter Switchover.

Thanks a lot @Trizek-WMF !

Oct 2 2023, 1:20 PM · MoveComms-Support (Jul-Sep-2023), serviceops, Datacenter-Switchover, SRE
kamila closed T345263: September 2023 Datacenter Switchover as Resolved.

All disruptive switchover-related work is finished and things are stable. The switchover went smoothly and had minimal user impact, while also uncovering issues that we need to know about in order to improve our infrastructure and processes. The read-only period lasted 2min 22s.

Oct 2 2023, 1:19 PM · Performance-Team, Data-Persistence, serviceops, Datacenter-Switchover, SRE

Sep 25 2023

kamila closed T346474: Sept 2023 Switchover Checklist: MediaWiki, a subtask of T345263: September 2023 Datacenter Switchover, as Resolved.
Sep 25 2023, 1:53 PM · Performance-Team, Data-Persistence, serviceops, Datacenter-Switchover, SRE
kamila closed T346474: Sept 2023 Switchover Checklist: MediaWiki as Resolved.
Sep 25 2023, 1:53 PM · serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346474: Sept 2023 Switchover Checklist: MediaWiki.
Sep 25 2023, 1:53 PM · serviceops, Datacenter-Switchover, SRE
kamila closed T346472: Sept 2023 Switchover: list new primary DC servers first in debug.json, a subtask of T345263: September 2023 Datacenter Switchover, as Resolved.
Sep 25 2023, 1:52 PM · Performance-Team, Data-Persistence, serviceops, Datacenter-Switchover, SRE
kamila closed T346472: Sept 2023 Switchover: list new primary DC servers first in debug.json as Resolved.
Sep 25 2023, 1:52 PM · serviceops, Datacenter-Switchover, SRE

Sep 22 2023

kamila created T347142: Grant access to security@wikimedia.org to user ksouckova.
Sep 22 2023, 9:33 AM · SecTeam-Processed, Security-Team

Sep 20 2023

RZamora-WMF awarded T345265: CommRel support for September 2023 Datacenter Switchover a Party Time token.
Sep 20 2023, 6:39 PM · MoveComms-Support (Jul-Sep-2023), serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346474: Sept 2023 Switchover Checklist: MediaWiki.
Sep 20 2023, 2:40 PM · serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346472: Sept 2023 Switchover: list new primary DC servers first in debug.json.
Sep 20 2023, 2:17 PM · serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346474: Sept 2023 Switchover Checklist: MediaWiki.
Sep 20 2023, 2:16 PM · serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346474: Sept 2023 Switchover Checklist: MediaWiki.
Sep 20 2023, 1:31 PM · serviceops, Datacenter-Switchover, SRE
kamila updated the task description for T346474: Sept 2023 Switchover Checklist: MediaWiki.
Sep 20 2023, 10:46 AM · serviceops, Datacenter-Switchover, SRE

Sep 19 2023

kamila closed T346330: Sept 2023 Switchover Checklist: Services & Traffic, a subtask of T345263: September 2023 Datacenter Switchover, as Resolved.
Sep 19 2023, 4:31 PM · Performance-Team, Data-Persistence, serviceops, Datacenter-Switchover, SRE