Page MenuHomePhabricator

jijiki (effie mouzeli)
is an animal

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Aug 14 2018, 10:50 AM (66 w, 2 d)
Availability
Available
IRC Nick
jijiki
LDAP User
Effie Mouzeli
MediaWiki User
EMouzeli (WMF) [ Global Accounts ]

Recent Activity

Today

jijiki added a comment to T195847: Clean up artifacts from LaTeX based math rendering.

@Dzahn do you know the status of this? can we mark it as resolved?

Thu, Nov 21, 2:03 PM · Operations, Math

Yesterday

jijiki added a comment to T236963: Deploy version 1.10.0 of wikidiff2 to production.

Hehe... @jijiki could you do deployment-mediawiki-parsoid-* as well please? all the rest.php requests are routed there..

Wed, Nov 20, 10:24 PM · serviceops, Operations, wikidiff2, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP), Core Platform Team Workboards (Green)
jijiki moved T238685: php-fpm isn't restarted when deploys are rolled back from Backlog to Doing on the serviceops board.
Wed, Nov 20, 6:54 PM · Core Platform Team Workboards (Clinic Duty Team), serviceops, Parsoid-PHP
jijiki moved T236699: Build a black-box httpd testing framework from Backlog to Doing on the serviceops board.
Wed, Nov 20, 6:54 PM · Patch-For-Review, Wikimedia-Apache-configuration, Operations, serviceops
jijiki moved T236963: Deploy version 1.10.0 of wikidiff2 to production from Backlog to Doing on the serviceops board.
Wed, Nov 20, 6:53 PM · serviceops, Operations, wikidiff2, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP), Core Platform Team Workboards (Green)
jijiki added a comment to T236963: Deploy version 1.10.0 of wikidiff2 to production.

Version 1.10.0-1~wmf1 has been deployed to deployment-mediawiki-09 and deployment-mediawiki-07. Please let me know if it works as it should so we can proceed with the canaries in prod.

Wed, Nov 20, 6:42 PM · serviceops, Operations, wikidiff2, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP), Core Platform Team Workboards (Green)
jijiki added a project to T236437: rack/setup/install mw13[49-84].eqiad.wmnet: serviceops.
Wed, Nov 20, 3:30 PM · serviceops, Operations, ops-eqiad

Tue, Nov 19

jijiki added a comment to T238685: php-fpm isn't restarted when deploys are rolled back.

@ssastr php-fpm will be restarted during scap deployments only when a server's opcache free is below 100MB, I can check the code if there is an exception to that

Tue, Nov 19, 10:34 PM · Core Platform Team Workboards (Clinic Duty Team), serviceops, Parsoid-PHP

Fri, Nov 15

jijiki added a comment to T231011: (euwiki) Mysterious, coordinated slowdowns every ~ 25 minutes on API servers.

@Theklan We suspect it is something on the production side since we have noticed this behaviour in the past. Moreover, this is not the only wiki where templates are changed, but it manages to standout in our investigations, which shouldn't happen. Nevertheless, thanks for letting us know :)

Fri, Nov 15, 5:42 PM · PHP 7.2 support, serviceops, Operations
jijiki renamed T231011: (euwiki) Mysterious, coordinated slowdowns every ~ 25 minutes on API servers from Mysterious, coordinated slowdowns every ~ 25 minutes on API servers to (euwiki) Mysterious, coordinated slowdowns every ~ 25 minutes on API servers.
Fri, Nov 15, 10:51 AM · PHP 7.2 support, serviceops, Operations

Thu, Nov 14

jijiki removed a project from T140443: Deploy new version of wikidiff2 package: serviceops.
Thu, Nov 14, 7:43 AM · Community-Tech, Patch-For-Review, Operations, wikidiff2
jijiki added projects to T236963: Deploy version 1.10.0 of wikidiff2 to production: Operations, serviceops.
Thu, Nov 14, 7:42 AM · serviceops, Operations, wikidiff2, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP), Core Platform Team Workboards (Green)
jijiki added a project to T140443: Deploy new version of wikidiff2 package: serviceops.
Thu, Nov 14, 7:42 AM · Community-Tech, Patch-For-Review, Operations, wikidiff2
jijiki added a comment to T236963: Deploy version 1.10.0 of wikidiff2 to production.

Currently we have 1.9.0 on releases.w.o and on the servers. Please upload the new version on releases, and I can go ahead with the build.

Thu, Nov 14, 7:41 AM · serviceops, Operations, wikidiff2, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP), Core Platform Team Workboards (Green)

Wed, Nov 13

jijiki added a comment to T224946: Netbox Alert Cleanups.

I have downtimed some of the alerts, but it will expire in a couple of hours from now

Wed, Nov 13, 6:11 PM · Operations, observability, User-crusnov, netbox, SRE-tools
jijiki reassigned T237239: Upgrade to PHP 7.2.24 from jijiki to Dzahn.
Wed, Nov 13, 5:35 PM · serviceops, Operations
jijiki updated subscribers of T237239: Upgrade to PHP 7.2.24.

@Dzahn I have left only phab* and scandium, can you take care of them? :)

Wed, Nov 13, 5:35 PM · serviceops, Operations
jijiki added a subtask for T237239: Upgrade to PHP 7.2.24: T231881: Conffile handling for PHP 7.2 packages.
Wed, Nov 13, 11:58 AM · serviceops, Operations
jijiki added a parent task for T231881: Conffile handling for PHP 7.2 packages: T237239: Upgrade to PHP 7.2.24.
Wed, Nov 13, 11:58 AM · Operations
jijiki triaged T231881: Conffile handling for PHP 7.2 packages as Normal priority.
Wed, Nov 13, 11:58 AM · Operations
jijiki added a project to T224946: Netbox Alert Cleanups: Operations.
Wed, Nov 13, 11:10 AM · Operations, observability, User-crusnov, netbox, SRE-tools
jijiki added a comment to T224946: Netbox Alert Cleanups.

I am not sure this is related, but we get many alerts of

Wed, Nov 13, 11:10 AM · Operations, observability, User-crusnov, netbox, SRE-tools

Tue, Nov 12

jijiki updated the task description for T236806: Reimage mwdebug1002 and mw1317.
Tue, Nov 12, 2:54 PM · Operations, serviceops
jijiki updated subscribers of T214734: PHP Fatal error: The UdpSocket to 127.0.0.1:10514 has been closed (from Monolog/SyslogUdp on mwdebug1002).

@Krinkle Please let me know if the issue on mwdebug1002 persists, as the server has been reimaged :)

Tue, Nov 12, 2:49 PM · Release-Engineering-Team, Performance-Team (Radar), serviceops, Wikimedia-production-error, User-fgiunchedi, Operations
jijiki claimed T237239: Upgrade to PHP 7.2.24.
Tue, Nov 12, 11:43 AM · serviceops, Operations
jijiki moved T237239: Upgrade to PHP 7.2.24 from Backlog to Doing on the serviceops board.
Tue, Nov 12, 11:43 AM · serviceops, Operations

Mon, Nov 4

jijiki updated the task description for T229792: Remove HHVM from production.
Mon, Nov 4, 9:30 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki updated the task description for T229792: Remove HHVM from production.
Mon, Nov 4, 8:44 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM

Fri, Nov 1

jijiki closed Unknown Object (Task), a subtask of T215411: (OoW) thumbor1004 memory errors, as Invalid.
Fri, Nov 1, 4:21 PM · User-jijiki, Thumbor, ops-eqiad, serviceops, Operations
jijiki updated the task description for T229792: Remove HHVM from production.
Fri, Nov 1, 9:43 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki added a comment to T130329: Icinga should alert on free disk space < 15% (now < 12%) on Elasticsearch hosts.

We really need to figure out what to do with the elastic1025 alert, it has been alerting even more aggressively lately I think

Fri, Nov 1, 9:02 AM · Discovery-Search (Current work), Patch-For-Review, Operations, Discovery, Elasticsearch

Thu, Oct 31

jijiki added a comment to T234283: Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal.

@Krinkle we are looking into it, tx

Thu, Oct 31, 7:33 PM · Performance-Team (Radar), Operations, serviceops, observability
jijiki reassigned T234418: Upgrade our php-xdebug package for php7.2 from jijiki to hashar.
Thu, Oct 31, 2:27 PM · Patch-For-Review, serviceops, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage
jijiki closed T234418: Upgrade our php-xdebug package for php7.2 as Resolved.

cool :)

Thu, Oct 31, 1:43 PM · Patch-For-Review, serviceops, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage
jijiki closed T234418: Upgrade our php-xdebug package for php7.2, a subtask of T232706: mwcore-phpunit-coverage-master times out after 5 hours, as Resolved.
Thu, Oct 31, 1:43 PM · Release-Engineering-Team-TODO (201911), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage
jijiki triaged T229792: Remove HHVM from production as High priority.
Thu, Oct 31, 11:39 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki added a comment to T234283: Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal.

Please ping if there are more things to be done for this task:)

Thu, Oct 31, 11:38 AM · Performance-Team (Radar), Operations, serviceops, observability
jijiki added a comment to T234418: Upgrade our php-xdebug package for php7.2.

Cherry picked and packaged. Please ping when you test it so I can upload it:)

Thu, Oct 31, 11:36 AM · Patch-For-Review, serviceops, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage

Wed, Oct 30

jijiki added a comment to T229286: Resolve service instability due to excessive event loop blockage since starting PCS response pregeneration.

@Mholloway Can we mark this as resolved?

Wed, Oct 30, 11:17 AM · Patch-For-Review, Product-Infrastructure-Team-Backlog, Epic, Page Content Service, Mobile-Content-Service, serviceops
jijiki renamed T236800: Ensure apcu incr/decr are atomic from Upgrade php-apcu to 5.1.18 to Ensure apcu incr/decr are atomic.
Wed, Oct 30, 6:14 AM · Performance-Team (Radar), MediaWiki-Cache, Core Platform Team, serviceops
jijiki closed T236872: Slow loading and connectivity issues on some wikis as Resolved.

I couldn't reproduce this slowness (from Europe) right the task was opened, I am marking it as resolved for now, please reopen if you experience it again, thank you!

Wed, Oct 30, 5:44 AM · Traffic, Operations
jijiki renamed T236800: Ensure apcu incr/decr are atomic from Ensure apcu incr/decr are atomic to Upgrade php-apcu to 5.1.18.
Wed, Oct 30, 5:40 AM · Performance-Team (Radar), MediaWiki-Cache, Core Platform Team, serviceops

Tue, Oct 29

jijiki created T236806: Reimage mwdebug1002 and mw1317.
Tue, Oct 29, 3:34 PM · Operations, serviceops
jijiki triaged T236799: mw2225 keeps sending cronspam for hhvm-needs-restart as Low priority.
Tue, Oct 29, 2:57 PM · Operations, User-jijiki, serviceops
jijiki created T236799: mw2225 keeps sending cronspam for hhvm-needs-restart .
Tue, Oct 29, 2:57 PM · Operations, User-jijiki, serviceops
jijiki renamed T231011: (euwiki) Mysterious, coordinated slowdowns every ~ 25 minutes on API servers from Mysterious, coordinated slowdowns every ~ 25 minutes on mw1347,mw1348 (php7 api servers) to Mysterious, coordinated slowdowns every ~ 25 minutes on API servers.
Tue, Oct 29, 1:29 PM · PHP 7.2 support, serviceops, Operations
jijiki claimed T234418: Upgrade our php-xdebug package for php7.2.
Tue, Oct 29, 11:31 AM · Patch-For-Review, serviceops, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage

Mon, Oct 28

jijiki removed a project from T236500: large number of 504 errors from ulsfo: Security.
Mon, Oct 28, 7:20 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations

Sun, Oct 27

jijiki changed the visibility for T236500: large number of 504 errors from ulsfo.
Sun, Oct 27, 11:26 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations
jijiki added a comment to T236500: large number of 504 errors from ulsfo.

Looks like it started on Friday ~12:00-13:00 UTC

Sun, Oct 27, 11:24 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations
jijiki added a comment to T236500: large number of 504 errors from ulsfo.

@Bugreporter oups:)

Sun, Oct 27, 10:36 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations
jijiki changed the visibility for T236500: large number of 504 errors from ulsfo.
Sun, Oct 27, 10:35 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations
jijiki triaged T236500: large number of 504 errors from ulsfo as High priority.
Sun, Oct 27, 10:29 AM · Security, ops-ulsfo, Traffic, Wikidata, Wikidata-Query-Service, Operations

Fri, Oct 25

jijiki added a comment to T229792: Remove HHVM from production.

merged the above because we were getting cron spam from appservers with "/usr/local/bin/hhvm-needs-restart: not found"

Fri, Oct 25, 4:13 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki closed T219127: SRE FY19-20 Q1 goal: complete the transition to PHP7 as Resolved.

I believe this is done:)
Removing HHVM is continuing under T229792

Fri, Oct 25, 6:24 AM · Operations, serviceops
jijiki updated the task description for T219127: SRE FY19-20 Q1 goal: complete the transition to PHP7.
Fri, Oct 25, 6:23 AM · Operations, serviceops

Thu, Oct 24

jijiki updated the task description for T229792: Remove HHVM from production.
Thu, Oct 24, 3:59 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki updated the task description for T236375: POST requests to pl.wikibooks.org fail with 'Maximum execution time of 180 seconds exceeded' since 2019-10-24 ~11:30 UTC.
Thu, Oct 24, 3:55 PM · MediaWiki-Parser, Parsing-Team, serviceops-radar, Wikimedia-production-error
jijiki claimed T234283: Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal.
Thu, Oct 24, 3:22 PM · Performance-Team (Radar), Operations, serviceops, observability
jijiki added a comment to T234283: Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal.

I will take a look tomorrow, sorry for delaying this

Thu, Oct 24, 3:22 PM · Performance-Team (Radar), Operations, serviceops, observability
jijiki added a project to T236375: POST requests to pl.wikibooks.org fail with 'Maximum execution time of 180 seconds exceeded' since 2019-10-24 ~11:30 UTC: serviceops-radar.
Thu, Oct 24, 12:22 PM · MediaWiki-Parser, Parsing-Team, serviceops-radar, Wikimedia-production-error
jijiki created T236375: POST requests to pl.wikibooks.org fail with 'Maximum execution time of 180 seconds exceeded' since 2019-10-24 ~11:30 UTC.
Thu, Oct 24, 12:22 PM · MediaWiki-Parser, Parsing-Team, serviceops-radar, Wikimedia-production-error

Wed, Oct 23

jijiki added a comment to T235755: Increased POST latency for MW app servers (Oct 2019).

Frankly, I am still trying to find out, it had some sort of a pattern for a bit, but then it wouldn't. What I have been noticing during the day, is that suddenly our appserver POST requests avg doubles or tiples for a while, and then goes back to "normal". I was just looking for a second opinion on this, I will keep monitoring it. T235872 regards GET requests, so it is another mystery we should solve.

Wed, Oct 23, 9:14 PM · Performance-Team (Radar), serviceops, Operations
jijiki updated the task description for T235755: Increased POST latency for MW app servers (Oct 2019).
Wed, Oct 23, 9:08 PM · Performance-Team (Radar), serviceops, Operations
jijiki added a comment to T236292: php-fpm invalid opcode on mw1317.

mw1317 will be reimaged, but not yet. We will keep it around (but off production) until someone can have a closer look

Wed, Oct 23, 4:21 PM · Operations, serviceops
jijiki created T236292: php-fpm invalid opcode on mw1317.
Wed, Oct 23, 4:18 PM · Operations, serviceops
jijiki created T236253: systemd-coredump can make a system unresponsive.
Wed, Oct 23, 12:46 PM · Patch-For-Review, serviceops, Operations
jijiki added a comment to T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC).

ps1-a6-eqiad is shown as down in icinga, I believe that is expected?

Wed, Oct 23, 7:15 AM · DC-Ops, Operations, ops-eqiad

Oct 22 2019

jijiki added a comment to T234887: ATS-tls nodes on the text cluster have a slightly higher rate of failed fetches on varnish-fe.

It appears we are also having fetch errors, possibly due to timeouts as well mostly on two servers where we have enabled ats-be, cp1075 and cp4027

Oct 22 2019, 12:45 PM · Patch-For-Review, Traffic, Operations
jijiki added a comment to T234418: Upgrade our php-xdebug package for php7.2.

@hashar I will look into it, sorry for not getting to it sooner

Oct 22 2019, 11:50 AM · Patch-For-Review, serviceops, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Test-Coverage
jijiki updated the task description for T235872: Investigate recurrent GET latency spikes on MediaWiki appservers (Oct 16).
Oct 22 2019, 5:26 AM · Performance-Team (Radar), Wikimedia-General-or-Unknown, serviceops, Operations

Oct 21 2019

jijiki added a comment to T214734: PHP Fatal error: The UdpSocket to 127.0.0.1:10514 has been closed (from Monolog/SyslogUdp on mwdebug1002).

Server will be reimaged, I will ping here when it is done :)

Oct 21 2019, 4:45 PM · Release-Engineering-Team, Performance-Team (Radar), serviceops, Wikimedia-production-error, User-fgiunchedi, Operations
jijiki added a subtask for T229792: Remove HHVM from production: T236048: Move debugging symbols and tools to a new class.
Oct 21 2019, 10:45 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki added a parent task for T236048: Move debugging symbols and tools to a new class: T229792: Remove HHVM from production.
Oct 21 2019, 10:45 AM · Patch-For-Review, Operations, serviceops
jijiki claimed T236048: Move debugging symbols and tools to a new class.
Oct 21 2019, 10:45 AM · Patch-For-Review, Operations, serviceops
jijiki created T236048: Move debugging symbols and tools to a new class.
Oct 21 2019, 10:44 AM · Patch-For-Review, Operations, serviceops
jijiki moved T226373: Swift object servers become briefly unresponsive on a regular basis from Next up to Backlog on the serviceops board.
Oct 21 2019, 9:41 AM · Performance-Team (Radar), User-jijiki, serviceops, Patch-For-Review, SRE-swift-storage, Operations
jijiki moved T91474: Custom 404 error page should apply to *.php from Doing to Next up on the serviceops board.
Oct 21 2019, 9:41 AM · serviceops, User-jijiki, Regression, Wikimedia-General-or-Unknown
jijiki moved T229792: Remove HHVM from production from Next up to Doing on the serviceops board.
Oct 21 2019, 9:41 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki added a comment to T203959: SRE quarterly goal: allow MediaWiki requests to be served by PHP7 alongside HHVM.

@Joe should we Resolve this?

Oct 21 2019, 9:36 AM · Operations
jijiki placed T236004: stat1005 cron spam from prometheus-amd-rocm-stat up for grabs.
Oct 21 2019, 1:47 AM · Analytics, Operations
jijiki created T236004: stat1005 cron spam from prometheus-amd-rocm-stat.
Oct 21 2019, 1:47 AM · Analytics, Operations

Oct 20 2019

jijiki added projects to T210137: Handle unknown stats in rsyslog_exporter: serviceops, User-jijiki.

This is polluting our messages on mw* servers as well:

Oct 20 2019, 7:59 AM · User-jijiki, serviceops, observability

Oct 18 2019

jijiki added a comment to T231586: Deploy Updated wikidiff2 C++ Engine.

@Pchelolo @WMDE-Fisch @thiemowmde version 1.9.0-2~wmf1 is live, please ping me if we need to rollback

Oct 18 2019, 11:48 AM · Core Platform Team Workboards (Green), CPT Initiatives (Core REST API in PHP), wikidiff2
jijiki closed T234175: Deploy wikidiff2 v1.9.0, a subtask of T231586: Deploy Updated wikidiff2 C++ Engine, as Resolved.
Oct 18 2019, 11:47 AM · Core Platform Team Workboards (Green), CPT Initiatives (Core REST API in PHP), wikidiff2
jijiki closed T234175: Deploy wikidiff2 v1.9.0 as Resolved.
Oct 18 2019, 11:47 AM · Operations, serviceops

Oct 17 2019

jijiki added a comment to T231586: Deploy Updated wikidiff2 C++ Engine.

I will roll out to production tomorrow. The process is to upload the package to our wikimedia-stretch repo, update all servers, and then rolling restart php-fpm.

Oct 17 2019, 10:14 PM · Core Platform Team Workboards (Green), CPT Initiatives (Core REST API in PHP), wikidiff2
jijiki created T235787: Missing annotations for sync-wikiversions.
Oct 17 2019, 5:15 PM · Scap, serviceops, Release-Engineering-Team
jijiki added a comment to T231586: Deploy Updated wikidiff2 C++ Engine.

@Pchelolo done

Oct 17 2019, 12:56 PM · Core Platform Team Workboards (Green), CPT Initiatives (Core REST API in PHP), wikidiff2
jijiki updated the task description for T235755: Increased POST latency for MW app servers (Oct 2019).
Oct 17 2019, 12:15 PM · Performance-Team (Radar), serviceops, Operations
jijiki updated the task description for T235755: Increased POST latency for MW app servers (Oct 2019).
Oct 17 2019, 12:14 PM · Performance-Team (Radar), serviceops, Operations
jijiki added projects to T235755: Increased POST latency for MW app servers (Oct 2019): Operations, serviceops, Performance-Team.
Oct 17 2019, 12:14 PM · Performance-Team (Radar), serviceops, Operations
jijiki created T235755: Increased POST latency for MW app servers (Oct 2019).
Oct 17 2019, 12:04 PM · Performance-Team (Radar), serviceops, Operations
jijiki created T235736: cp3032 and cp3040 occasional failed fetches.
Oct 17 2019, 9:18 AM · Operations, Traffic

Oct 16 2019

Physikerwelt awarded T195847: Clean up artifacts from LaTeX based math rendering a Like token.
Oct 16 2019, 3:10 PM · Operations, Math
jijiki updated the task description for T229792: Remove HHVM from production.
Oct 16 2019, 3:01 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki closed T195847: Clean up artifacts from LaTeX based math rendering as Resolved.

mw* servers will be reimaged as part of T229792, this is resolved

Oct 16 2019, 3:00 PM · Operations, Math
jijiki updated the task description for T195847: Clean up artifacts from LaTeX based math rendering.
Oct 16 2019, 2:59 PM · Operations, Math
jijiki updated the task description for T195847: Clean up artifacts from LaTeX based math rendering.
Oct 16 2019, 2:59 PM · Operations, Math
jijiki added a comment to T195847: Clean up artifacts from LaTeX based math rendering.

Since we will be reimaging all mw* servers, those packages will need to be removed actually from snapshot*, deploy*, mwmaint* and labweb*

Oct 16 2019, 12:43 PM · Operations, Math