Page MenuHomePhabricator

jijiki (effie mouzeli)
is an animal

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 14 2018, 10:50 AM (75 w, 6 h)
Availability
Available
IRC Nick
effie
LDAP User
Effie Mouzeli
MediaWiki User
EMouzeli (WMF) [ Global Accounts ]

Recent Activity

Today

jijiki updated subscribers of T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.

@Cmjohnson @Jclark-ctr please let us know when we can have those servers online

Tue, Jan 21, 12:51 PM · serviceops, ops-eqiad, Operations

Yesterday

jijiki added a comment to T241852: rack/setup/install new codfw mw systems.

@Papaul @Joe Do you think it is ok if we rack everything else, have the new servers in rows ABD up and running, then decom some servers in C and rack the new ones?

Mon, Jan 20, 12:24 PM · ops-codfw, serviceops, Operations

Sun, Jan 19

jijiki created T243149: Increased latency in CODFW API and APP monitoring urls (~07:20 UTC 19 Jan 2020).
Sun, Jan 19, 11:53 AM · Performance-Team, serviceops, Operations

Fri, Jan 17

jijiki closed T241222: Update Wikimedia production to PHP 7.2.26 as Resolved.
Fri, Jan 17, 1:13 PM · Release-Engineering-Team-TODO, Release-Engineering-Team, serviceops
jijiki added a comment to T236800: Ensure apcu incr/decr are atomic (Upgrade php-apcu).

@Krinkle I backported https://github.com/krakjoe/apcu/pull/384/commits/c8849b0894712d0ede63552fb99ca8dc4d3d884f and packaged it in php-apcu_5.1.17+4.0.11-1+0~20190217111312.9+stretch~1.gbp192528+wmf2. You can find the package in your home directory on deployment-mediawiki-07 and install it. If we are happy with it, we can proceed with rolling it out to the canaries and then production.

Fri, Jan 17, 12:47 PM · Performance-Team (Radar), MediaWiki-Cache, Core Platform Team, serviceops

Thu, Jan 16

jijiki added a comment to T242606: No mw canary servers in codfw.

Yeah, we need at least a total of 4 api and 4 app canary servers in codfw. In eqiad our canary app (5) and api (4) servers are in the same rack actually, we can spread them a bit when we install the new servers

Thu, Jan 16, 12:47 PM · Patch-For-Review, Operations, serviceops
jijiki updated the task description for T217924: Make canary wait time configurable .
Thu, Jan 16, 8:02 AM · Operations, Scap, serviceops

Wed, Jan 15

jijiki added a comment to T217924: Make canary wait time configurable .

@thcipriani as per our discussion, we can consider merging and testing first for syncing files and then on the train. How does that sound?

Wed, Jan 15, 6:25 PM · Operations, Scap, serviceops
jijiki edited projects for T217924: Make canary wait time configurable , added: Operations; removed Patch-For-Review.
Wed, Jan 15, 6:24 PM · Operations, Scap, serviceops

Tue, Jan 14

jijiki added a comment to T236800: Ensure apcu incr/decr are atomic (Upgrade php-apcu).

@Krinkle are there other fixes we'd like to backport apart from https://github.com/krakjoe/apcu/pull/384 ?

Tue, Jan 14, 1:09 PM · Performance-Team (Radar), MediaWiki-Cache, Core Platform Team, serviceops
jijiki added a comment to T241097: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs.

@Krinkle, I am fine to merge any time, as soon as you have tested it:)

Tue, Jan 14, 11:09 AM · serviceops, Performance-Team
jijiki awarded T242609: Move thumbor to the logging pipeline a Love token.
Tue, Jan 14, 10:33 AM · observability, Wikimedia-Logstash, User-fgiunchedi, Operations
jijiki added a comment to T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet.

@Jclark-ctr Can you provide a date that is convenient for you for racking these? Thank you!

Tue, Jan 14, 9:43 AM · serviceops, ops-eqiad, Operations

Mon, Jan 13

jijiki added a comment to T238305: servers freeze across the caching cluster.

prometheus-trafficserver-tls-exporter.service initially failed to start on both cp3065 and cp3061 after reboot

Mon, Jan 13, 12:59 AM · Operations, Traffic

Fri, Jan 10

jijiki updated the task description for T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet.
Fri, Jan 10, 3:36 PM · serviceops, ops-eqiad, Operations

Thu, Jan 9

jijiki updated the task description for T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet.
Thu, Jan 9, 3:53 PM · serviceops, ops-eqiad, Operations
jijiki renamed T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet from (No Need By Date Provided) rack/setup/install kubernetes10[08-15].eqiad.wmnet to (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet.
Thu, Jan 9, 3:47 PM · serviceops, ops-eqiad, Operations
jijiki updated the task description for T241849: (No Need By Date Provided) rack/setup/install mw[1385-1413].eqiad.wmnet.
Thu, Jan 9, 3:16 PM · serviceops, ops-eqiad, Operations
jijiki renamed T241849: (No Need By Date Provided) rack/setup/install mw[1385-1413].eqiad.wmnet from (No Need By Date Provided) rack/setup/install new eqiad mw systems to (No Need By Date Provided) rack/setup/install mw[1385-1413].eqiad.wmnet.
Thu, Jan 9, 3:15 PM · serviceops, ops-eqiad, Operations
jijiki updated the task description for T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet.
Thu, Jan 9, 2:48 PM · serviceops, ops-eqiad, Operations
jijiki renamed T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet from (No Need By Date Provided) rack/setup/install new eqiad kubenetes systems to (No Need By Date Provided) rack/setup/install kubernetes10[08-15].eqiad.wmnet.
Thu, Jan 9, 2:46 PM · serviceops, ops-eqiad, Operations
jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

Hey @Jclark-ctr ! Do we have an update about when we will have those servers ready? Thank you!

Thu, Jan 9, 12:31 PM · serviceops, Operations, ops-eqiad
jijiki removed a subtask for T240684: Upgrade and improve our application object caching service (memcached): T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet.
Thu, Jan 9, 12:26 PM · Operations, serviceops
jijiki removed a parent task for T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet: T240684: Upgrade and improve our application object caching service (memcached).
Thu, Jan 9, 12:26 PM · ops-codfw, serviceops, Operations
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet.
Thu, Jan 9, 12:26 PM · Operations, serviceops
jijiki added a parent task for T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet: T240684: Upgrade and improve our application object caching service (memcached).
Thu, Jan 9, 12:26 PM · Operations, ops-codfw

Tue, Jan 7

jijiki closed T240824: PHP Fatal error: Allowed memory size of 524288000 bytes exhausted (tried to allocate 20480 bytes) in /var/www/php-monitoring/lib.php on line 35 as Resolved.

There is was bug in the monitoring script, I will reopen in case it misbehaves again.

Tue, Jan 7, 10:51 AM · serviceops, Operations

Mon, Jan 6

Dzahn awarded T229792: Remove HHVM from production a Barnstar token.
Mon, Jan 6, 9:02 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
Dzahn awarded T229792: Remove HHVM from production a Hungry Hippo token.
Mon, Jan 6, 8:58 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki triaged T240684: Upgrade and improve our application object caching service (memcached) as Medium priority.
Mon, Jan 6, 2:45 PM · Operations, serviceops
jijiki updated the task description for T240684: Upgrade and improve our application object caching service (memcached).
Mon, Jan 6, 10:32 AM · Operations, serviceops

Sun, Jan 5

jijiki closed T229792: Remove HHVM from production, a subtask of T219127: SRE FY19-20 Q1 goal: complete the transition to PHP7, as Resolved.
Sun, Jan 5, 8:45 PM · Operations, serviceops
jijiki closed T229792: Remove HHVM from production as Resolved.
Sun, Jan 5, 8:45 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki closed T229792: Remove HHVM from production, a subtask of T234384: Drop HHVM from CI, as Resolved.
Sun, Jan 5, 8:45 PM · HHVM, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201910)
jijiki renamed T240684: Upgrade and improve our application object caching service (memcached) from Plan to upgrade our application object caching service (memcached) to Upgrade and improve our application object caching service (memcached).
Sun, Jan 5, 8:34 PM · Operations, serviceops
jijiki closed T166233: Update redis puppet class to support stretch as Resolved.

Given we have installed redis servers in stretch, I think we can mark this as resolved

Sun, Jan 5, 8:26 PM · Cloud-Services, Operations, Puppet
jijiki added a parent task for T213089: Upgrade memcached for Debian Stretch/Buster: T240684: Upgrade and improve our application object caching service (memcached).
Sun, Jan 5, 8:18 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): T213089: Upgrade memcached for Debian Stretch/Buster.
Sun, Jan 5, 8:18 PM · Operations, serviceops
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): Unknown Object (Task).
Sun, Jan 5, 8:15 PM · Operations, serviceops
jijiki removed a subtask for T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen: Unknown Object (Task).
Sun, Jan 5, 7:58 PM · Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
jijiki added a subtask for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet: Unknown Object (Task).
Sun, Jan 5, 7:58 PM · serviceops, ops-eqiad, Operations
jijiki added a subtask for T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet: Unknown Object (Task).
Sun, Jan 5, 7:57 PM · ops-codfw, serviceops, Operations
jijiki removed a subtask for T240684: Upgrade and improve our application object caching service (memcached): Unknown Object (Task).
Sun, Jan 5, 7:57 PM · Operations, serviceops
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Sun, Jan 5, 7:52 PM · Operations, serviceops
jijiki edited parent tasks for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet, added: T240684: Upgrade and improve our application object caching service (memcached); removed: Unknown Object (Task).
Sun, Jan 5, 7:52 PM · serviceops, ops-eqiad, Operations
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet.
Sun, Jan 5, 7:52 PM · Operations, serviceops
jijiki edited parent tasks for T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet, added: T240684: Upgrade and improve our application object caching service (memcached); removed: Unknown Object (Task).
Sun, Jan 5, 7:52 PM · ops-codfw, serviceops, Operations
jijiki removed a subtask for T203786: Mcrouter periodically reports soft TKOs for mc1029 (was mc1035, mc1022) leading to MW Memcached exceptions: T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen.
Sun, Jan 5, 7:49 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error, User-Elukey, MediaWiki-Cache, Operations
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen.
Sun, Jan 5, 7:49 PM · Operations, serviceops
jijiki edited parent tasks for T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen, added: T240684: Upgrade and improve our application object caching service (memcached); removed: T203786: Mcrouter periodically reports soft TKOs for mc1029 (was mc1035, mc1022) leading to MW Memcached exceptions.
Sun, Jan 5, 7:49 PM · Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
jijiki removed a subtask for T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen: Unknown Object (Task).
Sun, Jan 5, 7:49 PM · Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
jijiki added a subtask for T240684: Upgrade and improve our application object caching service (memcached): Unknown Object (Task).
Sun, Jan 5, 7:49 PM · Operations, serviceops
jijiki updated the task description for T240684: Upgrade and improve our application object caching service (memcached).
Sun, Jan 5, 7:43 PM · Operations, serviceops
jijiki changed IRC Nick from jijiki to effie on jijiki.
Sun, Jan 5, 7:24 PM
jijiki renamed T241905: Investigate JobQueue outage from 2020-01-04 22:00 UTC from JobQueue stuck as of 2020-01-14 22:00 UTC to JobQueue stuck as of 2020-01-04 22:00 UTC.
Sun, Jan 5, 7:21 PM · Core Platform Team Workboards (Clinic Duty Team), Wikimedia-Incident, WMF-JobQueue

Dec 20 2019

jijiki awarded T241222: Update Wikimedia production to PHP 7.2.26 a Love token.
Dec 20 2019, 11:04 AM · Release-Engineering-Team-TODO, Release-Engineering-Team, serviceops
Dzahn awarded T239054: Reimage all mediawiki servers a Yellow Medal token.
Dec 20 2019, 2:41 AM · Operations, serviceops

Dec 19 2019

jijiki updated the task description for T229792: Remove HHVM from production.
Dec 19 2019, 7:12 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki closed T239054: Reimage all mediawiki servers , a subtask of T229792: Remove HHVM from production, as Resolved.
Dec 19 2019, 7:11 PM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki closed T239054: Reimage all mediawiki servers as Resolved.

FIN!

Dec 19 2019, 7:11 PM · Operations, serviceops
jijiki updated the task description for T239054: Reimage all mediawiki servers .
Dec 19 2019, 7:11 PM · Operations, serviceops

Dec 18 2019

jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

@Jclark-ctr if you feel this will work better, we are happy. Either way, this racking is still better than the original one (30 servers in D 5). Thank you!

Dec 18 2019, 10:59 AM · serviceops, Operations, ops-eqiad

Dec 16 2019

jijiki added a comment to T240824: PHP Fatal error: Allowed memory size of 524288000 bytes exhausted (tried to allocate 20480 bytes) in /var/www/php-monitoring/lib.php on line 35.

[06:58] <_joe_> !log clearing apcu across multiple api servers to allow metrics to be collected again (task coming soon)

Dec 16 2019, 8:02 AM · serviceops, Operations
jijiki added a comment to T240518: Some jobs are not being processed / are processed slowly.

It appears that processing backlog started increasing on 11 Dec between 21:15-21:30 UTC

Dec 16 2019, 4:44 AM · Wikimedia-Incident, SDC-Statements (Machine-vision-depicts), Machine vision, Product-Infrastructure-Team-Backlog, Structured-Data-Backlog, MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Core Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, Operations

Dec 13 2019

jijiki updated the task description for T240684: Upgrade and improve our application object caching service (memcached).
Dec 13 2019, 10:17 PM · Operations, serviceops
jijiki removed a project from T240684: Upgrade and improve our application object caching service (memcached): serviceops.
Dec 13 2019, 10:13 PM · Operations, serviceops
jijiki created T240684: Upgrade and improve our application object caching service (memcached).
Dec 13 2019, 3:17 PM · Operations, serviceops
jijiki placed T151304: tmpreaper possible race condition up for grabs.
Dec 13 2019, 12:40 PM · cloud-services-team, Operations

Dec 12 2019

jcrespo awarded T240205: High APCu fragmentation can impact server performance a Like token.
Dec 12 2019, 7:23 PM · Operations, serviceops
jijiki closed T240205: High APCu fragmentation can impact server performance as Resolved.

I think we can mark this as resolved, our solution seems to be working. If something breaks, we will open a new task or reopen this one. Thank you!

Dec 12 2019, 7:20 PM · Operations, serviceops
jijiki added a comment to T214734: All debug hosts give (likely spurious) message: PHP Fatal error: The UdpSocket to 127.0.0.1:10514 has been closed (from Monolog/SyslogUdp).

First of all, we found that this fatal error is present on mwdebug1001, not only mwdebug1002 like we have been assuming in this task.

Dec 12 2019, 4:39 PM · Patch-For-Review, Release-Engineering-Team, Performance-Team (Radar), serviceops, Wikimedia-production-error, User-fgiunchedi, Operations
jijiki updated the title for P9863 omkafka: action will suspended due to kafka error across fleet from untitled to omkafka: action will suspended due to kafka error across fleet.
Dec 12 2019, 11:39 AM
jijiki created T240560: rsyslogd: omkafka: action will suspended due to kafka error -187: Local: All broker connections are down.
Dec 12 2019, 11:38 AM · Operations, serviceops, observability

Dec 11 2019

jijiki updated the task description for T240419: Graphoid "No graph found." failures - 11 Dec 2019.
Dec 11 2019, 7:41 AM · serviceops, Operations, Graphoid
jijiki triaged T240419: Graphoid "No graph found." failures - 11 Dec 2019 as Unbreak Now! priority.
Dec 11 2019, 7:38 AM · serviceops, Operations, Graphoid
jijiki created T240419: Graphoid "No graph found." failures - 11 Dec 2019.
Dec 11 2019, 7:37 AM · serviceops, Operations, Graphoid

Dec 9 2019

jijiki claimed T240205: High APCu fragmentation can impact server performance .
Dec 9 2019, 12:56 PM · Operations, serviceops
jijiki created T240205: High APCu fragmentation can impact server performance .
Dec 9 2019, 12:47 PM · Operations, serviceops
jijiki added a comment to T240079: post-donation survey link broken .

@Pcoombe Yes, they had visited Canada about a month ago, that explains it:)

Dec 9 2019, 8:11 AM · Wikimedia-Fundraising

Dec 7 2019

jijiki updated the task description for T240079: post-donation survey link broken .
Dec 7 2019, 6:43 PM · Wikimedia-Fundraising
jijiki created T240079: post-donation survey link broken .
Dec 7 2019, 6:42 PM · Wikimedia-Fundraising

Dec 6 2019

jijiki added a comment to T73466: Use memcached by default.

@bd808 is this task still valid?

Dec 6 2019, 9:25 AM · MediaWiki-Vagrant
jijiki updated the task description for T229792: Remove HHVM from production.
Dec 6 2019, 9:08 AM · Patch-For-Review, MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Performance-Team (Radar), serviceops, Operations, HHVM
jijiki updated the task description for T239054: Reimage all mediawiki servers .
Dec 6 2019, 9:05 AM · Operations, serviceops

Dec 5 2019

jijiki added a comment to T239831: VP9-enabled ffmpeg doesn't get installed after reimage of mw job runner/video scaler.

@brion thank you! You can mark this as resolved if there is nothing else to be done

Dec 5 2019, 6:41 PM · serviceops, Operations
jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

That would be lovely, thank you!

Dec 5 2019, 5:43 PM · serviceops, Operations, ops-eqiad
jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

@wiki_willy given we are responsible for this delay, we would like to check with your team and tell us what is the earliest we can do, and we can in turn can plan around that. Sorry for the delays on our end. I would like to avoid requesting a date that is unrealistic, given that xmas is round the corner. Thank you!

Dec 5 2019, 5:29 PM · serviceops, Operations, ops-eqiad
jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

@Jclark-ctr Do we have an estimate when we are able to have those servers racked?

Dec 5 2019, 4:46 PM · serviceops, Operations, ops-eqiad
jijiki added a comment to T239831: VP9-enabled ffmpeg doesn't get installed after reimage of mw job runner/video scaler.

@brion It looks like all our videoscalers were lacking VP9 codec support. What do you think we should do ?

Dec 5 2019, 12:19 PM · serviceops, Operations

Dec 4 2019

jijiki added a comment to T236566: "hat-imagescalers" Cloud VPS project jessie deprecation.

@bd808 Thank you very much for the ping, I can deal with destroying the jessie instances next week, would that work?

Dec 4 2019, 11:11 PM · Cloud-VPS (Debian Jessie Deprecation)
jijiki added a comment to T239594: Unexpected general module "jquery.ui.core.styles" in styles queue..

Deployments happen once a week through the train. This is expected to roll out next week with 1.35.0-wmf.10, unless backported sooner which I proposed with the above cherry pick, but not sure it'll actually happen given other priorities.
Is this a volume risk currently? In terms of MW functionality the message is harmless and the fix didn't change behavior.

Dec 4 2019, 10:30 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Performance-Team, Wikimedia-production-error, Wikidata, MediaWiki-extensions-WikibaseRepository
jijiki claimed T239831: VP9-enabled ffmpeg doesn't get installed after reimage of mw job runner/video scaler.
Dec 4 2019, 4:35 PM · serviceops, Operations
jijiki added a comment to T239594: Unexpected general module "jquery.ui.core.styles" in styles queue..

@Krinkle I am afraid we are still seeing those errors in https://logstash.wikimedia.org/goto/dd9770ce7a81dc3c7614e03cf7b9de15

Dec 4 2019, 10:44 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Performance-Team, Wikimedia-production-error, Wikidata, MediaWiki-extensions-WikibaseRepository

Dec 2 2019

jijiki updated the task description for T239594: Unexpected general module "jquery.ui.core.styles" in styles queue..
Dec 2 2019, 4:00 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Performance-Team, Wikimedia-production-error, Wikidata, MediaWiki-extensions-WikibaseRepository
jijiki created T239607: MediaWiki::restInPeace: transaction round 'MediaWiki\Linter\RecordLintJob::run' still running.
Dec 2 2019, 2:12 PM · Core Platform Team Workboards (Clinic Duty Team), serviceops
jijiki updated the task description for T239054: Reimage all mediawiki servers .
Dec 2 2019, 12:55 PM · Operations, serviceops
jijiki created T239594: Unexpected general module "jquery.ui.core.styles" in styles queue..
Dec 2 2019, 11:28 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Performance-Team, Wikimedia-production-error, Wikidata, MediaWiki-extensions-WikibaseRepository
jijiki added a comment to T237605: Create kerberos principals for users.

@elukey Hey luca, I think I will need one too :) Thank you very much

Dec 2 2019, 10:32 AM · Patch-For-Review, Analytics-Kanban, Analytics

Nov 29 2019

jijiki updated the task description for T239054: Reimage all mediawiki servers .
Nov 29 2019, 4:14 PM · Operations, serviceops
jijiki added a comment to T239054: Reimage all mediawiki servers .

Completed auto-reimage of hosts:

['mw1308.eqiad.wmnet']

Of which those FAILED:

['mw1308.eqiad.wmnet']
Nov 29 2019, 1:44 PM · Operations, serviceops