Page MenuHomePhabricator

CDanis (Chris Danis)
SRE

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Nov 5 2018, 2:54 PM (45 w, 6 d)
Availability
Busy Busy until Sep 23.
IRC Nick
cdanis
LDAP User
CDanis
MediaWiki User
CDanis (WMF) [ Global Accounts ]

Recent Activity

Wed, Sep 11

CDanis added a comment to T232412: HTTPS-blog on blog.wikimedia.org is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:SSL connect attempt failed.

I’m on vacation starting yesterday, so won’t be able to answer everything,
but: errors began intermittently appearing Saturday. Source IP is the A
record that icinga.wikimedia.org resolves to

Wed, Sep 11, 5:51 AM · Wikimedia-Blog
CDanis committed rOSCT7dc91237432a: dbctl: use explicit keyword arguments for the callback (authored by CDanis).
dbctl: use explicit keyword arguments for the callback
Wed, Sep 11, 5:25 AM

Mon, Sep 9

CDanis closed T231108: upload LB: retry swift 404s cross-cluster, a subtask of T231086: Picture from Commons not found from Singapore, as Declined.
Mon, Sep 9, 11:20 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis closed T231108: upload LB: retry swift 404s cross-cluster as Declined.
Mon, Sep 9, 11:19 PM · Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T231108: upload LB: retry swift 404s cross-cluster.

Yeah, all fair points. We don't seem to be experiencing too many of these
404s (a handful per day), and other mitigations are available, so it's
probably not worth it.

Mon, Sep 9, 11:19 PM · Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis updated subscribers of T231108: upload LB: retry swift 404s cross-cluster.

@BBlack @ema can you weigh in at some point soon with how feasible this seems? I'm pretty unfamiliar with the current setup here.

Mon, Sep 9, 11:10 PM · Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis updated subscribers of T232412: HTTPS-blog on blog.wikimedia.org is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:SSL connect attempt failed.

@Varnent Can you pass on this report to Auttomatic?

Mon, Sep 9, 11:08 PM · Wikimedia-Blog
CDanis added a comment to T231086: Picture from Commons not found from Singapore.

On each Swift frontend host, I:

  • grepped today's logs for GETs that resulted in 404s against wikipedia-commons-local-public buckets (originals, not thumbnails).
  • took a random sample of 5000 of those (today there were about 50k-70k 404s per frontend, so total sample is a bit shy of 10%)
  • attempted to fetch the same files from the alternate datacenter.
Mon, Sep 9, 10:43 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis reassigned T231110: bring swiftrepl back to life from CDanis to fgiunchedi.

Leaving some notes here before I'm gone for two weeks.

Mon, Sep 9, 10:30 PM · Patch-For-Review, User-fgiunchedi, Commons, MediaWiki-File-management, media-storage, Operations
CDanis closed T232352: Remove Michal Anna Marble from security@ alias in exim, a subtask of T232348: Offboard Michal Anna from Security Team, as Resolved.
Mon, Sep 9, 7:20 PM · Security-Team
CDanis closed T232352: Remove Michal Anna Marble from security@ alias in exim as Resolved.
Mon, Sep 9, 7:20 PM · Security-Team, Operations
CDanis closed Restricted Task, a subtask of T232224: September 2019 DoS attacks [Public], as Resolved.
Mon, Sep 9, 2:02 PM · Wikimedia-Incident, Operations
CDanis created P9064 swiftrepl errors.
Mon, Sep 9, 1:37 PM
CDanis created P9063 ~/.config/fontconfig/fonts.config.
Mon, Sep 9, 1:14 PM

Sun, Sep 8

CDanis moved T229677: #dbctl: add 'comment'/'description' metadata to instances from in progress to awaiting release on the conftool board.
Sun, Sep 8, 3:16 AM · Patch-For-Review, DBA, conftool
CDanis added a comment to T229677: #dbctl: add 'comment'/'description' metadata to instances.

I wound up implementing a simpler UI for this, as the code was a lot less annoying than adding a --set-note flag to each subcommand.

Sun, Sep 8, 3:16 AM · Patch-For-Review, DBA, conftool
CDanis moved T231871: dbctl config commit !log message is identical whether the operation succeeded or failed from Backlog to awaiting release on the conftool board.
Sun, Sep 8, 3:14 AM · Patch-For-Review, conftool

Sat, Sep 7

CDanis closed Restricted Task, a subtask of T232224: September 2019 DoS attacks [Public], as Resolved.
Sat, Sep 7, 3:06 PM · Wikimedia-Incident, Operations
CDanis edited P9056 (An Untitled Masterwork).
Sat, Sep 7, 2:36 PM
CDanis edited P9056 (An Untitled Masterwork).
Sat, Sep 7, 2:36 PM
CDanis edited P9056 (An Untitled Masterwork).
Sat, Sep 7, 2:34 PM
CDanis edited P9056 (An Untitled Masterwork).
Sat, Sep 7, 2:16 PM
CDanis created P9056 (An Untitled Masterwork).
Sat, Sep 7, 1:55 PM
CDanis created P9055 (An Untitled Masterwork).
Sat, Sep 7, 3:53 AM
CDanis created P9054 (An Untitled Masterwork).
Sat, Sep 7, 3:51 AM
CDanis created P9053 (An Untitled Masterwork).
Sat, Sep 7, 3:43 AM

Fri, Sep 6

CDanis added a comment to T229677: #dbctl: add 'comment'/'description' metadata to instances.

I saw this was merged - thanks!
From that commit message there is still some work to be done, do you have a rough ETA on when we'll be able to start using those fields?
Thank you!

Fri, Sep 6, 12:26 PM · Patch-For-Review, DBA, conftool

Wed, Sep 4

CDanis committed rOSCT2b215fcfde7e: dbctl: schema: allow wikitech in readOnlyBySection (authored by CDanis).
dbctl: schema: allow wikitech in readOnlyBySection
Wed, Sep 4, 8:33 PM
CDanis closed T231642: Empty db-eqiad.php, db-codfw.php s1-s8+wikitech lines as Resolved.
Wed, Sep 4, 2:31 PM · conftool
CDanis moved T231642: Empty db-eqiad.php, db-codfw.php s1-s8+wikitech lines from Backlog to in progress on the conftool board.
Wed, Sep 4, 12:44 PM · conftool

Tue, Sep 3

CDanis updated the task description for T231871: dbctl config commit !log message is identical whether the operation succeeded or failed.
Tue, Sep 3, 1:12 PM · Patch-For-Review, conftool
CDanis created T231871: dbctl config commit !log message is identical whether the operation succeeded or failed.
Tue, Sep 3, 1:09 PM · Patch-For-Review, conftool

Mon, Sep 2

CDanis added a comment to T229686: #dbctl: manage 'externalLoads' data.

My thought with wanting to build templates into dbctl was mostly that it would mean dbctl's output would make it obvious what was happening, without needing to also look at the associated Mediawiki configs. Also I didn't want to add more PHP glue on the mediawiki-config side, because the status quo is already a bit gross.

Mon, Sep 2, 2:46 PM · Performance-Team (Radar), DBA, conftool
CDanis committed rOSCTa82295a81caf: dbctl: document config commit --batch (authored by CDanis).
dbctl: document config commit --batch
Mon, Sep 2, 1:00 PM
CDanis closed T231629: dbctl: support --background or --force? as Resolved.
Mon, Sep 2, 12:42 PM · conftool

Fri, Aug 30

CDanis updated subscribers of T229686: #dbctl: manage 'externalLoads' data.

Here's a mini-design proposal for the dbctl feature itself (@Volans and @Joe please review):

Fri, Aug 30, 8:23 PM · Performance-Team (Radar), DBA, conftool
CDanis closed T229631: noc.wm.o/db.php: remove hosts information, or fetch it from etcd somehow, a subtask of T197126: Create tool to handle the state of database configuration in MediaWiki in etcd, as Resolved.
Fri, Aug 30, 4:40 PM · Performance-Team (Radar), User-ArielGlenn, Patch-For-Review, User-Joe, MediaWiki-Configuration, Operations, DBA
CDanis closed T229631: noc.wm.o/db.php: remove hosts information, or fetch it from etcd somehow as Resolved.

The remaining work on this task is now part of T231642: Empty db-eqiad.php, db-codfw.php s1-s8+wikitech lines

Fri, Aug 30, 4:40 PM · Performance-Team (Radar), conftool, MediaWiki-Configuration, Operations
CDanis added a comment to T231620: https://en.wikipedia.org/wiki/Heteromyidae shows the mobile version on desktop.

Seems likely this is the same as T231504: Unexpectedly received mobile version of an article while logged out ?

Fri, Aug 30, 4:21 PM · Traffic, Operations, MobileFrontend, Mobile
CDanis updated subscribers of T229686: #dbctl: manage 'externalLoads' data.

In db-eqiad/codfw.php we currently provide IP addresses as the values of the keys in externalLoads instead of using hostsByName to translate hostnames to IP, e.g.:

'externalLoads' => [
	# es2
	'cluster24' => [
		'10.64.32.184' => 0, # es1015, C2 11TB 128GB, master
		'10.64.0.6'    => 1, # es1011, A2 11TB 128GB
		'10.64.16.186' => 1, # es1013, B1 11TB 128GB
	],
]
Fri, Aug 30, 3:42 PM · Performance-Team (Radar), DBA, conftool
CDanis added a comment to T231629: dbctl: support --background or --force?.

There's a --batch flag, intended for use from scripts, that I think should work for this?

Fri, Aug 30, 3:14 PM · conftool
CDanis added a comment to T231642: Empty db-eqiad.php, db-codfw.php s1-s8+wikitech lines.

I can help with the removal.
My proposal would be to remove it on codfw.php first, and navigate the site from mwdebug on codfw to check that everything works fine. Who knows what else uses sectionLoads :)

Fri, Aug 30, 2:18 PM · conftool
CDanis added a comment to T231642: Empty db-eqiad.php, db-codfw.php s1-s8+wikitech lines.

There's no blocker here I know of aside from needing to be careful about it -- I haven't found the time yet to do this.

Fri, Aug 30, 1:52 PM · conftool

Wed, Aug 28

CDanis added a comment to T231086: Picture from Commons not found from Singapore.

Fortunately this occurrence seems to be quite rare.

Wed, Aug 28, 9:56 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T231014: ServiceContainer.php: Circular dependency when creating MobileFrontend service "AMC.UserMode > AMC.Manager > FeaturesManager > UserModes > AMC.UserMode".

Seeing this again in production on enwiki as of about 15:15 UTC today https://logstash.wikimedia.org/goto/971e0dc9c8f3d9cfe4d76c30a6446a9b

Wed, Aug 28, 4:33 PM · MW-1.34-notes (1.34.0-wmf.19; 2019-08-20), Readers-Web-Backlog (Readers-Web-Kanbanana-2019-20-Q1), MobileFrontend, Wikimedia-production-error

Tue, Aug 27

CDanis committed rOSCT15f6a5e2f3ae: dbctl: initial support for hostsByName (authored by CDanis).
dbctl: initial support for hostsByName
Tue, Aug 27, 4:20 PM
CDanis committed rOSCTfe2adaeb2826: dbctl: always validate vs JSON schema (authored by CDanis).
dbctl: always validate vs JSON schema
Tue, Aug 27, 3:32 PM

Mon, Aug 26

CDanis added a comment to T231071: /w/api.php... ErrorException from line 0 of : PHP Notice: Unable to unserialize ... Size of serialized string ... exceeds max.

Per @Anomie's comment this isn't caused by any change in Mediawiki code, and thus IMO shouldn't be a train blocker?

Mon, Aug 26, 7:21 PM · Core Platform Team Workboards (Clinic Duty Team), MediaWiki-API, Wikimedia-production-error
CDanis edited P8974 Masterwork From Distant Lands.
Mon, Aug 26, 1:14 PM
CDanis added a comment to T231019: set min_replicas on database sections in dbctl.

That's right -- this just enforces total number of replicas pooled at the top level of a section.

Mon, Aug 26, 12:25 PM · conftool, DBA
CDanis created T231195: Please add operations/software/conftool repo to 'Wikimedia operations' section.
Mon, Aug 26, 12:23 PM · User-Ladsgroup, conftool, VPS-project-codesearch

Sat, Aug 24

CDanis added a comment to T231086: Picture from Commons not found from Singapore.

@Aklapper I haven't investigated in any depth, but T231078 looks like a different issue. In that case, the file is not available from any datacenter:

Sat, Aug 24, 7:42 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a project to T231119: Uploading a big PDF file failed: serviceops.

Thanks for this report!

Sat, Aug 24, 1:56 PM · User-Urbanecm, Internet-Archive, Commons
CDanis closed T231121: 503: backend fetch failed as Resolved.

Almost certainly due to codfw network troubles at the time https://wikitech.wikimedia.org/wiki/Incident_documentation/20190823-network_codfw

Sat, Aug 24, 4:38 AM · User-DannyS712, Traffic, Operations

Aug 23 2019

CDanis renamed T231108: upload LB: retry swift 404s cross-cluster from upload LB: retry 404s cross-cluster to upload LB: retry swift 404s cross-cluster.
Aug 23 2019, 8:57 PM · Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T231086: Picture from Commons not found from Singapore.

@Der_Keks yes, that is the purpose of the aforementioned swiftrepl daemon.

Aug 23 2019, 7:44 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis created T231110: bring swiftrepl back to life.
Aug 23 2019, 6:26 PM · Patch-For-Review, User-fgiunchedi, Commons, MediaWiki-File-management, media-storage, Operations
CDanis created T231108: upload LB: retry swift 404s cross-cluster.
Aug 23 2019, 6:15 PM · Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis created T231107: Improve MediaWiki logging of errors uploading files to Swift.
Aug 23 2019, 6:01 PM · Performance-Team, Commons, MediaWiki-File-management, media-storage
CDanis added a comment to T231086: Picture from Commons not found from Singapore.

BTW for posterity, here's how I looked for logs:

Aug 23 2019, 5:09 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T231086: Picture from Commons not found from Singapore.

I did some digging in the swift logs around the time the file was uploaded; there's no record of swift in codfw ever receiving a PUT for this file.

Aug 23 2019, 5:03 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis updated subscribers of T231086: Picture from Commons not found from Singapore.

Isn't there a swiftrepl background process to fix this?

Aug 23 2019, 3:03 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T204245: Run MediaWiki media originals active/active.

This confirms that all those 3 types are configured to write to both DCs when objects are created (even if for the purpose of this task only Score mattered). The replication is synchronous, meaning that if pushing a new object to the secondary DC fails, it should result in an error for the upload. And the web request will wait until the object is pushed to all DCs before completing.

Aug 23 2019, 2:56 PM · Patch-For-Review, Services (watching), Core Platform Team Legacy (Watching / External), Performance-Team (Radar), User-fgiunchedi, Availability (MediaWiki-MultiDC), Epic
CDanis added a comment to T231086: Picture from Commons not found from Singapore.

Indeed, the object exists on eqiad, but never made it to codfw swift:

Aug 23 2019, 2:42 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis edited projects for T231086: Picture from Commons not found from Singapore, added: media-storage; removed ops-eqsin.

Looks like we are indeed serving 404s for this object from codfw and all points west:

Aug 23 2019, 2:29 PM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
CDanis added a comment to T231071: /w/api.php... ErrorException from line 0 of : PHP Notice: Unable to unserialize ... Size of serialized string ... exceeds max.

Judging from logstash, this seems to only occur on HHVM. Likely, PHP7 is writing some things to memcache that HHVM can't unserialize.

Aug 23 2019, 2:01 PM · Core Platform Team Workboards (Clinic Duty Team), MediaWiki-API, Wikimedia-production-error

Aug 22 2019

CDanis closed T231035: update master DC switch script for a post-dbctl world as Invalid.
17:44:35	<volans>	cdanis: that was the switcdc repo that become the basis to write spicerack, now lives as cookbooks in the cookbooks repo
17:44:44	<volans>	but
17:44:58	<volans>	it doesn't use anymore the readonlybysection IIRC and uses the global RO instead
Aug 22 2019, 5:47 PM · serviceops, conftool, Operations
CDanis created T231035: update master DC switch script for a post-dbctl world.
Aug 22 2019, 5:46 PM · serviceops, conftool, Operations
CDanis created P8961 md-tools.sh.
Aug 22 2019, 5:24 PM
CDanis created T231019: set min_replicas on database sections in dbctl.
Aug 22 2019, 3:24 PM · conftool, DBA
CDanis created T231018: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1.
Aug 22 2019, 3:16 PM · conftool, DBA
CDanis created T231016: expand list of those who have permissions to edit the #wikimedia-operations topic.
Aug 22 2019, 3:06 PM · Operations

Aug 20 2019

CDanis triaged T229631: noc.wm.o/db.php: remove hosts information, or fetch it from etcd somehow as Low priority.

https://noc.wikimedia.org/db.php will now stay up-to-date with dbctl changes.

Aug 20 2019, 7:03 PM · Performance-Team (Radar), conftool, MediaWiki-Configuration, Operations
CDanis triaged T230799: icinga-wm bot truncating long messages as Normal priority.
Aug 20 2019, 6:46 PM · observability, Icinga, Operations
CDanis added a comment to T230799: icinga-wm bot truncating long messages.

Ahhh, thanks! Those lines ending in GET explains the T; that makes it seem very likely that icinga-wm is splitting the lines internally, but has a different/wrong idea of the max line length than the IRC server does.

Aug 20 2019, 1:32 PM · observability, Icinga, Operations
CDanis created T230799: icinga-wm bot truncating long messages.
Aug 20 2019, 1:29 PM · observability, Icinga, Operations

Aug 19 2019

CDanis added a project to T230733: Expose pooled status of gdnsd and conftool managed services as metrics: User-CDanis.
Aug 19 2019, 8:32 PM · User-CDanis, Operations, observability
CDanis updated subscribers of T230733: Expose pooled status of gdnsd and conftool managed services as metrics.
Aug 19 2019, 8:32 PM · User-CDanis, Operations, observability
CDanis triaged T230638: Move old transparency report pages to historical URLs and setup redirect as Normal priority.
Aug 19 2019, 8:31 PM · Patch-For-Review, serviceops, Operations, WMF-Legal
CDanis triaged T230733: Expose pooled status of gdnsd and conftool managed services as metrics as Normal priority.
Aug 19 2019, 8:31 PM · User-CDanis, Operations, observability

Aug 16 2019

CDanis triaged T230633: automation: issue reminders for about-to-expire downtimes as Normal priority.
Aug 16 2019, 9:12 PM · observability, Operations
CDanis created T230633: automation: issue reminders for about-to-expire downtimes.
Aug 16 2019, 9:12 PM · observability, Operations
CDanis added a comment to T230435: Set up mailing list for Santali Wikipedia.

Oh, also, please note that list administrators are not automatically subscribed to the list -- subscribe yourselves if you want to receive posts.

Aug 16 2019, 5:02 PM · Operations, Wikimedia-Mailing-lists
CDanis closed T230435: Set up mailing list for Santali Wikipedia as Resolved.

List created! @Manik87 you should have received an email with your administrator password for the mailing list.

Aug 16 2019, 5:01 PM · Operations, Wikimedia-Mailing-lists
CDanis added a comment to T230435: Set up mailing list for Santali Wikipedia.

Hi Manik,

Aug 16 2019, 1:37 PM · Operations, Wikimedia-Mailing-lists
CDanis moved T230435: Set up mailing list for Santali Wikipedia from Backlog to List creation on the Wikimedia-Mailing-lists board.
Aug 16 2019, 1:03 PM · Operations, Wikimedia-Mailing-lists
CDanis triaged T226633: PDF renderer needs better CJK font as Normal priority.
Aug 16 2019, 1:03 PM · Operations, Patch-For-Review, Chinese-Sites, PDF-Rendering, Product-Infrastructure-Team-Backlog, Proton
CDanis triaged T226782: a1-eqiad pdu refresh (Date TBD) as Normal priority.
Aug 16 2019, 1:02 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T227133: a8-eqiad pdu refresh (Date TBA) as Normal priority.
Aug 16 2019, 1:02 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC) as Normal priority.
Aug 16 2019, 1:02 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T227143: a7-eqiad pdu refresh as Normal priority.
Aug 16 2019, 1:02 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T227536: b1-eqiad pdu refresh (Thursday 10/10 @11am UTC) as Normal priority.
Aug 16 2019, 1:02 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC) as Normal priority.
Aug 16 2019, 1:01 PM · DC-Ops, Operations, ops-eqiad
CDanis triaged T228433: server-cache did neither update on uploading nor with ?action=purge as Normal priority.
Aug 16 2019, 1:01 PM · Traffic, Operations, MediaWiki-File-management, Commons, Multimedia
CDanis triaged T230110: Stop forcing RUNNER=php for foreachwiki/foreachwikiindblist as Normal priority.
Aug 16 2019, 1:00 PM · Patch-For-Review, serviceops, MediaWiki-Maintenance-scripts, Operations
CDanis added a subtask for T195392: Switch cronjobs on maintenance hosts to PHP7: T230110: Stop forcing RUNNER=php for foreachwiki/foreachwikiindblist.
Aug 16 2019, 1:00 PM · MediaWiki-extensions-Mailgun, User-jijiki, cloud-services-team, Services (watching), serviceops, Core Platform Team Legacy (Watching / External), User-ArielGlenn, Operations
CDanis added a parent task for T230110: Stop forcing RUNNER=php for foreachwiki/foreachwikiindblist: T195392: Switch cronjobs on maintenance hosts to PHP7.
Aug 16 2019, 1:00 PM · Patch-For-Review, serviceops, MediaWiki-Maintenance-scripts, Operations
CDanis triaged T229998: decom cookbook: dry-run mode not working / PuppetDB and Debmonitor removals can fail as Normal priority.
Aug 16 2019, 12:59 PM · Operations
CDanis triaged T229950: Merge or delete grantswiki as Normal priority.
Aug 16 2019, 12:59 PM · Operations, Wiki-Setup (Delete / Redirect)
CDanis triaged T229916: Create a cassandra.service which subsumes casandra-{a,b,c} services using PartsOf=cassandra.service as Normal priority.
Aug 16 2019, 12:58 PM · Cassandra, Operations