Page MenuHomePhabricator

Joe (Giuseppe Lavagetto)
Spy

Projects (24)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 5:57 AM (258 w, 2 d)
Availability
Available
LDAP User
Giuseppe Lavagetto
MediaWiki User
GLavagetto (WMF) [ Global Accounts ]

Recent Activity

Fri, Sep 13

Joe added a comment to T232613: LBFactoryMulti.php: PHP Notice: Undefined index: .

We have a first core dump on mw1348 - I moved it under /root/T232613, if you need access ping me on IRC.

Fri, Sep 13, 8:27 AM · Patch-For-Review, MW-1.34-notes (1.34.0-wmf.22; 2019-09-10), Core Platform Team Workboards (Clinic Duty Team), Wikimedia-Rdbms, PHP 7.2 support, Wikimedia-production-error
Joe added a comment to T230570: De-noise systemd alerts (Reduce Icinga alert noise goal).

It's not true that "important services are monitored via dedicated service specific checks", quite the contrary on a lot of systems, I would rather improve the systemd alert instead of silencing it, and maybe be finally done with using those hacky checks for the number of running processes.

Fri, Sep 13, 6:28 AM · Patch-For-Review, Goal, observability

Thu, Sep 12

Joe added a comment to T232692: Should MediaWiki stop storing sessions on the server?.

Storing session state on server side (whether in memcached or whatever) instead of as an encrypted blob on the client has lots of upsides

  • Its much harder to screw up from a security perspective
  • You are not tossing around large blobs (I assume in this proposal everything in $_SESSION would be stored as an encrypted JWT token?)

I'm not really aware of any significant scalability concerns that would justify backing away from these upsides, but the ops aspect of session storage is not an area I follow too closely

Thu, Sep 12, 4:18 PM · MediaWiki-Authentication-and-authorization
Joe added a comment to T232692: Should MediaWiki stop storing sessions on the server?.

So what's the specific problem with regard to MediaWiki? Is this blocking something? Would it enable us to do something of particular value in our contexts here?

I think the problem (in my mind) is that MediaWiki is storing state on the server, that isn't necessary to store. In other words, it creates a scalability and reliability hurdle that, when removed, simplifies the infrastructure needed to run MediaWiki.

Thu, Sep 12, 4:12 PM · MediaWiki-Authentication-and-authorization
Joe added a comment to T232613: LBFactoryMulti.php: PHP Notice: Undefined index: .

mw1347 and mw1348 receive more traffic than the rest of the php api servers, so it makes sense this happens more frequently there.

Thu, Sep 12, 9:08 AM · Patch-For-Review, MW-1.34-notes (1.34.0-wmf.22; 2019-09-10), Core Platform Team Workboards (Clinic Duty Team), Wikimedia-Rdbms, PHP 7.2 support, Wikimedia-production-error
Joe added a comment to T231089: WikibaseClient.php: PHP Notice: Undefined index:.

Smells like T229433. Which is also about '' array index, and PHP 7.2. It's obviously a bug in PHP 7.2, but I've not been able to find evidence that it is due to opcache/T224491.

Thu, Sep 12, 9:00 AM · PHP 7.2 support, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikimedia-production-error
Joe added a comment to T231089: WikibaseClient.php: PHP Notice: Undefined index:.
Thu, Sep 12, 8:59 AM · PHP 7.2 support, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikimedia-production-error
Joe closed T232698: 503 errors when trying to log in to Wikimedia sites as Resolved.

Hi, we had some connectivity issues earlier. As soon as we were alerted and started checking, the issues recovered. We suspect the root cause to be a network maintenance ongoing at the time, but the problem is now resolved.

Thu, Sep 12, 6:00 AM · netops, Traffic, Operations

Wed, Sep 11

Daimona awarded T219150: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters a Love token.
Wed, Sep 11, 9:40 AM · Patch-For-Review, Performance-Team (Radar), User-jijiki, Operations, serviceops
Joe moved T226516: deploy CoreDNS as a in-cluster DNS service from Next up to Doing on the serviceops board.
Wed, Sep 11, 7:20 AM · serviceops
Joe moved T180696: Terminate Thumbor with SSL from Next up to Backlog on the serviceops board.
Wed, Sep 11, 7:20 AM · User-jijiki, serviceops, Performance-Team (Radar), Thumbor
Joe closed T224857: Enhance MediaWiki deployments for support of php7.x, a subtask of T176370: Migrate to PHP 7 in WMF production, as Resolved.
Wed, Sep 11, 7:17 AM · MW-1.34-notes (1.34.0-wmf.22; 2019-09-10), CPT Initiatives (PHP7 (TEC4)), Patch-For-Review, TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, Operations
Joe closed T224857: Enhance MediaWiki deployments for support of php7.x as Resolved.

This is now 99% done. We just need a confctl release to be able to make scap pull work as intended. Resolving this task though, as the work on the mw deployment side has been completed.

Wed, Sep 11, 7:16 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (Deployment services), Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe moved T224857: Enhance MediaWiki deployments for support of php7.x from Externally Blocked to Doing on the serviceops board.
Wed, Sep 11, 7:14 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (Deployment services), Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe moved T228965: set up limitranges and resourcequotas to protect the cluster from resource abuse and starvation from Doing to Backlog on the serviceops board.
Wed, Sep 11, 7:14 AM · User-fsero, serviceops, Prod-Kubernetes
Joe moved T228967: Set up PodSecurityPolicies in clusters from Doing to Backlog on the serviceops board.
Wed, Sep 11, 7:14 AM · Patch-For-Review, User-fsero, serviceops, Prod-Kubernetes
Joe closed T228837: recreate codfw cluster state from code stored in deployment-charts with helmfile [MIGHT CAUSE DOWNTIME], a subtask of T212123: Kubernetes clusters roadmap, as Resolved.
Wed, Sep 11, 7:13 AM · User-fsero, serviceops, Prod-Kubernetes
Joe closed T228837: recreate codfw cluster state from code stored in deployment-charts with helmfile [MIGHT CAUSE DOWNTIME] as Resolved.
Wed, Sep 11, 7:13 AM · User-fsero, serviceops, Prod-Kubernetes
Joe closed T232233: mw1317 issue: "DatabaseMysqli.php: Class undefined: stdClass" as Resolved.

I'm not sure about php7 - it was fixed when I got around fixing this machine.

Wed, Sep 11, 7:12 AM · Performance-Team (Radar), Wikimedia-Rdbms, serviceops
Joe added a comment to T232128: Make MultiHttpClient use CURLMOPT_MAX_HOST_CONNECTIONS and reuse connections.

While I support the use of this patch, the problem you're seeing should be greatly mitigated when we start using a middleware to manage service-to-service RPC. For now that's still in its infancy, but we already use that approach for cirrussearch, where requests are proxied via a local nginx on each appserver.

Wed, Sep 11, 6:31 AM · MediaWiki-libs-HTTP, Patch-For-Review, Performance-Team (Radar), Core Platform Team Workboards (Clinic Duty Team)
Joe added a comment to T232538: Make the parsoid server on the beta cluster a mediawiki app server.

hi @ssastry just a clarification: how would we load the parsoid code, if it can't be merged in the wmf vendor repository? Same way we do on scandium?

Wed, Sep 11, 6:26 AM · Patch-For-Review, Beta-Cluster-Infrastructure, Core Platform Team Workboards (Purple), RESTBase, Parsoid-PHP

Sat, Sep 7

Joe edited P9058 Various failing mtrs.
Sat, Sep 7, 2:39 PM
Joe created P9058 Various failing mtrs.
Sat, Sep 7, 2:37 PM
Joe created P9057 From Vodafone IT.
Sat, Sep 7, 2:00 PM

Fri, Sep 6

Joe committed rOSCTf5dbddac4e2c: Fix configuration file lookup when running with sudo (authored by Joe).
Fix configuration file lookup when running with sudo
Fri, Sep 6, 1:45 PM
Joe committed rOSCTcfc1388ec26f: kvobject: fix some class property ordering (authored by Joe).
kvobject: fix some class property ordering
Fri, Sep 6, 1:10 PM
Joe created P9048 Semantic Versioning.
Fri, Sep 6, 11:00 AM
Joe added a comment to T231027: Cassandra instances outages (was: Outage of restbase2017-b).
Fri, Sep 6, 9:09 AM · User-Eevans, Core Platform Team Workboards (Clinic Duty Team)
Joe created MediaWiki-extensions-WebToolsManager.
Fri, Sep 6, 5:41 AM

Thu, Sep 5

Joe added a comment to T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500.

So the real issue was:

  • termbox correctly uses the api-ro.discovery.wmnet host
  • the discovery record was incorrectly set to active-active
  • so requests from termbox would just go to the nearest dc, meaning that in codfw it would face super-cold caches after every release
  • as a consequence, some requests would time out because of the cold caches at all levels
Thu, Sep 5, 8:55 AM · Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), serviceops, Wikidata, Wikidata-Termbox, Release-Engineering-Team-TODO (201909), Release, Train Deployments
Joe updated subscribers of T224857: Enhance MediaWiki deployments for support of php7.x.

I did some tests, and we still have one problem with scap pull:

Thu, Sep 5, 8:26 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (Deployment services), Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe added a comment to T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500.

I think this is probably the same as T229313. We suspected it might be related to T231011; perhaps the new train makes this problem more pronounced

Thu, Sep 5, 7:40 AM · Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), serviceops, Wikidata, Wikidata-Termbox, Release-Engineering-Team-TODO (201909), Release, Train Deployments

Wed, Sep 4

Joe added a comment to T231192: mw2231 is down and unable to reboot.

@Papaul that looks fine - I don't think we need to swap out the SSDs, so just do it if we have a better use of those disks (they're pretty useless on an appserver).

Wed, Sep 4, 4:17 PM · ops-codfw, DC-Ops, Operations
Joe added a comment to T224857: Enhance MediaWiki deployments for support of php7.x.

@thcipriani should we create a new package/release?

Wed, Sep 4, 6:56 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (Deployment services), Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops

Tue, Sep 3

Joe added a comment to T231192: mw2231 is down and unable to reboot.

I second what @MoritzMuehlenhoff suggested. The system is not scheduled for replacement for another 2 years, so if we can salvage it somehow, that'd be great.

Tue, Sep 3, 4:47 AM · ops-codfw, DC-Ops, Operations

Mon, Sep 2

Joe added a comment to T229686: #dbctl: manage 'externalLoads' data.

A couple comments:

  • I concur with @Volans I'd keep the first iteration (at least) *very* simple
  • I know adding tags to a schema is a pain (in fact, it will need a data migration) but the flavour thing you were proposing seems like the kind of thing that should be a tag, so scope=eqiad,flavour=main could be a set of tags for an instance object
Mon, Sep 2, 7:07 AM · Performance-Team (Radar), DBA, conftool
Joe updated the task description for T227541: b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC).
Mon, Sep 2, 5:39 AM · DC-Ops, Operations, ops-eqiad
Joe updated the task description for T227541: b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC).
Mon, Sep 2, 5:38 AM · DC-Ops, Operations, ops-eqiad

Tue, Aug 27

Joe committed rOSCT9af0cc2c074b: Remove the service object for the default schema (authored by Joe).
Remove the service object for the default schema
Tue, Aug 27, 7:58 AM

Mon, Aug 26

Joe triaged T231200: CI performance issues as Unbreak Now! priority.

For context, the actual time to run the tests for operations/puppet is under one minute for most patches.

Mon, Aug 26, 1:46 PM · Patch-For-Review, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201908), Continuous-Integration-Infrastructure
Joe triaged T231192: mw2231 is down and unable to reboot as Normal priority.
Mon, Aug 26, 9:52 AM · ops-codfw, DC-Ops, Operations
Joe created T231192: mw2231 is down and unable to reboot.
Mon, Aug 26, 9:52 AM · ops-codfw, DC-Ops, Operations
Joe closed T231016: expand list of those who have permissions to edit the #wikimedia-operations topic as Resolved.
Mon, Aug 26, 9:25 AM · Operations
Joe added a comment to T231016: expand list of those who have permissions to edit the #wikimedia-operations topic.

I did some cleanup removing non-sres and adding a few people from the US TZ.

Mon, Aug 26, 9:25 AM · Operations
Joe added a comment to T231009: Make jobprocessor's test not depend on external files.

@Mathew.onipe can I ask further details on the error you get? It should definitely not be an issue if the test works in a docker image locally.

Mon, Aug 26, 8:24 AM · Release Pipeline, Operations, Maps (Kartotherian)
Joe triaged T231009: Make jobprocessor's test not depend on external files as Normal priority.
Mon, Aug 26, 8:19 AM · Release Pipeline, Operations, Maps (Kartotherian)
Joe added a comment to T216750: Article recommendation API: replace WDQS with MW API.

@leila I just stumbled upon this task, and besides being happy that patch was merged I'm asking myself:

Mon, Aug 26, 8:16 AM · Patch-For-Review, Google-Summer-of-Code (2019), Article-Recommendation, Research, Recommendation-API
Joe triaged T231086: Picture from Commons not found from Singapore as Normal priority.
Mon, Aug 26, 8:13 AM · User-fgiunchedi, Structured-Data-Backlog, Structured Data Engineering, Multimedia, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), Patch-For-Review, Commons, MediaWiki-File-management, media-storage, Traffic, Operations
Joe added a comment to T231119: Uploading a big PDF file failed.

A file of 473 MB surely goes over the large file limits unless something changed recently.
https://commons.wikimedia.org/wiki/Help:Server-side_upload still seems to suggest you should request that. Untagging Operations as I don't think there is anything SRE should do here.

All of upload_by_url, bigChunkedUpload and the UploadWizard are supposed to allow uploads up to 4 GB. If that's not the case, please say it clearly and fix the documentation. Otherwise, this is a bug.

Mon, Aug 26, 7:42 AM · User-Urbanecm, Commons, Internet-Archive
Joe removed projects from T231119: Uploading a big PDF file failed: serviceops, Operations.
Mon, Aug 26, 6:51 AM · User-Urbanecm, Commons, Internet-Archive
Joe added a comment to T231119: Uploading a big PDF file failed.

A file of 473 MB surely goes over the large file limits unless something changed recently.

Mon, Aug 26, 6:51 AM · User-Urbanecm, Commons, Internet-Archive
Joe updated subscribers of T229980: Need help to create and deploy Debian-packaged Python 3 app.

BTW I see the patch is still under review, and @Volans is on it.

Mon, Aug 26, 5:59 AM · serviceops, Operations, Packaging, CPT Initiatives (Session Management Service (CDP2))
Joe added a comment to T229980: Need help to create and deploy Debian-packaged Python 3 app.

Packaging is primarily handled by serviceops / Operations (meta: which should I tag here, SRE folks?)

Mon, Aug 26, 5:58 AM · serviceops, Operations, Packaging, CPT Initiatives (Session Management Service (CDP2))

Fri, Aug 23

Joe created P8970 Thank you puppet, you're the gift that keeps on giving..
Fri, Aug 23, 5:22 PM
Joe added a comment to T231011: Mysterious, coordinated slowdowns every ~ 25 minutes on mw1347,mw1348 (php7 api servers).

First smoking gun is in all the intervals I controlled the offender was parsoid-batch with quite large requests. I'm trying to gathering more cases to create a better statistics.

Fri, Aug 23, 1:35 PM · PHP 7.2 support, serviceops, Operations
Ladsgroup awarded T219150: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters a Love token.
Fri, Aug 23, 10:27 AM · Patch-For-Review, Performance-Team (Radar), User-jijiki, Operations, serviceops
Joe added a comment to T231063: Allow blocking requests from specific networks on the edge.

I think it's good to have a first, simple implementation, like the one above, but I think going further we would need a "block" object in puppet (or elsewhere, more on that below) that includes:

Fri, Aug 23, 8:17 AM · Operations, Traffic

Thu, Aug 22

Joe added a comment to T204056: Move wikimedia.ee under WM-EE.

@Dzahn Can you confirm how @tramm should configure the MX records? I think he'll need to add 18-19 from here to WMEE's elkdata DNS info, but want to make sure that's correct.
Is there anything else SRE needs to do before I change the nameserver with the registrar?

Thu, Aug 22, 4:35 PM · WMF-Legal, Patch-For-Review, Operations, Domains, Traffic
Joe added a comment to T229697: Investigate Kask request latency.

@WDoranWMF we will get to this as soon as our resources allow it.

Thu, Aug 22, 2:27 PM · User-Eevans, Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (Session Management Service (CDP2)), Performance-Team (Radar)
Joe triaged T231011: Mysterious, coordinated slowdowns every ~ 25 minutes on mw1347,mw1348 (php7 api servers) as High priority.
Thu, Aug 22, 2:20 PM · PHP 7.2 support, serviceops, Operations
Joe created T231011: Mysterious, coordinated slowdowns every ~ 25 minutes on mw1347,mw1348 (php7 api servers).
Thu, Aug 22, 2:20 PM · PHP 7.2 support, serviceops, Operations
Joe triaged T230861: PHP 7.2 is very slow on an allocation-intensive benchmark as Normal priority.
Thu, Aug 22, 10:12 AM · PHP 7.3 support, PHP 7.2 support, serviceops, Operations
Joe claimed T230917: celery-ores-worker service failed on ores100[2,4,5] without any apparent reason or significant log.
Thu, Aug 22, 10:11 AM · Scoring-platform-team (Current), ORES, Operations, serviceops
Joe added a comment to T230917: celery-ores-worker service failed on ores100[2,4,5] without any apparent reason or significant log.

I think this is a reasonable explanation, but how would you suggest we should fix our monitoring?

Thu, Aug 22, 10:11 AM · Scoring-platform-team (Current), ORES, Operations, serviceops
Joe triaged T230934: Elevated php7 latency during mw deploy as Normal priority.

it's indeed strange. In particular I find it strange that it affects mainly 400s and 404s. Maybe the Performance-Team might have an insight to why 4xx and 301s are so slow after a deploy of the train (where I assume a ton of caches, local and not, are invalidated all at once)

Thu, Aug 22, 10:08 AM · Operations, MediaWiki-General
Joe added a comment to T230951: Transfer ownership of mediawiki-security mailman list to Security Team.

@Reedy @JBennett I've set you up as list administrators. You now need to change the list admin password, I'm happy to help if you can't reset it yourself (I don't think you can). Just ping me on IRC so we can change the password in sync.

Thu, Aug 22, 9:47 AM · Wikimedia-Mailing-lists, Operations
Joe claimed T230951: Transfer ownership of mediawiki-security mailman list to Security Team.
Thu, Aug 22, 9:38 AM · Wikimedia-Mailing-lists, Operations
Joe triaged T230964: Netbox LibreNMS report fails as Normal priority.
Thu, Aug 22, 9:35 AM · netbox, Operations
Joe renamed T230969: Cannot access project workboards or log in after phabricator upgrade from Cannot access project workboards or log in: Failed to load class or interface "PhabricatorWorkboardViewState" to Cannot access project workboards or log in after phabricator upgrade.
Thu, Aug 22, 6:28 AM · Regression, Phabricator
Joe renamed T230969: Cannot access project workboards or log in after phabricator upgrade from Cannot access project workboards: Failed to load class or interface "PhabricatorWorkboardViewState" to Cannot access project workboards or login after phabricator upgrade..
Thu, Aug 22, 6:27 AM · Regression, Phabricator
Joe added a comment to T230969: Cannot access project workboards or log in after phabricator upgrade.

This seems to be the consequence of yesterday's upgrade of phabricator. SRE is trying to reach anyone in release engineering to try to help with the rollback.

Thu, Aug 22, 6:14 AM · Regression, Phabricator

Wed, Aug 21

Marostegui awarded T185084: Allow use of EtcdConfig to configure slave databases a Mountain of Wealth token.
Wed, Aug 21, 10:55 AM · DBA, discovery-system, MediaWiki-Configuration, Operations
Joe closed T185084: Allow use of EtcdConfig to configure slave databases as Resolved.

Indeed! we're doing more than this!

Wed, Aug 21, 10:55 AM · DBA, discovery-system, MediaWiki-Configuration, Operations
Joe closed T185084: Allow use of EtcdConfig to configure slave databases, a subtask of T182597: Use EtcdConfig in production to allow automation of a datacenter switch, as Resolved.
Wed, Aug 21, 10:55 AM · MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), discovery-system, MediaWiki-Configuration, Operations
Joe added projects to T230861: PHP 7.2 is very slow on an allocation-intensive benchmark: Operations, serviceops, PHP 7.2 support, PHP 7.3 support.
Wed, Aug 21, 7:24 AM · PHP 7.3 support, PHP 7.2 support, serviceops, Operations
Joe triaged T230774: Run jstack / jmap / etc... with PrivateTmp=true as Normal priority.
Wed, Aug 21, 6:12 AM · Discovery-Search (Current work), Operations

Tue, Aug 20

Joe added a comment to T230774: Run jstack / jmap / etc... with PrivateTmp=true.

You can automate the process with a simple alias:

Tue, Aug 20, 9:25 AM · Discovery-Search (Current work), Operations
Joe added a comment to T224033: Fix operations/puppet.git "rebase hell".

Here is my two lepta:

Tue, Aug 20, 8:32 AM · Release-Engineering-Team (Development services), Gerrit, Release-Engineering-Team-TODO, Continuous-Integration-Config, Operations

Mon, Aug 19

Joe added a comment to T230638: Move old transparency report pages to historical URLs and setup redirect.

Sorry, the indications you give here are in contrast with each other:

Mon, Aug 19, 7:12 AM · Patch-For-Review, serviceops, Operations, WMF-Legal

Aug 14 2019

Joe added a comment to T230076: Preprocessor_Hash: Allowed memory size exhausted.

T214984 seems somewhat related.

Aug 14 2019, 9:56 AM · CPT Initiatives (PHP7 (TEC4)), MediaWiki-Parser, Core Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error

Aug 7 2019

Joe added a comment to T229858: Pick a simple (short-term) deployment option for scandium.

My main worry is that anything you could do would be wiped out by the next scap run, unless we find a way to inject the code into mediawiki in a way that does avoid that.

Aug 7 2019, 7:54 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), serviceops, Parsoid-PHP
Joe added a comment to T227776: Generalize ParserCache into a generic service class for large "current" page-derived data.

I like the idea of having the ParserCache being a more generalized caching mechanism for MediaWiki. I have serious doubts about other things hinted here, specifically exposing a caching endpoint to other services. I'd argue that such a caching service should be separated from MediaWiki, have a simple API, and probably be structured around the page/revision identifier. We also probably don't want such a system to be written in PHP, as we would aim for the highest possible throughput.

Aug 7 2019, 7:51 AM · User-mobrovac, CPT Initiatives (Parsoid PHP (CDP2)), TechCom, User-Daniel, Proposal

Aug 2 2019

Joe added a project to T179901: Create a tmp directory just for MediaWiki: serviceops.
Aug 2 2019, 7:05 AM · Performance-Team (Radar), serviceops, Security, MediaWiki-General
Joe added a comment to T179901: Create a tmp directory just for MediaWiki.

I would rather do what @Anomie suggested, that is using PrivateTmp=true for php-fpm. I'll look into it.

Aug 2 2019, 7:05 AM · Performance-Team (Radar), serviceops, Security, MediaWiki-General
Joe closed T166081: rack/setup/install conf1004-conf1006, a subtask of T182924: Refresh zookeeper nodes in eqiad, as Resolved.
Aug 2 2019, 7:03 AM · User-Elukey, Analytics-Kanban
Joe closed T166081: rack/setup/install conf1004-conf1006 as Resolved.
Aug 2 2019, 7:03 AM · User-Joe, Operations

Aug 1 2019

Joe added a comment to T97972: Figure out a security model for etcd.
In T97972#5352851, @Joe wrote:

IIRC we already have an account specialized for accessing only mwconfig, we could expand on the concept.

Not in etcd, we only have a root user (see v2/auth/users) and root and guest roles (see v2/auth/roles). The guest role having access to eventlogging objects, but I don't see them, so maybe relic from the past of setup in anticipation of something that never happened.

Aug 1 2019, 11:20 AM · conftool, Patch-For-Review, Operations, services-tooling, discovery-system, Traffic
Joe closed T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile, a subtask of T228069: Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing, as Resolved.
Aug 1 2019, 10:11 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), serviceops, Parsoid-PHP
Joe closed T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile as Resolved.

I tested reimaging one application server and it went flawlessly and it's now running without any trace of HHVM. I'll resolve this ticket.

Aug 1 2019, 10:11 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
Joe updated the task description for T219150: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters.
Aug 1 2019, 6:50 AM · Patch-For-Review, Performance-Team (Radar), User-jijiki, Operations, serviceops
Joe updated the task description for T219150: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters.
Aug 1 2019, 6:49 AM · Patch-For-Review, Performance-Team (Radar), User-jijiki, Operations, serviceops
Joe updated the task description for T219150: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters.
Aug 1 2019, 6:49 AM · Patch-For-Review, Performance-Team (Radar), User-jijiki, Operations, serviceops

Jul 29 2019

Joe added a comment to T208524: RfC: Standards for external services in the Wikimedia infrastructure..

@Aklapper the RfC has been edited to reflect what's on phabricator at https://www.mediawiki.org/wiki/Requests_for_comment/Standards_for_external_services

Jul 29 2019, 8:11 AM · TechCom-RFC (TechCom-Approved), serviceops
Joe closed T91104: PHP thumbnailer as a service, a subtask of T43371: Thumbnail/imagescaler (tracking), as Declined.
Jul 29 2019, 6:01 AM · media-storage, Commons, Tracking-Neverending
Joe closed T91104: PHP thumbnailer as a service as Declined.

I think this was an idea before the thumbor project started, in order to have a thumbnail service that didn't need to have access to the databases, for instance.

Jul 29 2019, 6:01 AM · MediaWiki-File-management, Multimedia

Jul 25 2019

Joe triaged T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile as High priority.
Jul 25 2019, 1:45 PM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
Joe edited projects for T187147: Port mediawiki/php/wmerrors to PHP7 and deploy, added: serviceops-radar; removed serviceops.
Jul 25 2019, 1:44 PM · observability, Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (PHP7 (TEC4)), MediaWiki-Debug-Logger, serviceops-radar, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, Operations, User-herron, PHP 7.2 support, Performance-Team (Radar)
Joe moved T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile from Backlog to Doing on the serviceops board.
Jul 25 2019, 1:42 PM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
Joe added a project to T213494: Installing composer modules for deployment: Release-Engineering-Team.

Given parsoid/PHP is intended to be used as a library by a MediaWiki extension, I think it should be included in the code we release with scap, and the extension be activated only on scandium for the time being.

Jul 25 2019, 10:31 AM · Patch-For-Review, Release-Engineering-Team-TODO, Release-Engineering-Team (Deployment services), Parsoid-PHP
Joe created T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile.
Jul 25 2019, 10:27 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP