Page MenuHomePhabricator

Joe (Giuseppe Lavagetto)
Spy

Projects (24)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 5:57 AM (249 w, 3 d)
Availability
Available
LDAP User
Giuseppe Lavagetto
MediaWiki User
GLavagetto (WMF) [ Global Accounts ]

Recent Activity

Wed, Jul 3

Joe added a comment to T207707: contint1001 store docker images on separate partition or disk.

@thcipriani why do we even need to save the images? We don't really care about locally-saved images, do we?

Wed, Jul 3, 5:22 AM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201907), serviceops, Operations, Continuous-Integration-Infrastructure

Tue, Jul 2

Joe moved T207707: contint1001 store docker images on separate partition or disk from Backlog to Next up on the serviceops board.
Tue, Jul 2, 10:16 AM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201907), serviceops, Operations, Continuous-Integration-Infrastructure
Joe added a comment to T187147: Port mediawiki/php/wmerrors to PHP7 and deploy.

Talking with @Krinkle I realized that the reason why I saw that ugly error message is because the endpoint doesn't initialize the skin at all. Actual fatals on real pages should include the skin.

Tue, Jul 2, 10:01 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a comment to T223391: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed.

@awight @jkroll should we rollout to all hosts?

Tue, Jul 2, 8:34 AM · WMDE-QWERTY-Sprint-2019-06-26, User-jijiki, WMDE-QWERTY-Sprint-2019-06-12, WMDE-QWERTY-Sprint-2019-05-29, serviceops, Operations, WMDE-QWERTY-Sprint-2019-05-15, WMDE-QWERTY-Team, wikidiff2
Joe added a comment to T216243: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null.

I think we can just let this be resolved by the (upcoming) transition to php7. See T195392 for details.

Tue, Jul 2, 6:41 AM · Performance-Team (Radar), Patch-For-Review, Operations, MediaWiki-Maintenance-scripts

Mon, Jul 1

Joe added a comment to T223336: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm..

Using a modified version of furl that now supports unix sockets, for segfaults I get:

$ sudo furl --server unix:///run/php/fpm-www.sock --script /w/fatal-error.php --docroot /srv/mediawiki/docroot/wikipedia.org/ 'http://en.wikipedia.org/w/fatal-error.php?password=<redacted>&action=segfault'
Fatal error: Uncaught exception 'Adoy\FastCGI\ForbiddenException' with message 'Not in white list. Check listen.allowed_clients.' in /home/oblivian/furl:585
Stack trace:
#0 /home/oblivian/furl(450): Adoy\FastCGI\Client->wait_for_response()
#1 /home/oblivian/furl(648): Adoy\FastCGI\Client->request()
#2 /home/oblivian/furl(695): Adoy\FastCGI\doFcgiRequest()
#3 {main}
Mon, Jul 1, 2:38 PM · Performance-Team (Radar), serviceops, User-jijiki, observability, Operations, PHP 7.2 support
Joe added a comment to T223336: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm..

As I explained in T187147#5295715, my understanding is that in case of a segfault php-fpm fails to properly respond in any ways, forcing apache to produce a 503 error given it's working in proxy mode.

Mon, Jul 1, 1:45 PM · Performance-Team (Radar), serviceops, User-jijiki, observability, Operations, PHP 7.2 support
Joe updated the task description for T187147: Port mediawiki/php/wmerrors to PHP7 and deploy.
Mon, Jul 1, 10:51 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a comment to T187147: Port mediawiki/php/wmerrors to PHP7 and deploy.

After installing wmerrors on the test servers, these are my results:

Mon, Jul 1, 10:47 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe moved T226965: conftool: upgrade fleet to use existing python3-conftool from Backlog to Doing on the serviceops board.
Mon, Jul 1, 10:33 AM · serviceops, Operations
Joe added a comment to T226965: conftool: upgrade fleet to use existing python3-conftool.

This is blocked until https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/491412 is merged and deployed. I'll take care of it during the week.

Mon, Jul 1, 10:32 AM · serviceops, Operations
Joe added a comment to T222041: cp3037 is currently unreachable.

Can someone start the decommission process? this host shows up in things like debdeploy runs or cumin runs and that's distracting.

Mon, Jul 1, 9:27 AM · ops-esams, Operations, Traffic
Joe added a comment to T226965: conftool: upgrade fleet to use existing python3-conftool.

What we need to do is:

Mon, Jul 1, 9:14 AM · serviceops, Operations
Joe claimed T226965: conftool: upgrade fleet to use existing python3-conftool.
Mon, Jul 1, 9:08 AM · serviceops, Operations

Fri, Jun 28

Joe added a comment to T223391: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed.

Yes :) It's by far the best option.

Fri, Jun 28, 1:24 PM · WMDE-QWERTY-Sprint-2019-06-26, User-jijiki, WMDE-QWERTY-Sprint-2019-06-12, WMDE-QWERTY-Sprint-2019-05-29, serviceops, Operations, WMDE-QWERTY-Sprint-2019-05-15, WMDE-QWERTY-Team, wikidiff2
Joe added a comment to T223391: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed.

@jijiki We're a bit confused because the beta cluster reports it's running HHVM and there's no beta feature to select PHP7. Can you advise us on how to access the PHP7 backend?

Fri, Jun 28, 1:20 PM · WMDE-QWERTY-Sprint-2019-06-26, User-jijiki, WMDE-QWERTY-Sprint-2019-06-12, WMDE-QWERTY-Sprint-2019-05-29, serviceops, Operations, WMDE-QWERTY-Sprint-2019-05-15, WMDE-QWERTY-Team, wikidiff2
Joe updated the task description for T226815: Gather metrics on request status codes, latencies from the MediaWiki appservers.
Fri, Jun 28, 10:36 AM · Patch-For-Review, Operations, observability, serviceops
Joe added a comment to T226815: Gather metrics on request status codes, latencies from the MediaWiki appservers.

One relatively easy way to go could be to use mtail, which we use for quite some other things too.

Fri, Jun 28, 10:36 AM · Patch-For-Review, Operations, observability, serviceops
Joe created T226815: Gather metrics on request status codes, latencies from the MediaWiki appservers.
Fri, Jun 28, 10:29 AM · Patch-For-Review, Operations, observability, serviceops
Joe added a project to T226555: Bootstrap initial Cassandra table properties configuration in Puppet: serviceops-radar.
Fri, Jun 28, 7:43 AM · serviceops-radar, Core Platform Team (Session Management Service (CDP2)), User-Clarakosi, User-Eevans
Joe claimed T223391: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed.

I did rollout the new version on the canary servers today.

Fri, Jun 28, 7:23 AM · WMDE-QWERTY-Sprint-2019-06-26, User-jijiki, WMDE-QWERTY-Sprint-2019-06-12, WMDE-QWERTY-Sprint-2019-05-29, serviceops, Operations, WMDE-QWERTY-Sprint-2019-05-15, WMDE-QWERTY-Team, wikidiff2
Joe moved T226675: Undeploy electron service from WMF production from Backlog to Next up on the serviceops board.
Fri, Jun 28, 7:03 AM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), serviceops
Joe triaged T226675: Undeploy electron service from WMF production as Normal priority.
Fri, Jun 28, 7:02 AM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), serviceops
Joe moved T220398: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline from Doing to Goal tasks on the serviceops board.
Fri, Jun 28, 7:02 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, Operations, serviceops
Joe moved T212801: TEC3:O3:O3.1:Q3 Goal - Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production CD Pipeline from Doing to Goal tasks on the serviceops board.
Fri, Jun 28, 7:02 AM · Release-Engineering-Team, Release-Engineering-Team-TODO, Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, serviceops
Joe moved T220405: TEC3:05:05.1:Q4 Services and the deployment pipeline are hosted on production-level infrastructure from Doing to Goal tasks on the serviceops board.
Fri, Jun 28, 7:02 AM · Operations, serviceops
Joe moved T220403: TEC3:Q4 Tracking task from Doing to Goal tasks on the serviceops board.
Fri, Jun 28, 7:02 AM · Operations, serviceops
Joe placed T197564: cronspam for slow queries in PageAssessments up for grabs.
Fri, Jun 28, 7:00 AM · serviceops, Operations

Thu, Jun 27

Joe moved T187147: Port mediawiki/php/wmerrors to PHP7 and deploy from Externally Blocked to Doing on the serviceops board.
Thu, Jun 27, 4:54 PM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a comment to T224857: Enhance MediaWiki deployments for support of php7.x.
  • Whenver a rolling restart is needed, that will happen only on the servers where it's needed.

If that is so, are we going to remove the service restart timers we have now?

Thu, Jun 27, 12:19 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe closed T224491: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) as Resolved.

The immediate problem seems to be resolved given we've not see corruptions since we stopped resetting the opcache status on live servers.

Thu, Jun 27, 12:17 PM · User-jijiki, serviceops, PHP 7.2 support, Performance-Team (Radar), Operations, Wikimedia-production-error
Joe added a comment to T226373: Swift object servers become briefly unresponsive on a regular basis.

Do we have metrics on the swift backends open connections / connections queues? without such information, I don't think we can understand what the problem is, what is causing it, and how to mitigate it.

Thu, Jun 27, 12:00 PM · User-jijiki, serviceops, Patch-For-Review, media-storage, Operations
Joe added a subtask for T176370: Migrate to PHP 7 in WMF production: T224857: Enhance MediaWiki deployments for support of php7.x.
Thu, Jun 27, 9:47 AM · Core Platform Team Workboards (Team 1), Core Platform Team (PHP7 (TEC4)), Patch-For-Review, TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, Operations
Joe added a parent task for T224857: Enhance MediaWiki deployments for support of php7.x: T176370: Migrate to PHP 7 in WMF production.
Thu, Jun 27, 9:47 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe added a comment to T224857: Enhance MediaWiki deployments for support of php7.x.

I had a meeting with @thcipriani yesterday and we came up with the following contingency plan:

Thu, Jun 27, 9:44 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops

Wed, Jun 26

Joe closed T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs as Resolved.

Using curl I can confirm the header is now added. I fear you might need to force-reload in your browser as I see apache sees the date of modification of the file as last Thursday, so you might get a 304 if your browser has the url cached.

Wed, Jun 26, 10:19 AM · serviceops, Patch-For-Review, DNS, Traffic, Operations, Wikimedia-Apache-configuration, Matrix
Joe closed T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs, a subtask of T215042: Set up a hosted Matrix.org / Riot instance on modular.im, as Resolved.
Wed, Jun 26, 10:18 AM · Matrix, User-Tgr
Joe moved T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs from Backlog to Doing on the serviceops board.
Wed, Jun 26, 10:13 AM · serviceops, Patch-For-Review, DNS, Traffic, Operations, Wikimedia-Apache-configuration, Matrix
Joe claimed T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs.
Wed, Jun 26, 10:12 AM · serviceops, Patch-For-Review, DNS, Traffic, Operations, Wikimedia-Apache-configuration, Matrix

Mon, Jun 24

Joe edited projects for T214431: Pipeline: provide a way to rebuild all blubber images, added: serviceops-radar; removed serviceops.
Mon, Jun 24, 3:36 PM · Release-Engineering-Team (Pipeline), serviceops-radar, Release-Engineering-Team-TODO, Release Pipeline
Joe triaged T216049: add CI job into operations/deployments-charts repo that helm lint packages and perform the helm index after merge. as Low priority.
Mon, Jun 24, 3:35 PM · Continuous-Integration-Config, Continuous-Integration-Infrastructure, User-fsero, Kubernetes, serviceops
Joe triaged T216234: Clarify and document our docker image building process and policies. as Normal priority.
Mon, Jun 24, 3:33 PM · MediaWiki-Docker, serviceops
Joe moved T216234: Clarify and document our docker image building process and policies. from Backlog to Next up on the serviceops board.
Mon, Jun 24, 3:33 PM · MediaWiki-Docker, serviceops
Joe added a comment to T224857: Enhance MediaWiki deployments for support of php7.x.

Should we be working to implement symlink swapping in scap? Sounds like it's currently a blocker for this task (although opcache exhaustion may be blocking both).
If so the key is that we need the realpath for each deployment needs to be unique, correct? Atomic deployments are a side effect of the unique real path?

Mon, Jun 24, 9:01 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops
Joe closed T208524: RfC: Standards for external services in the Wikimedia infrastructure. as Resolved.
Mon, Jun 24, 8:56 AM · TechCom-RFC (TechCom-Approved), serviceops
Joe moved T187147: Port mediawiki/php/wmerrors to PHP7 and deploy from Doing to Externally Blocked on the serviceops board.
Mon, Jun 24, 6:33 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a comment to T210704: Migrate node-based services in production to node10.

@KartikMistry if we trigger a rebuild of the production container, it should now use the newer nodejs10-slim image and work as expected. Can you please confirm?

Mon, Jun 24, 6:32 AM · serviceops, Core Platform Team Backlog (Later), Services (next), Operations
Joe closed T226346: Update nodejs10 image to use the latest version of the package as Resolved.
Mon, Jun 24, 6:31 AM · Core Platform Team Backlog (Watching / External), Services (watching), serviceops, Operations
Joe closed T226346: Update nodejs10 image to use the latest version of the package, a subtask of T210704: Migrate node-based services in production to node10, as Resolved.
Mon, Jun 24, 6:31 AM · serviceops, Core Platform Team Backlog (Later), Services (next), Operations
Joe moved T218342: Our docker base images lack tags from Backlog to Doing on the serviceops board.
Mon, Jun 24, 6:10 AM · Release-Engineering-Team, Release-Engineering-Team-TODO, serviceops
Joe closed T218342: Our docker base images lack tags as Resolved.

This is already fixed, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/501564

Mon, Jun 24, 6:09 AM · Release-Engineering-Team, Release-Engineering-Team-TODO, serviceops
Joe claimed T226346: Update nodejs10 image to use the latest version of the package.

While I'm at it, I'll upgrade all production images with new base images too.

Mon, Jun 24, 6:04 AM · Core Platform Team Backlog (Watching / External), Services (watching), serviceops, Operations
Joe triaged T226346: Update nodejs10 image to use the latest version of the package as High priority.
Mon, Jun 24, 6:04 AM · Core Platform Team Backlog (Watching / External), Services (watching), serviceops, Operations
Joe moved T226346: Update nodejs10 image to use the latest version of the package from Backlog to Doing on the serviceops board.
Mon, Jun 24, 6:04 AM · Core Platform Team Backlog (Watching / External), Services (watching), serviceops, Operations
Joe created T226346: Update nodejs10 image to use the latest version of the package.
Mon, Jun 24, 6:02 AM · Core Platform Team Backlog (Watching / External), Services (watching), serviceops, Operations

Fri, Jun 21

Joe added a comment to T187147: Port mediawiki/php/wmerrors to PHP7 and deploy.

@Legoktm kindly did my job and created an upstream package for wmerrors. Thanks a lot!

Fri, Jun 21, 10:25 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe moved T187147: Port mediawiki/php/wmerrors to PHP7 and deploy from Backlog to Doing on the serviceops board.
Fri, Jun 21, 10:20 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a project to T187147: Port mediawiki/php/wmerrors to PHP7 and deploy: serviceops.
Fri, Jun 21, 10:19 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a subtask for T187147: Port mediawiki/php/wmerrors to PHP7 and deploy: T223336: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm..
Fri, Jun 21, 10:17 AM · Core Platform Team Workboards (Team 1), serviceops, Patch-For-Review, MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), wmerrors, Wikimedia-Logstash, MediaWiki-Logging, Operations, User-herron, PHP 7.2 support, Core Platform Team (PHP7 (TEC4)), Performance-Team (Radar)
Joe added a parent task for T223336: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm.: T187147: Port mediawiki/php/wmerrors to PHP7 and deploy.
Fri, Jun 21, 10:17 AM · Performance-Team (Radar), serviceops, User-jijiki, observability, Operations, PHP 7.2 support
Joe triaged T212934: etcd1004-1006 is unused and idle, use the cluster or kill it. as High priority.
Fri, Jun 21, 10:15 AM · decommission, Prod-Kubernetes, Kubernetes, serviceops
Joe edited projects for T215864: Travis tests for mediawiki-libs-etcd broken, added: Release-Engineering-Team, serviceops-radar; removed serviceops.
Fri, Jun 21, 10:14 AM · Release-Engineering-Team-TODO, serviceops-radar, Release-Engineering-Team, Librarization
Joe triaged T217764: Find which machines will be over 5 years old during FY19-20 as Low priority.
Fri, Jun 21, 10:13 AM · serviceops
Joe raised the priority of T217764: Find which machines will be over 5 years old during FY19-20 from Low to Needs Triage.
Fri, Jun 21, 10:13 AM · serviceops
Joe triaged T217764: Find which machines will be over 5 years old during FY19-20 as Low priority.
Fri, Jun 21, 10:11 AM · serviceops
Joe triaged T215662: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it as Low priority.
Fri, Jun 21, 10:08 AM · cloud-services-team (Kanban), Technical-Debt, Patch-For-Review, serviceops, Puppet, Cloud-VPS
Joe edited projects for T220775: Consider creating a puppet-compiler equivalent for mediawiki-config.git, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 10:07 AM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, serviceops-radar, Continuous-Integration-Config, Wikimedia-Site-requests
Joe edited projects for T220836: Guidelines for Rust/Go tools deployment, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 10:05 AM · serviceops-radar, Packaging
Joe added a comment to T220836: Guidelines for Rust/Go tools deployment.

While we wanted to get into a discussion about go packaging at the offsite, we had no time for it.

Fri, Jun 21, 10:05 AM · serviceops-radar, Packaging
Joe triaged T220836: Guidelines for Rust/Go tools deployment as Low priority.
Fri, Jun 21, 10:01 AM · serviceops-radar, Packaging
Joe triaged T218342: Our docker base images lack tags as Normal priority.
Fri, Jun 21, 9:59 AM · Release-Engineering-Team, Release-Engineering-Team-TODO, serviceops
Joe moved T219127: SRE FY2019 Q4 goal: complete the transition to PHP7 from Next up to Backlog on the serviceops board.
Fri, Jun 21, 9:37 AM · Operations, serviceops
Joe moved T217924: Make canary wait time configurable from Backlog to Externally Blocked on the serviceops board.
Fri, Jun 21, 9:10 AM · Patch-For-Review, Scap, serviceops
Joe moved T218328: Scap2 to use etcd for target servers from Backlog to Externally Blocked on the serviceops board.
Fri, Jun 21, 9:10 AM · serviceops, Scap
Joe moved T218412: Define a mediawiki "version" from Backlog to Externally Blocked on the serviceops board.
Fri, Jun 21, 9:10 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops, Scap
Joe added a project to T218412: Define a mediawiki "version": Release-Engineering-Team.
Fri, Jun 21, 9:09 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops, Scap
Joe moved T219748: Rebuild integration/config images based on jessie from Backlog to Externally Blocked on the serviceops board.
Fri, Jun 21, 9:08 AM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Patch-For-Review, serviceops
Joe changed the status of T219748: Rebuild integration/config images based on jessie from Open to Stalled.

The situation is, release engineering doesn't feel confident doing general upgrades like this, and are doing it on a case-by-case basis. I'll let the ticket open as blocked on others and abandon my patch

Fri, Jun 21, 9:08 AM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Patch-For-Review, serviceops
Joe changed the status of T219748: Rebuild integration/config images based on jessie, a subtask of T219580: Remove backports from wikimedia-jessie, from Open to Stalled.
Fri, Jun 21, 9:08 AM · Patch-For-Review, docker-pkg, serviceops, Release Pipeline
Joe edited Description on serviceops.
Fri, Jun 21, 8:54 AM
Joe moved T209881: Introduce state to Scap from Backlog to Watched on the serviceops-radar board.
Fri, Jun 21, 8:53 AM · serviceops-radar, Scap, Operations, User-jijiki
Joe moved T212129: Use a multi-dc aware store for ObjectCache's MainStash if needed. from Backlog to Watched on the serviceops-radar board.
Fri, Jun 21, 8:52 AM · MediaWiki-General-or-Unknown, Core Platform Team (Mainstash Multi-DC), serviceops-radar, User-mobrovac, User-jijiki, Performance-Team (Radar), Operations
Joe moved T213371: Document and possibly fine-tune how Proton interacts with Varnish from Backlog to Watched on the serviceops-radar board.
Fri, Jun 21, 8:52 AM · serviceops-radar, Readers-Web-Backlog (Tracking), Services (watching), Traffic, Reading-Infrastructure-Team-Backlog, Proton, Operations
Joe moved T218217: Make services swagger specs standard compliant from Backlog to Watched on the serviceops-radar board.
Fri, Jun 21, 8:52 AM · serviceops-radar, Core Platform Team (Security, stability, performance and scalability (TEC1)), Reading-Infrastructure-Team-Backlog, Proton, Graphoid, CX-cxserver, Citoid, Mathoid, Recommendation-API, Services (later), Mobile-Content-Service, RESTBase-API, Operations
Joe moved T218812: Provide the ability to have time-delayed or time-offset jobs in the job queue from Backlog to Watched on the serviceops-radar board.
Fri, Jun 21, 8:52 AM · serviceops-radar, TechCom-RFC, Core Platform Team Backlog (Designing), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics, ChangeProp, EventBus, WMF-JobQueue, Community-Tech
Joe edited projects for T213371: Document and possibly fine-tune how Proton interacts with Varnish, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:52 AM · serviceops-radar, Readers-Web-Backlog (Tracking), Services (watching), Traffic, Reading-Infrastructure-Team-Backlog, Proton, Operations
Joe edited projects for T213345: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice?, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:51 AM · serviceops-radar, TechCom-RFC, User-mobrovac, Core Platform Team Backlog (Designing), Services (designing), MediaWiki-Language-converter, Parsoid, Language-Team, Parsing-Team
Joe edited projects for T213566: Transferring data from Hadoop to production MySQL database, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:50 AM · Research-Backlog, serviceops-radar, Core Platform Team Backlog (Watching / External), Services (watching), User-Marostegui, Operations, Article-Recommendation, Analytics
Joe edited projects for T218217: Make services swagger specs standard compliant, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:49 AM · serviceops-radar, Core Platform Team (Security, stability, performance and scalability (TEC1)), Reading-Infrastructure-Team-Backlog, Proton, Graphoid, CX-cxserver, Citoid, Mathoid, Recommendation-API, Services (later), Mobile-Content-Service, RESTBase-API, Operations
Joe moved T211139: Convert Gerrit to use H2 as the database from Incoming to Externally Blocked on the serviceops board.
Fri, Jun 21, 8:49 AM · serviceops, Patch-For-Review, Operations, Gerrit
Joe edited projects for T209881: Introduce state to Scap, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:47 AM · serviceops-radar, Scap, Operations, User-jijiki
Joe edited projects for T212129: Use a multi-dc aware store for ObjectCache's MainStash if needed., added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:47 AM · MediaWiki-General-or-Unknown, Core Platform Team (Mainstash Multi-DC), serviceops-radar, User-mobrovac, User-jijiki, Performance-Team (Radar), Operations
Joe moved T210411: Applayer services without TLS from Incoming to Backlog on the serviceops board.
Fri, Jun 21, 8:47 AM · serviceops, Operations, Traffic
Joe moved T146055: Improve privilege separation for phabricator's config files and mysql credentials from Incoming to Backlog on the serviceops board.
Fri, Jun 21, 8:46 AM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, DBA, User-MModell, Security, Phabricator
Joe moved T213089: Upgrade memcached for Debian Stretch/Buster from Incoming to Backlog on the serviceops board.
Fri, Jun 21, 8:45 AM · User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
Joe moved T218342: Our docker base images lack tags from Incoming to Backlog on the serviceops board.
Fri, Jun 21, 8:44 AM · Release-Engineering-Team, Release-Engineering-Team-TODO, serviceops
Joe edited projects for T218812: Provide the ability to have time-delayed or time-offset jobs in the job queue, added: serviceops-radar; removed serviceops.
Fri, Jun 21, 8:44 AM · serviceops-radar, TechCom-RFC, Core Platform Team Backlog (Designing), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics, ChangeProp, EventBus, WMF-JobQueue, Community-Tech
Joe set the color for serviceops-radar to Violet.
Fri, Jun 21, 8:41 AM
Joe moved T224491: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) from PHP7 migration: backlog to Doing on the serviceops board.
Fri, Jun 21, 8:01 AM · User-jijiki, serviceops, PHP 7.2 support, Performance-Team (Radar), Operations, Wikimedia-production-error
Joe moved T224857: Enhance MediaWiki deployments for support of php7.x from PHP7 migration: backlog to Doing on the serviceops board.
Fri, Jun 21, 7:57 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Patch-For-Review, User-jijiki, PHP 7.2 support, Scap, serviceops