Page MenuHomePhabricator

dduvall (Dan Duvall)
Automation Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:24 PM (267 w, 22 h)
Availability
Available
IRC Nick
marxarelli
LDAP User
Dduvall
MediaWiki User
DDuvall (WMF) [ Global Accounts ]

Recent Activity

Wed, Nov 6

dduvall added a comment to T237450: Evaluate Airflow's suitability for CI.

Ah interesting, good to know.
Can Argo Events also produce external events, perhaps via Kafka? Something like that might make integration with other systems easier.

Wed, Nov 6, 4:47 PM · Release-Engineering-Team-TODO (201911), Continuous-Integration-Infrastructure

Tue, Nov 5

dduvall added a comment to T237450: Evaluate Airflow's suitability for CI.

For accuracy, IIUC Argo is a building block, from which Argo-CI is built.

Tue, Nov 5, 11:20 PM · Release-Engineering-Team-TODO (201911), Continuous-Integration-Infrastructure

Wed, Oct 23

dduvall added a comment to T236203: Add CI checks for golang admission controllers.

We do have a docker-pkg managed image for Go projects, docker-registry.wikimedia.org/golang:1.11.5-1. It's currently being utilized as a base image by Blubber, and it can likely be used here.

Wed, Oct 23, 5:45 PM · Release-Engineering-Team-TODO (201911), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, Toolforge, cloud-services-team (Kanban), Kubernetes
dduvall closed T233316: Deployment Pipeline fails with CPS error for Kartotherian as Resolved.
Wed, Oct 23, 5:06 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)

Oct 10 2019

dduvall added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

@Mathew.onipe no problem!

Oct 10 2019, 7:54 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall closed T233849: 1.35.0-wmf.1 deployment blockers as Resolved.
Oct 10 2019, 7:29 PM · Release, Train Deployments

Oct 9 2019

dduvall added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

Looks like lerna is one of devDependencies and so npm install --production (command used when the blubber config has node: { env: production }) fails due to the missing lerna binary. This isn't an issue with the pipeline or Blubber and will have to be worked out on the project's end.

Oct 9 2019, 9:32 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall added a parent task for T223393: switch wikitech to PHP 7.2: T233849: 1.35.0-wmf.1 deployment blockers.
Oct 9 2019, 7:12 PM · cloud-services-team (Kanban), Release-Engineering-Team-TODO, wikitech.wikimedia.org, PHP 7.2 support, serviceops, Operations
dduvall added a subtask for T233849: 1.35.0-wmf.1 deployment blockers: T223393: switch wikitech to PHP 7.2.
Oct 9 2019, 7:12 PM · Release, Train Deployments

Oct 8 2019

dduvall moved T233316: Deployment Pipeline fails with CPS error for Kartotherian from INBOX to Done (within RelEng) on the Release-Engineering-Team-TODO (201910) board.
Oct 8 2019, 5:43 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall edited projects for T233316: Deployment Pipeline fails with CPS error for Kartotherian, added: Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO (201909).
Oct 8 2019, 5:43 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall closed T232024: Branch REL1_34 for MediaWiki and deployed extensions, a subtask of T232023: Release MW 1.34, as Resolved.
Oct 8 2019, 5:25 PM · Core Platform Team, MediaWiki-Releasing, MW-1.34-notes, MW-1.34-release
dduvall closed T232024: Branch REL1_34 for MediaWiki and deployed extensions, a subtask of T232030: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched, as Resolved.
Oct 8 2019, 5:25 PM · Release-Engineering-Team-TODO (201910), Release, MediaWiki-General
dduvall closed T232024: Branch REL1_34 for MediaWiki and deployed extensions as Resolved.

The REL1_34 branch has been created for all active repos and core was prepared with submodules from the "base" bundle. I initially had some trouble with the branch.py script but got it fixed up in the end. I also moved its implementation to a shared module and created a branch-version.py script which should ease the entire process for the next branch cut.

Oct 8 2019, 5:25 PM · Release-Engineering-Team-TODO (201910), Core Platform Team, MediaWiki-Releasing, MW-1.34-notes, MW-1.34-release

Oct 7 2019

dduvall awarded Blog Post: Introducing Phatality a Party Time token.
Oct 7 2019, 5:16 PM · Phabricator
dduvall triaged T233849: 1.35.0-wmf.1 deployment blockers as Normal priority.
Oct 7 2019, 5:04 PM · Release, Train Deployments

Oct 3 2019

dduvall closed T220750: 1.34.0-wmf.25 deployment blockers as Resolved.
Oct 3 2019, 7:31 PM · Release-Engineering-Team (Deployment services), Release, Train Deployments

Oct 2 2019

dduvall added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

Looks like it can't find node_modules:

npm WARN Local package.json exists, but node_modules missing, did you mean to install?

Oct 2 2019, 5:31 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall placed T205923: TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics up for grabs.
Oct 2 2019, 5:22 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure
dduvall closed T182759: Add Prometheus exporter to Jenkins instances, a subtask of T177197: Export Prometheus-compatible JVM metrics from JVMs in production, as Declined.
Oct 2 2019, 5:21 PM · User-fgiunchedi, Goal, Operations
dduvall closed T182759: Add Prometheus exporter to Jenkins instances, a subtask of T205927: Collect and expose Jenkins build metrics for visualization, reporting, and analysis, as Declined.
Oct 2 2019, 5:21 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure
dduvall closed T182759: Add Prometheus exporter to Jenkins instances as Declined.

Marking this as "declined" to remove the task from view. We can always revive or reference this task should be pick the work back up.

Oct 2 2019, 5:21 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, User-fgiunchedi, Goal, Operations
dduvall changed the status of T182759: Add Prometheus exporter to Jenkins instances, a subtask of T177197: Export Prometheus-compatible JVM metrics from JVMs in production, from Open to Stalled.
Oct 2 2019, 5:20 PM · User-fgiunchedi, Goal, Operations
dduvall changed the status of T182759: Add Prometheus exporter to Jenkins instances from Open to Stalled.

Work on this has stalled. I've uninstalled the Prometheus plugin from Jenkins for now.

Oct 2 2019, 5:20 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, User-fgiunchedi, Goal, Operations
dduvall changed the status of T182759: Add Prometheus exporter to Jenkins instances, a subtask of T205927: Collect and expose Jenkins build metrics for visualization, reporting, and analysis, from Open to Stalled.
Oct 2 2019, 5:20 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure

Sep 27 2019

dduvall closed T229246: Gerrit/Argo CI proof of concept as Resolved.
Sep 27 2019, 5:44 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure

Sep 24 2019

dduvall awarded T233369: Blubberoid endpoint intermittently routing to MediaWiki backend a Like token.
Sep 24 2019, 7:12 PM · Traffic, Operations, Release-Engineering-Team-TODO
dduvall added a comment to T233298: Proposal: simplify set up of basic CI jobs for new projects.

Although it wasn't the primary use case for which the tools were designed, this kind of arbitrary repo-/project-/user-supplied CI workflow is possible now using Blubber + Pipelinelib.

Sep 24 2019, 6:59 PM · Release-Engineering-Team (CI & Testing services), serviceops-radar, Continuous-Integration-Infrastructure

Sep 19 2019

dduvall edited projects for T233369: Blubberoid endpoint intermittently routing to MediaWiki backend, added: Traffic; removed serviceops.
Sep 19 2019, 10:38 PM · Traffic, Operations, Release-Engineering-Team-TODO
dduvall moved T233369: Blubberoid endpoint intermittently routing to MediaWiki backend from INBOX to Watching/External on the Release-Engineering-Team-TODO board.
Sep 19 2019, 10:29 PM · Traffic, Operations, Release-Engineering-Team-TODO
dduvall triaged T233369: Blubberoid endpoint intermittently routing to MediaWiki backend as High priority.
Sep 19 2019, 10:28 PM · Traffic, Operations, Release-Engineering-Team-TODO
dduvall created T233369: Blubberoid endpoint intermittently routing to MediaWiki backend.
Sep 19 2019, 10:25 PM · Traffic, Operations, Release-Engineering-Team-TODO
thcipriani awarded T233316: Deployment Pipeline fails with CPS error for Kartotherian a Love token.
Sep 19 2019, 7:35 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.

There are a few things going on here.

Sep 19 2019, 7:07 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall claimed T233316: Deployment Pipeline fails with CPS error for Kartotherian.
Sep 19 2019, 5:22 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)

Sep 12 2019

dduvall updated the task description for T229246: Gerrit/Argo CI proof of concept.
Sep 12 2019, 4:17 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall updated the task description for T229246: Gerrit/Argo CI proof of concept.
Sep 12 2019, 4:13 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure

Sep 11 2019

dduvall updated the task description for T229246: Gerrit/Argo CI proof of concept.
Sep 11 2019, 11:07 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall updated subscribers of T229246: Gerrit/Argo CI proof of concept.

@thcipriani, @hashar, @LarsWirzenius, @zeljkofilipin, @brennen see the updated description for my summary.

Sep 11 2019, 11:07 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall updated the task description for T229246: Gerrit/Argo CI proof of concept.
Sep 11 2019, 11:05 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure

Sep 10 2019

dduvall added a comment to T229246: Gerrit/Argo CI proof of concept.

I'm still working on a thorough summary of this PoC setup, but here's a quickstart on how users might interact with it—on the command line.

Sep 10 2019, 11:21 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure

Jul 29 2019

dduvall updated the task description for T229246: Gerrit/Argo CI proof of concept.
Jul 29 2019, 4:39 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall moved T229246: Gerrit/Argo CI proof of concept from INBOX to Doing on the Release-Engineering-Team-TODO (201908) board.
Jul 29 2019, 4:35 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall triaged T229246: Gerrit/Argo CI proof of concept as Normal priority.
Jul 29 2019, 4:34 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure
dduvall created T229246: Gerrit/Argo CI proof of concept.
Jul 29 2019, 4:33 PM · User-zeljkofilipin, Release-Engineering-Team-TODO (201909), Release Pipeline, Continuous-Integration-Infrastructure

Jun 8 2019

dduvall created T225336: Pipelinelib `.pipeline/config.yaml` could use some kind of includes or inheritance feature.
Jun 8 2019, 12:21 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Release Pipeline
dduvall created T225335: Implement pipeline config validation.
Jun 8 2019, 12:14 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Patch-For-Review, Release Pipeline

Jun 7 2019

dduvall placed T222820: Experiment with hosted kubernetes solutions for Beta up for grabs.

I was able to run Mathoid just fine on GKE using the latest chart. What remains of this experiment, however, is getting Beta Cluster's MediaWiki talking to a service deployed to GKE (or Amazon EKS if that makes sense for us policy/budget wise) and vice versa; Basically the networking part.

Jun 7 2019, 11:55 PM · Release-Engineering-Team-TODO, Beta-Cluster-Infrastructure, Release Pipeline
dduvall closed T224035: Create service-pipeline job aware of .pipeline/config.yaml as Resolved.
Jun 7 2019, 11:43 PM · Patch-For-Review, Release Pipeline, Release-Engineering-Team (Kanban)

Jun 5 2019

dduvall edited P8591 build_report.rb.
Jun 5 2019, 1:44 AM
dduvall updated the title for P8591 build_report.rb from untitled to build_report.rb.
Jun 5 2019, 1:42 AM

May 31 2019

dduvall added a comment to T193824: Determine a standard way of installing MediaWiki lib/extension dependencies within containers.

We have filled this task following Release-Engineering-Team May 2018 offsite. One of the need is for CI to be able to **optionally* install dependent extensions. The current system is centrally managed in integration/config.git and suffers from several issues:

  • requires review/merge/deploy from one of the CI maintainer
  • is not aware of branches
May 31 2019, 7:13 PM · Release-Engineering-Team-TODO, MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), Patch-For-Review, Release Pipeline

May 30 2019

dduvall closed T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): as Resolved.

Alert is back to OK.

May 30 2019, 11:03 PM · Release-Engineering-Team-TODO (201907), Continuous-Integration-Infrastructure, Operations
dduvall closed T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):, a subtask of T207707: contint1001 store docker images on separate partition or disk, as Resolved.
May 30 2019, 11:03 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201907), serviceops, Operations, Continuous-Integration-Infrastructure
dduvall claimed T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):.
May 30 2019, 10:50 PM · Release-Engineering-Team-TODO (201907), Continuous-Integration-Infrastructure, Operations

May 24 2019

dduvall committed rGBLBR9959bec8cb53: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 24 2019, 5:24 PM

May 23 2019

dduvall committed rGBLBRd2e2cf5622c3: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 23 2019, 7:37 PM

May 22 2019

dduvall committed rGBLBR22f3bd5d4b20: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 22 2019, 7:24 PM
dduvall committed rGBLBR545ba3de1a89: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 22 2019, 12:01 AM
dduvall committed rGBLBR184e6cc838e5: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 22 2019, 12:01 AM
dduvall committed rGBLBR1e986afdd03c: pipeline: Include .pipeline/config.yaml (authored by dduvall).
pipeline: Include .pipeline/config.yaml
May 22 2019, 12:01 AM

May 21 2019

dduvall updated the task description for T224069: Add/reserve a Jenkins node for the pipeline's trigger jobs.
May 21 2019, 10:59 PM · Release-Engineering-Team (Kanban), Release Pipeline
dduvall created T224069: Add/reserve a Jenkins node for the pipeline's trigger jobs.
May 21 2019, 10:55 PM · Release-Engineering-Team (Kanban), Release Pipeline

May 9 2019

dduvall awarded Blog Post: Nova-network is gone! a Evil Spooky Haunted Tree token.
May 9 2019, 3:46 PM · Toolforge, Cloud-VPS

May 8 2019

dduvall updated the task description for T222820: Experiment with hosted kubernetes solutions for Beta.
May 8 2019, 6:01 PM · Release-Engineering-Team-TODO, Beta-Cluster-Infrastructure, Release Pipeline
dduvall updated the task description for T222820: Experiment with hosted kubernetes solutions for Beta.
May 8 2019, 6:01 PM · Release-Engineering-Team-TODO, Beta-Cluster-Infrastructure, Release Pipeline
dduvall updated the task description for T222820: Experiment with hosted kubernetes solutions for Beta.
May 8 2019, 5:48 PM · Release-Engineering-Team-TODO, Beta-Cluster-Infrastructure, Release Pipeline
dduvall updated the task description for T222820: Experiment with hosted kubernetes solutions for Beta.
May 8 2019, 5:41 PM · Release-Engineering-Team-TODO, Beta-Cluster-Infrastructure, Release Pipeline

May 7 2019

dduvall closed T222199: Post generated docs for pipelinelib as Resolved.

API documentation for pipelinelib is now available at https://doc.wikimedia.org/pipelinelib/ and kept up-to-date via a postmerge job.

May 7 2019, 11:10 PM · Patch-For-Review, Continuous-Integration-Config, Release-Engineering-Team (Kanban), Release Pipeline
dduvall closed T222199: Post generated docs for pipelinelib, a subtask of T210267: Execution of the deployment pipeline should be configurable via .pipeline/config.yaml, as Resolved.
May 7 2019, 11:10 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Release Pipeline, Operations, ORES, Scoring-platform-team

Apr 24 2019

dduvall committed rGBLBRa260ee40f28a: experimental: Support buildkit (authored by dduvall).
experimental: Support buildkit
Apr 24 2019, 1:33 PM

Apr 19 2019

dduvall committed rGBLBR5853dfac8258: experimental: Support buildkit (authored by dduvall).
experimental: Support buildkit
Apr 19 2019, 11:05 PM
dduvall committed rGBLBRbbf4e93072f4: experimental: Support buildkit (authored by dduvall).
experimental: Support buildkit
Apr 19 2019, 11:05 PM

Apr 17 2019

dduvall committed rGBLBR11dab2e54a22: experimental: Support LLB output format (authored by dduvall).
experimental: Support LLB output format
Apr 17 2019, 7:01 PM

Apr 9 2019

dduvall edited P8379 analyze-thread-dumps.sh.
Apr 9 2019, 9:47 PM

Apr 8 2019

dduvall closed T206678: 1.33.0-wmf.24 deployment blockers as Resolved.
Apr 8 2019, 8:00 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall closed T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons as Resolved.
Apr 8 2019, 8:00 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall closed T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons, a subtask of T206678: 1.33.0-wmf.24 deployment blockers, as Resolved.
Apr 8 2019, 8:00 PM · Release-Engineering-Team (Kanban), Release, Train Deployments

Apr 4 2019

dduvall added a comment to T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons.
{
  "_index": "logstash-2019.04.04",
  "_type": "mediawiki",
  "_id": "AWnqCdqV8aQffZ3HyCjK",
  "_version": 1,
  "_score": null,
  "_source": {
    "exception": {
      "trace": "#0 /srv/mediawiki/php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php(289): Wikimedia\\Rdbms\\LBFactory->commitMasterChanges(string, array)\n#1 /srv/mediawiki/php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php(67): JobExecutor->commitMasterChanges(Wikimedia\\Rdbms\\LBFactoryMulti, string)\n#2 /srv/mediawiki/rpc/RunSingleJob.php(77): JobExecutor->execute(array)\n#3 {main}",
      "code": 0,
      "file": "/srv/mediawiki/php-1.33.0-wmf.24/includes/libs/rdbms/lbfactory/LBFactory.php:261",
      "message": "RefreshLinksJob::run: transaction round 'RefreshLinksJob::runForTitle' still running.",
      "class": "Wikimedia\\Rdbms\\DBTransactionError"
    },
    "server": "jobrunner.discovery.wmnet",
    "phpversion": "5.6.99-hhvm",
    "wiki": "enwiki",
    "channel": "exception",
    "exception_id": "XKW0twpAAEUAAFjwWmIAAAAA",
    "program": "mediawiki",
    "type": "mediawiki",
    "message_checksum": "40dee3e70084b7a6f50133fd69b72f74",
    "caught_by": "mwe_handler",
    "exception_url": "/rpc/RunSingleJob.php",
    "http_method": "POST",
    "host": "mw1302",
    "@version": 1,
    "shard": "s1",
    "timestamp": "2019-04-04T20:28:27+00:00",
    "severity": "err",
    "level": "ERROR",
    "ip": "10.64.16.67",
    "mwversion": "1.33.0-wmf.24",
    "logsource": "mw1302",
    "message": "[XKW0twpAAEUAAFjwWmIAAAAA] /rpc/RunSingleJob.php   Wikimedia\\Rdbms\\DBTransactionError from line 261 of /srv/mediawiki/php-1.33.0-wmf.24/includes/libs/rdbms/lbfactory/LBFactory.php: RefreshLinksJob::run: transaction round 'RefreshLinksJob::runForTitle' still running.",
    "normalized_message": "[{exception_id}] {exception_url}   Wikimedia\\Rdbms\\DBTransactionError from line 261 of /srv/mediawiki/php-1.33.0-wmf.24/includes/libs/rdbms/lbfactory/LBFactory.php: RefreshLinksJob::run: transaction round 'RefreshLinksJob::runForTitle' still running.",
    "url": "/rpc/RunSingleJob.php",
    "reqId": "XKW0twpAAEUAAFjwWmIAAAAA",
    "tags": [
      "input-kafka-rsyslog-udp-localhost",
      "rsyslog-udp-localhost",
      "kafka",
      "es"
    ],
    "referrer": null,
    "@timestamp": "2019-04-04T20:28:27.000Z",
    "facility": "user"
  },
  "fields": {
    "@timestamp": [
      1554409707000
    ]
  },
  "sort": [
    1554409707000
  ]
}
Apr 4 2019, 8:29 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall added a comment to T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons.

Following syncs and promotion to all wikis, the error rate is much lower than the previously seen spike, but it's still rather high, average of 800/min.

Apr 4 2019, 8:28 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform

Apr 3 2019

dduvall edited projects for T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons, added: Event-Platform, Core Platform Team; removed MediaWiki-JobQueue.
Apr 3 2019, 8:17 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall renamed T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons from Spike in DBTransactionError following 1.33.0-wmf.24 group1 promotion to RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons.
Apr 3 2019, 8:16 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall triaged T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons as Unbreak Now! priority.
Apr 3 2019, 8:07 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall created T220037: RefreshLinksJob::runForTitle: transaction round 'RefreshLinksJob::run' already started on commons.
Apr 3 2019, 8:06 PM · Core Platform Team Workboards (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Analytics, Event-Platform
dduvall added a comment to T219510: Citoid should only usurp "<ref" / ctrl+shift+k shortcuts if configured.

@Mvolz I'm currently holding the group1 train deployment. Do you have an update or ETA on a fix?

I'm not 100% it's worth holding the train over - it'll disable citoid on at least one small language wiki. But the fix isn't done yet, no. Alternatively we can just revert it but note it requires reverting the change in the cite extension as well.

Apr 3 2019, 7:42 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), VisualEditor (Current work), Citoid
dduvall added a comment to T219510: Citoid should only usurp "<ref" / ctrl+shift+k shortcuts if configured.

@Mvolz I'm currently holding the group1 train deployment. Do you have an update or ETA on a fix?

Apr 3 2019, 7:28 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), VisualEditor (Current work), Citoid
dduvall triaged T219510: Citoid should only usurp "<ref" / ctrl+shift+k shortcuts if configured as Unbreak Now! priority.

Marking UBN as is our policy with train blockers.

Apr 3 2019, 7:03 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), VisualEditor (Current work), Citoid

Apr 2 2019

dduvall assigned T219738: PHP Warning: Array key should be either a string or an integer to Pchelolo.

The https://gerrit.wikimedia.org/r/500363 fixes it. Don't want to self-merge my own patch though.

Apr 2 2019, 6:10 PM · Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Workboards (Done with CPT), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Beta-Cluster-reproducible, Analytics, Event-Platform, Wikimedia-production-error
dduvall triaged T219738: PHP Warning: Array key should be either a string or an integer as Unbreak Now! priority.

Marking this UBN as is policy for all deployment blockers.

Apr 2 2019, 5:46 PM · Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Core Platform Team Workboards (Done with CPT), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Beta-Cluster-reproducible, Analytics, Event-Platform, Wikimedia-production-error
dduvall raised the priority of T217087: Error "A non well formed numeric value encountered" (from ImageMap) from Normal to Unbreak Now!.

Thanks for escalating this prior to deployment. Marking UBN as is policy with all deployment blockers.

Apr 2 2019, 5:44 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), User-notice, User-Ryasmeen, Editing-team, VisualEditor, PHP 7.2 support, ImageMap, Wikimedia-production-error

Apr 1 2019

dduvall added a comment to T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish.

Nominally fixed -> not UBN.

Apr 1 2019, 4:45 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), Patch-For-Review, MediaWiki-Logging, Wikimedia-production-error
dduvall removed a parent task for T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish: T206677: 1.33.0-wmf.23 deployment blockers.
Apr 1 2019, 4:44 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), Patch-For-Review, MediaWiki-Logging, Wikimedia-production-error
dduvall removed a subtask for T206677: 1.33.0-wmf.23 deployment blockers: T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish.
Apr 1 2019, 4:44 PM · Release-Engineering-Team (Kanban), Release, Train Deployments

Mar 28 2019

dduvall closed T206677: 1.33.0-wmf.23 deployment blockers as Resolved.
Mar 28 2019, 7:42 PM · Release-Engineering-Team (Kanban), Release, Train Deployments

Mar 26 2019

dduvall added a comment to T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish.

@pmiazga And more importantly: thanks for the fix! :)

Mar 26 2019, 5:24 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), Patch-For-Review, MediaWiki-Logging, Wikimedia-production-error
dduvall added a comment to T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish.

@dduvall we're very close to merge. Can you give us couple more minutes, please?

Mar 26 2019, 5:20 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), Patch-For-Review, MediaWiki-Logging, Wikimedia-production-error
dduvall added a comment to T218940: Exception "At least one of: RCID, revision ID, and log ID MUST be specified" from ManualLogEntry::publish.

FYI I'll be starting the 1.33.0-wmf.23 branch cut shortly. If https://gerrit.wikimedia.org/r/c/mediawiki/core/+/498809 doesn't make it in I will plan on backporting it prior to actual deployment.

Mar 26 2019, 5:06 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), Patch-For-Review, MediaWiki-Logging, Wikimedia-production-error

Mar 22 2019

dduvall renamed T218827: Evaluate Argo from Evaluate Argo Workflow to Evaluate Argo.
Mar 22 2019, 8:07 PM · Release-Engineering-Team (Kanban)
dduvall closed T218827: Evaluate Argo as Resolved.

First, some clarification about the various Argo projects.

Mar 22 2019, 7:05 PM · Release-Engineering-Team (Kanban)
dduvall closed T218827: Evaluate Argo, a subtask of T217325: Consider and evaluate possible new CI tooling, as Resolved.
Mar 22 2019, 7:05 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban)