Page MenuHomePhabricator

dduvall (Dan Duvall)
Automation Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:24 PM (232 w, 2 d)
Availability
Available
IRC Nick
marxarelli
LDAP User
Dduvall
MediaWiki User
DDuvall (WMF) [ Global Accounts ]

Recent Activity

Yesterday

dduvall closed T217912: Evaluate Tekton Pipeline as Resolved.
Thu, Mar 21, 7:25 PM · Release-Engineering-Team (Kanban)
dduvall closed T217912: Evaluate Tekton Pipeline, a subtask of T217325: Consider and evaluate possible new CI tooling, as Resolved.
Thu, Mar 21, 7:25 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban)
dduvall added a comment to T217912: Evaluate Tekton Pipeline.

Tekton is narrow in scope but it seems to do what it does well: It provides a coherent set of Custom Resource Definitions (CRD) necessary to get CI type workloads running on k8s efficiently and quickly. It's narrowness in scope and CRD nature yield these benefits and drawbacks:

Thu, Mar 21, 7:10 PM · Release-Engineering-Team (Kanban)
dduvall moved T218827: Evaluate ArgoCD from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
Thu, Mar 21, 6:12 PM · Release-Engineering-Team (Kanban)
dduvall claimed T218827: Evaluate ArgoCD.
Thu, Mar 21, 6:12 PM · Release-Engineering-Team (Kanban)

Wed, Mar 20

dduvall created T218827: Evaluate ArgoCD.
Wed, Mar 20, 9:31 PM · Release-Engineering-Team (Kanban)
dduvall closed T218334: Evaluate Jenkins X as Resolved.

I followed the Getting Started section of the Jenkins X documentation to get it installed locally using minikube.

Wed, Mar 20, 4:08 PM · Jenkins, Release-Engineering-Team (Kanban)
dduvall closed T218334: Evaluate Jenkins X, a subtask of T217325: Consider and evaluate possible new CI tooling, as Resolved.
Wed, Mar 20, 4:08 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban)

Tue, Mar 19

dduvall edited P8237 Blubber built on Tekton + minikube.
Tue, Mar 19, 10:30 PM

Fri, Mar 15

dduvall added a comment to T205911: Track and install additional npm packages for all service container images.

Or: Isn't packages-lock.json supposed to be used for this kind of thing? Is that not the official way to declare an authoritative list of dependencies that a Node project requires in a production environment? (That thought is informed by a Ruby background, however, as Gemfile.lock is such a solution in that world.)

Fri, Mar 15, 9:32 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Core Platform Team Backlog (Watching / External), Services (watching), Operations, Release Pipeline
dduvall added a comment to T205911: Track and install additional npm packages for all service container images.

Having services depend on a meta package that itself depends on the real package is a recipe for deps hell, and I'd really like to avoid that. Not only does it create more manual work, but it also adds complexity that is delegated to humans, and, as we know, PEBKAC is the number #1 reason things go wrong.

Fri, Mar 15, 8:01 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Core Platform Team Backlog (Watching / External), Services (watching), Operations, Release Pipeline

Thu, Mar 14

dduvall added a comment to T205911: Track and install additional npm packages for all service container images.

I'm pushing back on the patchset to Blubber for a couple of reasons:

Thu, Mar 14, 10:25 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Core Platform Team Backlog (Watching / External), Services (watching), Operations, Release Pipeline

Fri, Mar 8

dduvall created T217912: Evaluate Tekton Pipeline.
Fri, Mar 8, 5:58 PM · Release-Engineering-Team (Kanban)

Tue, Mar 5

dduvall committed rGBLBRec0af37c1e41: Include a basic go.mod that declares the package name (authored by dduvall).
Include a basic go.mod that declares the package name
Tue, Mar 5, 7:37 PM
dduvall committed rGBLBR71f4eb2581e6: Include a basic go.mod that declares the package name (authored by dduvall).
Include a basic go.mod that declares the package name
Tue, Mar 5, 7:16 PM

Feb 15 2019

dduvall updated subscribers of T209106: Setup session storage service testing/continuous integration.

Just wanted to offer some feedback based on the patches you already have and some of the things @Eevans, @thcipriani and I talked about during All Hands.

Feb 15 2019, 11:43 PM · Patch-For-Review, User-Clarakosi, Core Platform Team Backlog (Next), Core Platform Team (Session Management Service (CDP2)), User-Eevans
dduvall added a comment to T210267: The continuous release pipeline should support more than one service per repo.

I went a little crazy with a new config proposal in anticipation of us implementing T216272: The pipeline should provide a way to save artifacts from a stage. It's more loosely coupled, like what @thcipriani proposed earlier, with some extra fields for clearly defining the way in which stages should be executed and different methods for publishing artifacts. We'd likely want some basic policy/validation that enforces sanity (e.g. if it's publishing an image in a stage, it must also specify testDeploy, etc.). Useful defaults would also be important to cut down on configuration duplication.

Feb 15 2019, 8:51 PM · Release Pipeline, Release-Engineering-Team (Backlog), Operations, ORES, Scoring-platform-team
dduvall edited P8093 .pipeline/config.yaml.
Feb 15 2019, 8:35 PM
dduvall edited P8093 .pipeline/config.yaml.
Feb 15 2019, 8:14 PM
dduvall closed T216204: contint1001 and contint2001 should allow port 9418 from their ipv6 addresses as Resolved.

Deployed and tested.

Feb 15 2019, 12:24 AM · Patch-For-Review, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure

Feb 14 2019

dduvall moved T216204: contint1001 and contint2001 should allow port 9418 from their ipv6 addresses from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
Feb 14 2019, 11:31 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
dduvall claimed T216204: contint1001 and contint2001 should allow port 9418 from their ipv6 addresses.
Feb 14 2019, 11:31 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
dduvall created T216204: contint1001 and contint2001 should allow port 9418 from their ipv6 addresses.
Feb 14 2019, 11:20 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure

Feb 13 2019

dduvall claimed T177867: Pipeline image build cleanup.
Feb 13 2019, 6:58 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline

Feb 7 2019

dduvall renamed T212247: Refactor integration/pipelinelib to use blubberoid.wikimedia.org from Refactor integration/pipelinelib to use blubberoid.discovery.wmnet to Refactor integration/pipelinelib to use blubberoid.wikimedia.org.
Feb 7 2019, 11:48 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall committed rMSCAa29a6c6f43cd: Rename and simplify some git deploy functions (authored by dduvall).
Rename and simplify some git deploy functions
Feb 7 2019, 12:09 PM
dduvall committed rMSCAd8bcae58233f: Execute distinct stages of deployment separately (authored by dduvall).
Execute distinct stages of deployment separately
Feb 7 2019, 12:09 PM
dduvall committed rMSCA4db5964f67fa: Execute distinct stages of deployment separately (authored by dduvall).
Execute distinct stages of deployment separately
Feb 7 2019, 12:09 PM
dduvall committed rMSCA2f3a15eae70a: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:09 PM
dduvall committed rMSCA32b482a51de5: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:09 PM
dduvall committed rMSCAd2b1d69d41a1: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:09 PM
dduvall committed rMSCA43e39fd7ae9a: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:09 PM
dduvall committed rMSCAe6a551829112: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:08 PM
dduvall committed rMSCA813daa3f9193: Support atomic promotion and rollback (authored by dduvall).
Support atomic promotion and rollback
Feb 7 2019, 12:08 PM
dduvall committed rMSCAb8ab63f8003c: Filter target host logging from stdout of main process (authored by dduvall).
Filter target host logging from stdout of main process
Feb 7 2019, 12:08 PM

Jan 28 2019

dduvall committed rGBLBR764c402034b6: Comment about copies config expansion inefficiency (authored by dduvall).
Comment about copies config expansion inefficiency
Jan 28 2019, 11:43 PM
dduvall committed rGBLBRfd20d346ddfe: Unify `copies` and `artifacts` configuration (authored by dduvall).
Unify `copies` and `artifacts` configuration
Jan 28 2019, 11:26 PM
dduvall committed rGBLBR20e9b739e56f: Bump version number (authored by thcipriani).
Bump version number
Jan 28 2019, 11:08 PM

Jan 19 2019

dduvall committed rGBLBRd10e4b3e00c6: Unify `copies` and `artifacts` configuration (authored by dduvall).
Unify `copies` and `artifacts` configuration
Jan 19 2019, 12:01 AM
dduvall committed rGBLBR6cc05320b12f: Unify `copies` and `artifacts` configuration (authored by dduvall).
Unify `copies` and `artifacts` configuration
Jan 19 2019, 12:01 AM
dduvall committed rGBLBR341794300c01: Unify `copies` and `artifacts` configuration (authored by dduvall).
Unify `copies` and `artifacts` configuration
Jan 19 2019, 12:01 AM

Jan 18 2019

dduvall renamed T211625: Unify configuration for local build-context copies and variant artifacts from Manually defining artifacts results in default copy of all project files to Unify configuration for local build-context copies and variant artifacts.
Jan 18 2019, 10:46 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)

Jan 17 2019

dduvall closed T206667: 1.33.0-wmf.13 deployment blockers as Resolved.
Jan 17 2019, 8:28 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall awarded Blog Post: Gerrit now automatically adds reviewers a 100 token.
Jan 17 2019, 5:58 PM · Release-Engineering-Team, Gerrit

Jan 16 2019

dduvall added a comment to T204871: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments.

Just a note that for today's promotion of group1 to 1.33.0-wmf.13 (T206667), I segmented the group1 error-log dashboard to have a view of just these timeout errors and a view excluding them. It was very helpful in keeping on both the rise in timeouts and side effects or unrelated errors. I plan on saving the dashboards and adding links in the train docs.

Jan 16 2019, 11:36 PM · Release-Engineering-Team (Kanban), User-zeljkofilipin, Wikimedia-Incident, Wikimedia-production-error
dduvall awarded D1138: pipeline: test scap with CD pipeline a Love token.
Jan 16 2019, 8:48 PM · Release-Engineering-Team

Jan 15 2019

dduvall claimed T206667: 1.33.0-wmf.13 deployment blockers.

Cutting the branch.

Jan 15 2019, 6:04 PM · Release-Engineering-Team (Kanban), Release, Train Deployments

Jan 12 2019

dduvall added a comment to T210267: The continuous release pipeline should support more than one service per repo.

What new problems does this create?

  • Potential that one merge can use a lot of CI executors (also currently the case, but RelEng has some ability to mitigate currently)
  • Others?
Jan 12 2019, 1:09 AM · Release Pipeline, Release-Engineering-Team (Backlog), Operations, ORES, Scoring-platform-team

Jan 10 2019

dduvall closed T206666: 1.33.0-wmf.12 deployment blockers as Resolved.

Today's all-wiki deployment went much like yesterday's: We saw an increase in the MediaWiki error rate due to a flood of "timed out" errors and what seemed like a side effect of 500s caused by nginx/apache/hhvm socket timeouts. The rate increases lasted from approximately 20:11 until 20:34 UTC at which point they subsided to pre-deployment levels.

Jan 10 2019, 8:59 PM · Release, Train Deployments
dduvall moved T211625: Unify configuration for local build-context copies and variant artifacts from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
Jan 10 2019, 7:36 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall added a watcher for Release-Engineering-Team (Kanban): dduvall.
Jan 10 2019, 7:35 PM

Jan 9 2019

dduvall committed rGBLBR867a9943d94e: Tweaked logo to play nice with preview renderer on commons (authored by dduvall).
Tweaked logo to play nice with preview renderer on commons
Jan 9 2019, 11:32 PM

Jan 8 2019

dduvall added a comment to T212427: No namespace configured for entity type `form`.

@greg @dduvall train is unblocked now, verified on beta.

Jan 8 2019, 6:34 PM · MW-1.33-notes (1.33.0-wmf.12; 2019-01-08), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikimedia-production-error, Wikidata

Dec 20 2018

dduvall committed rGBLBR1ac4655cefaf: Provide a new Blubber logo (authored by dduvall).
Provide a new Blubber logo
Dec 20 2018, 9:25 PM
dduvall committed rGBLBR013ab435a354: Provide a new Blubber logo (authored by dduvall).
Provide a new Blubber logo
Dec 20 2018, 9:25 PM

Dec 19 2018

dduvall added a comment to T212251: Expose blubberoid to the public allowing CI in WMCS to be able to reach out as well to it.

As SREs I don't think we would like to expose a public service that is not going to receive any traffic any time soon, mostly for operational and complexity reasons. I guess we don't really have that need yet, do we?

But that is exactly the point of this task? Open the service to the outside world so we can start using it. Instances on WMCS can not reach production internal network (10.0.0.0/8), they get out via PAT/NAT and are considered to be just like any internet traffic (== untrusted).

Is it? Cause I gather that it is specifically for CI. In which case an instance in CI is architecturally a better construct.

Dec 19 2018, 6:36 PM · Patch-For-Review, serviceops, Release-Engineering-Team (Kanban), Release Pipeline

Dec 18 2018

dduvall updated the task description for T212251: Expose blubberoid to the public allowing CI in WMCS to be able to reach out as well to it.
Dec 18 2018, 8:48 PM · Patch-For-Review, serviceops, Release-Engineering-Team (Kanban), Release Pipeline
dduvall moved T212251: Expose blubberoid to the public allowing CI in WMCS to be able to reach out as well to it from Backlog to Blocked (externally) on the Release-Engineering-Team (Kanban) board.
Dec 18 2018, 8:44 PM · Patch-For-Review, serviceops, Release-Engineering-Team (Kanban), Release Pipeline
dduvall moved T212247: Refactor integration/pipelinelib to use blubberoid.wikimedia.org from Blocked (externally) to In-progress on the Release-Engineering-Team (Kanban) board.
Dec 18 2018, 8:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall moved T212247: Refactor integration/pipelinelib to use blubberoid.wikimedia.org from Backlog to Blocked (externally) on the Release-Engineering-Team (Kanban) board.
Dec 18 2018, 8:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall triaged T212247: Refactor integration/pipelinelib to use blubberoid.wikimedia.org as Normal priority.
Dec 18 2018, 8:43 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall created T212251: Expose blubberoid to the public allowing CI in WMCS to be able to reach out as well to it.
Dec 18 2018, 8:43 PM · Patch-For-Review, serviceops, Release-Engineering-Team (Kanban), Release Pipeline
dduvall created T212247: Refactor integration/pipelinelib to use blubberoid.wikimedia.org.
Dec 18 2018, 8:24 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall updated images of M271: Blubber(oid) logos.
Dec 18 2018, 8:05 PM · Release Pipeline (Blubber), Release-Engineering-Team
dduvall updated images of M271: Blubber(oid) logos.
Dec 18 2018, 8:03 PM · Release Pipeline (Blubber), Release-Engineering-Team

Dec 17 2018

dduvall updated images of M271: Blubber(oid) logos.
Dec 17 2018, 6:43 PM · Release Pipeline (Blubber), Release-Engineering-Team
dduvall created M271: Blubber(oid) logos.
Dec 17 2018, 6:39 PM · Release Pipeline (Blubber), Release-Engineering-Team

Dec 15 2018

dduvall committed rGBLBRdb27311386af: WIP Playing with new logos (authored by dduvall).
WIP Playing with new logos
Dec 15 2018, 12:51 AM

Dec 14 2018

dduvall committed rGBLBR752aaa2bab4e: WIP Playing with new logos (authored by dduvall).
WIP Playing with new logos
Dec 14 2018, 11:51 PM
dduvall committed rGBLBR928c748fe936: Playing with possible new logos (authored by dduvall).
Playing with possible new logos
Dec 14 2018, 9:37 PM

Dec 13 2018

dduvall added a comment to T211625: Unify configuration for local build-context copies and variant artifacts.

@thcipriani +1 to the proposal in general. I think it adds clarity to artifact definition and to the purpose of copies. Just some notes on config structs and implementation of sane defaults.

Dec 13 2018, 7:49 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall awarded Blog Post: Production Excellence: November 2018 a Like token.
Dec 13 2018, 12:07 AM

Dec 12 2018

dduvall closed T205920: Blubberoid – create swagger spec as Resolved.

Latest image for use with initial chart implementation is docker-registry.wikimedia.org/wikimedia/blubber:20181212233039-production.

Dec 12 2018, 11:42 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall closed T205920: Blubberoid – create swagger spec, a subtask of T205919: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline, as Resolved.
Dec 12 2018, 11:42 PM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, Operations, Release-Engineering-Team
dduvall committed rGBLBR4df63a83c93b: Provide OpenAPI spec for Blubberoid (authored by dduvall).
Provide OpenAPI spec for Blubberoid
Dec 12 2018, 11:03 PM
dduvall committed rGBLBR8a283f3ee5bd: Provide OpenAPI spec for Blubberoid (authored by dduvall).
Provide OpenAPI spec for Blubberoid
Dec 12 2018, 10:53 PM
dduvall committed rGBLBR662fa3de1fe5: Provide OpenAPI spec for Blubberoid (authored by dduvall).
Provide OpenAPI spec for Blubberoid
Dec 12 2018, 8:37 PM
dduvall committed rGBLBRa69fad03e884: Provide OpenAPI spec for Blubberoid (authored by dduvall).
Provide OpenAPI spec for Blubberoid
Dec 12 2018, 8:22 PM
dduvall committed rGBLBR27e774e414fe: Provide OpenAPI spec for Blubberoid (authored by dduvall).
Provide OpenAPI spec for Blubberoid
Dec 12 2018, 7:23 PM

Dec 10 2018

dduvall added a comment to T211625: Unify configuration for local build-context copies and variant artifacts.

One refactoring option I can think of would be to make the config for copying of project/application files explicit and use a sane default.

Dec 10 2018, 8:20 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall created T211625: Unify configuration for local build-context copies and variant artifacts.
Dec 10 2018, 8:03 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall committed rGBLBRd24243e647b8: Ignore blubberoid binary created by `make` (authored by dduvall).
Ignore blubberoid binary created by `make`
Dec 10 2018, 6:59 PM

Dec 2 2018

greg awarded T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode a Yellow Medal token.
Dec 2 2018, 7:27 AM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests
dduvall closed T210301: Status of deployment-redis0[56]?, a subtask of T208101: Migrate deployment-prep to eqiad1, as Resolved.
Dec 2 2018, 12:01 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Beta-Cluster-Infrastructure, Epic, Cloud-Services
dduvall closed T210301: Status of deployment-redis0[56]? as Resolved.

I just merged the final patch related to T210030: RedisBagOStuff is broken on beta that removed the stale redis server entries from the labs related mediawiki-config, and the instances are terminated.

Dec 2 2018, 12:01 AM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure

Dec 1 2018

dduvall added a comment to T210301: Status of deployment-redis0[56]?.

I'm also seeing MW fatals resulting from the inoperable redis servers and the mediawiki-config for redis_lock that still references them.

Dec 1 2018, 11:39 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
dduvall closed T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode as Resolved.

AFAICT, this and related errors are no longer occurring. It's not yet clear exactly what the underlying issue might have been, but in troubleshooting with @thcipriani yesterday we noticed a few things that might have had an impact. Since @thcipriani graciously jumped in to take care of the actual fixes, he might want/need to correct some of this information. :)

Dec 1 2018, 11:32 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests

Nov 30 2018

dduvall claimed T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode.

Looking at @Krenair 's log of replication lag, there's no indication of lag at any given time since the first entry ("Thu Nov 29 20:13:01 UTC 2018").

Nov 30 2018, 6:25 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests

Nov 29 2018

dduvall added a comment to T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode.

Sorry, I posted the wrong server's output. :) Editing...

Nov 29 2018, 8:37 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests
dduvall added a comment to T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode.

I'm also wondering how MediaWiki detects replication lag. Does it look at all DB hosts? And which MaridDB variables does it inspect? Is it possible that it's looking at deployment-db03 and errantly detecting it as a lagged replica?

Nov 29 2018, 8:31 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests
dduvall added a comment to T210557: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode.

Looking at SHOW SLAVE STATUS\G on deployment-db04 shows no current lag according to seconds_behind_master, but I'm looking into where we might see its historical values. I thought there was a prometheus collector set up for beta's maria DBs, but I'm not whether it monitors master/slave replication status.

Nov 29 2018, 8:24 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, User-zeljkofilipin, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Browser-Tests

Nov 27 2018

sbassett awarded Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party! a Like token.
Nov 27 2018, 10:06 PM · Continuous-Integration-Infrastructure, Release-Engineering-Team

Nov 26 2018

dduvall awarded T210341: beta-update-databases-eqiad is failing with a composer problem a Like token.
Nov 26 2018, 7:06 PM · Beta-Cluster-Infrastructure
dduvall added a comment to T209086: Programatically enable/disable extensions from the command line.

@Legoktm could you clarify whether this task tracks progress of programatic extension installation or of programatic enablement/disablement? The title says (and comments seem to relate to) the latter but the description links to a section of the feedback article for the former.

Nov 26 2018, 6:50 PM · Core Platform Team Kanban (Doing), Core Platform Team (Extension Management (TEC13)), MediaWiki-Configuration

Nov 22 2018

phuedx awarded Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party! a Love token.
Nov 22 2018, 5:21 PM · Continuous-Integration-Infrastructure, Release-Engineering-Team

Nov 20 2018

dduvall removed a subtask for T207694: Adopt JSON as blubber's internal configuration format: T207695: Refactor validation system to use jsonschema.
Nov 20 2018, 5:38 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall removed a parent task for T207695: Refactor validation system to use jsonschema: T207694: Adopt JSON as blubber's internal configuration format.
Nov 20 2018, 5:38 PM · Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall renamed T207696: Retain YAML support by converting to JSON in Blubber from Remove yaml unmarshalling from blubber to Retain YAML support by converting to JSON in Blubber.
Nov 20 2018, 5:37 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)

Nov 19 2018

awight awarded Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party! a Love token.
Nov 19 2018, 6:30 PM · Continuous-Integration-Infrastructure, Release-Engineering-Team
zeljkofilipin awarded Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party! a Pterodactyl token.
Nov 19 2018, 9:47 AM · Continuous-Integration-Infrastructure, Release-Engineering-Team