dduvall (Dan Duvall)
Automation Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:24 PM (201 w, 5 d)
Availability
Available
IRC Nick
marxarelli
LDAP User
Dduvall
MediaWiki User
DDuvall (WMF) [ Global Accounts ]

Recent Activity

Fri, Aug 17

dduvall created T202160: Evaluate different strategy for Docker CI instances.
Fri, Aug 17, 7:05 PM · Release Pipeline, Release-Engineering-Team (Next), Continuous-Integration-Infrastructure
dduvall committed rGBLBR81fdce8b45b9: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Fri, Aug 17, 3:43 PM

Wed, Aug 15

dduvall committed rGBLBR15a2e362c7e4: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Wed, Aug 15, 5:49 PM
dduvall committed rGBLBR60469ea8cb69: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Wed, Aug 15, 4:10 PM
dduvall committed rGBLBR1ec67aa17545: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Wed, Aug 15, 4:10 PM

Tue, Aug 14

dduvall committed rGBLBRf0b15721f188: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Tue, Aug 14, 11:43 PM
dduvall committed rGBLBR0438b26394db: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Tue, Aug 14, 11:40 PM
dduvall committed rGBLBR56264ec879d8: WIP Having fun making a stateless blubberd microservice (authored by dduvall).
WIP Having fun making a stateless blubberd microservice
Tue, Aug 14, 11:40 PM
dduvall committed rGBLBRb6c9e31fc5e7: Bump Blubber config version to v3 (authored by dduvall).
Bump Blubber config version to v3
Tue, Aug 14, 4:27 PM
dduvall committed rGBLBR374976d445b6: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Tue, Aug 14, 4:04 PM

Mon, Aug 13

dduvall committed rGBLBR0d7b3be867c2: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Mon, Aug 13, 11:34 PM
dduvall committed rGBLBRee76e705e7ce: Change import paths from Phabricator to Gerrit (authored by dduvall).
Change import paths from Phabricator to Gerrit
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR933c22c0f5e8: Change import paths from Phabricator to Gerrit (authored by dduvall).
Change import paths from Phabricator to Gerrit
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR561cd6be1276: Change import paths from Phabricator to Gerrit (authored by dduvall).
Change import paths from Phabricator to Gerrit
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR3b09b6d97a5a: Test Blubber in CI using Blubber (authored by dduvall).
Test Blubber in CI using Blubber
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRd8b25ae15b67: Provide Makefile rules for running linters and tests (authored by dduvall).
Provide Makefile rules for running linters and tests
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRe4472e92ee73: Test Blubber in CI using Blubber (authored by dduvall).
Test Blubber in CI using Blubber
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRb49be0cb930a: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR60cb75eda119: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRca2b9869889a: Makefile lint rule now tests gofmt output and runs go vet correctly (authored by dduvall).
Makefile lint rule now tests gofmt output and runs go vet correctly
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR48ecc80b81f5: Bump Blubber config version to v3 (authored by dduvall).
Bump Blubber config version to v3
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR224fc897f57f: Test Blubber in CI using Blubber (authored by dduvall).
Test Blubber in CI using Blubber
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR3be75f4a21de: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR8eb85b5ae190: Modify access rules (authored by dduvall).
Modify access rules
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR5be89e06e538: Give JenkinsBot rights to merge. (authored by dduvall).
Give JenkinsBot rights to merge.
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR4b1989b35a37: Removed temporary forge rights. (authored by dduvall).
Removed temporary forge rights.
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR574acf5af3d6: Modify access rules (authored by dduvall).
Modify access rules
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR3e172a4646a9: Adding temporary forge rights for import existing git history. (authored by dduvall).
Adding temporary forge rights for import existing git history.
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBRdf000af9db44: Created project (authored by dduvall).
Created project
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR518f812bef8a: Modify access rules (authored by dduvall).
Modify access rules
Mon, Aug 13, 11:29 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rGBLBR73190e4ed08f: Giving Release Engineering ownership. (authored by dduvall).
Giving Release Engineering ownership.
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRef3b51c2886c: Refactor builder to support file requirements and run pre-install (authored by dduvall).
Refactor builder to support file requirements and run pre-install
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRe7ce38ca630a: Test Blubber in CI using Blubber (authored by dduvall).
Test Blubber in CI using Blubber
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBRd8d277308387: Makefile lint rule now tests gofmt output and runs go vet correctly (authored by dduvall).
Makefile lint rule now tests gofmt output and runs go vet correctly
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR44dc7d91313e: Provide Makefile rules for running linters and tests (authored by dduvall).
Provide Makefile rules for running linters and tests
Mon, Aug 13, 11:29 PM
dduvall committed rGBLBR80936122c749: Change import paths from Phabricator to Gerrit (authored by dduvall).
Change import paths from Phabricator to Gerrit
Mon, Aug 13, 11:29 PM
dduvall closed T200452: Move Blubber repo and CR to Gerrit, a subtask of T191182: Stop using Differential for code review, as Resolved.
Mon, Aug 13, 10:54 PM · Release-Engineering-Team (Someday), Phabricator, Gerrit
dduvall closed T200452: Move Blubber repo and CR to Gerrit as Resolved.

Diffusion repo is now read-only and mirrors gerrit.wikimedia.org/r/blubber.

Mon, Aug 13, 10:54 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall updated the task description for T200452: Move Blubber repo and CR to Gerrit.
Mon, Aug 13, 10:53 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall added a comment to T201875: Allow new blubber builders to be implemented in yaml.

Just playing around:

Mon, Aug 13, 8:18 PM · Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall closed T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment as Resolved.
Mon, Aug 13, 7:30 PM · Release-Engineering-Team (Kanban), Release Pipeline
dduvall closed T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment, a subtask of T157469: Mathoid CI Container Build, as Resolved.
Mon, Aug 13, 7:30 PM · Tracking, User-mobrovac, Release Pipeline
dduvall closed T188935: modify service-pipeline to include helm install/helm test as Resolved.
Mon, Aug 13, 7:30 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall closed T188935: modify service-pipeline to include helm install/helm test, a subtask of T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment, as Resolved.
Mon, Aug 13, 7:30 PM · Release-Engineering-Team (Kanban), Release Pipeline

Fri, Aug 10

dduvall closed T200843: Generic CI job for running Blubber-built test entry points as Resolved.

Both blubber and integration/config are being tested by the new job upon new patches.

Fri, Aug 10, 6:17 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline

Fri, Aug 3

dduvall changed the status of T196940: Establish shared library for pipeline code used in Jenkins from Stalled to Open.

Blocker has been resolved.

Fri, Aug 3, 6:30 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall changed the status of T196940: Establish shared library for pipeline code used in Jenkins, a subtask of T196414: Build Math extension container on Postmerge, from Stalled to Open.
Fri, Aug 3, 6:30 PM · Release-Engineering-Team (Kanban), Release Pipeline
dduvall closed T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib as Resolved.

The checkout of the shared integration/pipelinelib plugin is now working on integration.wikimedia.org.

Fri, Aug 3, 6:29 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall closed T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib, a subtask of T196940: Establish shared library for pipeline code used in Jenkins, as Resolved.
Fri, Aug 3, 6:29 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
zeljkofilipin awarded T198640: Perform scap canary checks after sync-wikiversions a Pterodactyl token.
Fri, Aug 3, 2:24 PM · User-zeljkofilipin, Release-Engineering-Team (Next), Scap

Thu, Aug 2

dduvall added a comment to T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib.

I pinged the #jenkins folks on IRC and the author of the shared library plugin seems to think fixing workspacesDir is the right solution.

Thu, Aug 2, 5:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall added a comment to T200843: Generic CI job for running Blubber-built test entry points.

/srv/ssd/jenkins is found in the Jenkins configuration:

/var/lib/jenkins/config.xml
<hudson>
  <workspaceDir>/srv/ssd/jenkins/workspace/${ITEM_FULLNAME}</workspaceDir>
  <buildsDir>/srv/jenkins/builds/${ITEM_FULL_NAME}</buildsDir>
  ...
</hudson>

Since Jenkins 2.121.2 T199448 , those parameters are passed to the command line via system properties:

java \
  -Djenkins.model.Jenkins.buildsDir=/srv/jenkins/builds/${ITEM_FULL_NAME} \
  -jar /usr/share/jenkins/jenkins.war \
Thu, Aug 2, 3:59 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall added a comment to T199489: Helm test failing for CI namespace.

I did some manual testing btw, I am guessing this is the error

servicechecker.CheckError: Generic connection error: HTTPConnectionPool(host='mathoid-mathoid-alex-test', port=10044): Max retries exceeded with url: /?spec (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5024be490>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Thu, Aug 2, 3:50 PM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), Release Pipeline

Wed, Aug 1

dduvall updated the task description for T200452: Move Blubber repo and CR to Gerrit.
Wed, Aug 1, 11:50 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall moved T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib from Backlog to CI on the Release Pipeline board.
Wed, Aug 1, 11:48 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall moved T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
Wed, Aug 1, 11:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall claimed T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib.
Wed, Aug 1, 11:43 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall changed the status of T196940: Establish shared library for pipeline code used in Jenkins from Open to Stalled.

Currently blocked on T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib.

Wed, Aug 1, 11:43 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall changed the status of T196940: Establish shared library for pipeline code used in Jenkins, a subtask of T196414: Build Math extension container on Postmerge, from Open to Stalled.
Wed, Aug 1, 11:43 PM · Release-Engineering-Team (Kanban), Release Pipeline
dduvall added a subtask for T196940: Establish shared library for pipeline code used in Jenkins: T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib.
Wed, Aug 1, 11:42 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall added a parent task for T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib: T196940: Establish shared library for pipeline code used in Jenkins.
Wed, Aug 1, 11:42 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall created T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib.
Wed, Aug 1, 11:42 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline, Continuous-Integration-Infrastructure
dduvall edited P7414 Masterwork From Distant Lands.
Wed, Aug 1, 4:34 PM
dduvall added a comment to T200843: Generic CI job for running Blubber-built test entry points.

I've create the new job in Jenkins but it appears that the library loading fails due to permissions: https://integration.wikimedia.org/ci/job/blubber-test/1/console

Wed, Aug 1, 12:05 AM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline

Tue, Jul 31

dduvall claimed T200843: Generic CI job for running Blubber-built test entry points.
Tue, Jul 31, 7:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall moved T200843: Generic CI job for running Blubber-built test entry points from Backlog to CI on the Release Pipeline board.
Tue, Jul 31, 7:44 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall created T200843: Generic CI job for running Blubber-built test entry points.
Tue, Jul 31, 7:43 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline

Mon, Jul 30

dduvall added a comment to T200452: Move Blubber repo and CR to Gerrit.

An upstream patch has been submitted to fix the auth requirement for the go-import Gerrit plugin. We've also forked our own repo for the plugin at operations/software/gerrit/plugins/go-import and have merged/deployed the patch to our servers.

Mon, Jul 30, 10:17 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall moved T200348: Get helm test to dump more information from Backlog to CI on the Release Pipeline board.
Mon, Jul 30, 7:05 PM · Operations, Release-Engineering-Team (Kanban), Release Pipeline

Fri, Jul 27

greg awarded T198642: Add some facility to scap for custom logstash canary checks a Like token.
Fri, Jul 27, 7:18 PM · Release-Engineering-Team (Next), Scap

Thu, Jul 26

dduvall updated the task description for T200452: Move Blubber repo and CR to Gerrit.
Thu, Jul 26, 5:58 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall added a comment to T200452: Move Blubber repo and CR to Gerrit.

Looks like the go-import Gerrit plugin handles requests to https://gerrit.wikimedia.org/r/blubber?go-get=1 but points to https://gerrit.wikimedia.org/r/a/blubber. Cloning from there requires authentication and so the git clone subprocess bails. I'll look into whether we can configure the plugin to simply use git clone https://gerrit.wikimedia.org/r/blubber.

Thu, Jul 26, 5:58 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall updated the task description for T200452: Move Blubber repo and CR to Gerrit.
Thu, Jul 26, 5:35 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall claimed T200452: Move Blubber repo and CR to Gerrit.
Thu, Jul 26, 5:20 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall created T200452: Move Blubber repo and CR to Gerrit.
Thu, Jul 26, 5:19 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
dduvall accepted D1080: Add slash to directory path if copying > 1 file.
Thu, Jul 26, 5:18 PM · Release-Engineering-Team

Wed, Jul 25

dduvall requested changes to D1080: Add slash to directory path if copying > 1 file.

Looks like there are some failing Docker specific tests:

Wed, Jul 25, 8:21 PM · Release-Engineering-Team

Jul 10 2018

Krinkle awarded T198640: Perform scap canary checks after sync-wikiversions a Orange Medal token.
Jul 10 2018, 10:29 PM · User-zeljkofilipin, Release-Engineering-Team (Next), Scap

Jul 2 2018

dduvall lowered the priority of T196940: Establish shared library for pipeline code used in Jenkins from High to Normal.
Jul 2 2018, 5:03 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
dduvall created T198642: Add some facility to scap for custom logstash canary checks.
Jul 2 2018, 4:50 PM · Release-Engineering-Team (Next), Scap
dduvall updated subscribers of T198640: Perform scap canary checks after sync-wikiversions.
Jul 2 2018, 4:39 PM · User-zeljkofilipin, Release-Engineering-Team (Next), Scap
dduvall added a project to T198640: Perform scap canary checks after sync-wikiversions: Release-Engineering-Team (Next).
Jul 2 2018, 4:39 PM · User-zeljkofilipin, Release-Engineering-Team (Next), Scap
dduvall created T198640: Perform scap canary checks after sync-wikiversions.
Jul 2 2018, 4:38 PM · User-zeljkofilipin, Release-Engineering-Team (Next), Scap

Jun 28 2018

dduvall closed T191056: 1.32.0-wmf.10 deployment blockers as Resolved.
Jun 28 2018, 9:14 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall removed a subtask for T191056: 1.32.0-wmf.10 deployment blockers: T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.
Jun 28 2018, 9:14 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall removed a parent task for T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment: T191056: 1.32.0-wmf.10 deployment blockers.
Jun 28 2018, 9:14 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
dduvall added a comment to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.

@Tgr's fix worked. Thanks again, @Tgr! I'll remove this as a blocker but leave it open for tracking related cleanup work.

Jun 28 2018, 9:14 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Current plan for today's train rollout is mentioned in {T198350#4322794}. It'll be a graduated rollout of (group1 - commonswiki), then commonswiki, then time for thorough verification that the issue is resolved, then group2 time permitting.

Jun 28 2018, 6:04 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a comment to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.

Thanks again for mobilizing! And thanks for the fix, @Tgr!

Jun 28 2018, 6:03 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
dduvall added a comment to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.

It seems there are two competing patches here, and given the resulting data loss/corruption from yesterday I'm hesitant to roll the train today with anything that only theoretically addresses the root cause.

Jun 28 2018, 5:27 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
jcrespo awarded T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment a Love token.
Jun 28 2018, 11:50 AM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors

Jun 27 2018

dduvall added a project to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment: DBA.

Pinging DBA on this since the symptoms include lock timeouts.

Jun 27 2018, 9:06 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Rolling back group1 due to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment blocker.

Jun 27 2018, 7:43 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a comment to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.

A number of SlowTimer errors have shown up in fatalmonitor as well. I'm going to roll back the train for now.

Jun 27 2018, 7:43 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors
dduvall created T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.
Jun 27 2018, 7:28 PM · MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), DBA, MediaWiki-Database, Wikimedia-log-errors

Jun 26 2018

dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Manually cleaned up old deployments (wmf.3, wmf.4, wmf.5) to ensure enough disk space. I'm re-syncing now.

Jun 26 2018, 8:57 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Since the sync succeeded other than cdb update on mwdeploy2002, I'm going to move ahead with group0 today.

Jun 26 2018, 8:01 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Now cdb-rebuild failed on mwdebug2002 due to lack of disk space.

Jun 26 2018, 7:46 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a comment to T191056: 1.32.0-wmf.10 deployment blockers.

Trace of l10update failure (see related task and SAL message):

Jun 26 2018, 6:18 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release, Train Deployments
dduvall added a parent task for T197918: Archive the MwEmbedSupport extension: T191056: 1.32.0-wmf.10 deployment blockers.
Jun 26 2018, 6:17 PM · Patch-For-Review, User-MarcoAurelio, Repository-Admins, GitHub-Mirrors, Cleanup, MediaWiki-extensions-MwEmbedSupport