Page MenuHomePhabricator

dduvall (Dan Duvall)
Automation Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:24 PM (344 w, 6 d)
Availability
Busy Busy until Jun 5.
IRC Nick
marxarelli
LDAP User
Dduvall
MediaWiki User
DDuvall (WMF) [ Global Accounts ]

Recent Activity

Fri, Apr 30

dduvall claimed T225335: Implement pipeline config validation.
Fri, Apr 30, 3:59 PM · Release-Engineering-Team (Yak Shaving 🐃🪒), Release Pipeline

Wed, Apr 21

dduvall committed rGBLBR459234d2acb7: user: Check for existing user/group before creating (authored by dduvall).
user: Check for existing user/group before creating
Wed, Apr 21, 8:35 PM

Tue, Apr 20

dduvall updated subscribers of T268819: Blubber needs to check if a user is present before creating it as part of its runs stanza.
Tue, Apr 20, 4:27 PM · Release-Engineering-Team (Seen), serviceops, MW-on-K8s, Release Pipeline (Blubber)
dduvall claimed T268819: Blubber needs to check if a user is present before creating it as part of its runs stanza.

Picking this back up in the context of MW-on-K8s work. Basically we want the ability to build an image in one PipelineLib stage and use that as a base image in a subsequent stage. The unconditional attempt to create users for each subsequent use of a Blubber built image as a base currently makes this impossible.

Tue, Apr 20, 4:26 PM · Release-Engineering-Team (Seen), serviceops, MW-on-K8s, Release Pipeline (Blubber)
dduvall raised the priority of T268819: Blubber needs to check if a user is present before creating it as part of its runs stanza from Low to Medium.
Tue, Apr 20, 4:24 PM · Release-Engineering-Team (Seen), serviceops, MW-on-K8s, Release Pipeline (Blubber)

Apr 13 2021

dduvall closed T278993: Automate the merge of the weekly branch cut, a subtask of T196515: Automate the Train, as Resolved.
Apr 13 2021, 7:16 PM · Release-Engineering-Team (Seen), Epic, Goal, Scap
dduvall closed T278993: Automate the merge of the weekly branch cut as Resolved.
Apr 13 2021, 7:16 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (Deployment services)

Apr 8 2021

dduvall closed T278344: 1.36.0-wmf.38 deployment blockers as Resolved.
Apr 8 2021, 11:27 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments
dduvall added projects to T279711: WikiPage constructed on a Title that cannot exist as a page: Special:Watchlist [Called from Article::newPage]: VisualEditor-MediaWiki, MediaWiki-API.
Apr 8 2021, 7:45 PM · Skipped QA, MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Editing-team (FY2020-21 Kanban Board), VisualEditor, MediaWiki-API, VisualEditor-MediaWiki, Wikimedia-production-error
dduvall added a comment to T279711: WikiPage constructed on a Title that cannot exist as a page: Special:Watchlist [Called from Article::newPage].

Looks similar to T273101: Accessing WikiPage that cannot exist as a page: w:Help:Books/Book creator text. [Called from WikiPage::exists] though that was from WikiPage::exists and extensions/Collection. This is from VisualEditor/API and Article::newPage.

Apr 8 2021, 7:43 PM · Skipped QA, MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), Editing-team (FY2020-21 Kanban Board), VisualEditor, MediaWiki-API, VisualEditor-MediaWiki, Wikimedia-production-error

Apr 7 2021

dduvall claimed T278993: Automate the merge of the weekly branch cut.

I've updated the job on releases-jenkins. I guess we'll see if it works next week.

Apr 7 2021, 11:13 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (Deployment services)
dduvall removed a subtask for T278344: 1.36.0-wmf.38 deployment blockers: T279113: Unable to parse title at new Title in Special:MediaSearch.
Apr 7 2021, 11:06 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments
dduvall added a subtask for T278345: 1.37.0-wmf.1 deployment blockers: T279113: Unable to parse title at new Title in Special:MediaSearch.
Apr 7 2021, 11:06 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments
dduvall edited parent tasks for T279113: Unable to parse title at new Title in Special:MediaSearch, added: T278345: 1.37.0-wmf.1 deployment blockers; removed: T278344: 1.36.0-wmf.38 deployment blockers.
Apr 7 2021, 11:06 PM · MW-1.36-notes (1.36.0-wmf.38; 2021-04-06), Structured-Data-Backlog (Current Work), JavaScript, WikibaseMediaInfo, Wikimedia-production-error
dduvall lowered the priority of T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json from Unbreak Now! to Medium.
Apr 7 2021, 11:04 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall removed a parent task for T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json: T278344: 1.36.0-wmf.38 deployment blockers.
Apr 7 2021, 11:04 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall removed a subtask for T278344: 1.36.0-wmf.38 deployment blockers: T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json.
Apr 7 2021, 11:04 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments
Restricted Application added a project to T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json: wdwb-tech.

I have not looked at any of the code though, just guessing. But my guess is that this will only affect Wikidata, and should not block the train from rolling out to other wikis (but it may be a blocker for Wikidata?). The query also works on mediawiki.org.

Apr 7 2021, 9:36 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall added a project to T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json: Wikidata-Campsite.
Apr 7 2021, 9:33 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall raised the priority of T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json from High to Unbreak Now!.

Marking UBN as I plan on blocking until this is resolved.

Apr 7 2021, 8:59 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall moved T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json from Untriaged to Mar2021/1.36.0-wmf.33+ on the Wikimedia-production-error board.
Apr 7 2021, 8:47 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall added a project to T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json: Wikimedia-production-error.
Apr 7 2021, 8:47 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall triaged T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json as High priority.

Not quite UBN for now and I haven't rolled back on account of it, but this needs more eyeballs.

Apr 7 2021, 8:46 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall added a subtask for T278344: 1.36.0-wmf.38 deployment blockers: T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json.
Apr 7 2021, 8:45 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments
dduvall added a parent task for T279585: Error on wmf.38 when parsing edit summary via API: Content serialization failed: Failed to decode as application/json: T278344: 1.36.0-wmf.38 deployment blockers.
Apr 7 2021, 8:45 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-production-error, wdwb-tech, Wikidata, Regression, MediaWiki-API
dduvall added a comment to T278344: 1.36.0-wmf.38 deployment blockers.

Thanks for that, @Ladsgroup!

Apr 7 2021, 8:44 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release, Train Deployments

Apr 6 2021

dduvall added a comment to T278993: Automate the merge of the weekly branch cut.

If we're auto +2'ing, does the branch cut change still need to go through review? I ask because branch.py already has a --no-review option which could skip that process entirely.

It's possible that the branch would fail CI for some reason; I think we'd want to know that ahead of time, rather than when we make the first back-port. --no-review bypasses CI and just force-pushes, I think?

Apr 6 2021, 5:26 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (Deployment services)
dduvall added a comment to T278993: Automate the merge of the weekly branch cut.

If we're auto +2'ing, does the branch cut change still need to go through review? I ask because branch.py already has a --no-review option which could skip that process entirely.

Apr 6 2021, 5:18 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (Deployment services)
dduvall added a comment to T279322: Design m8s deployment workflows and tooling.

scap backport --revert change_id [change_id]

One quick point here, a change_id can refer to more than 1 change: https://gerrit.wikimedia.org/r/q/If04177f2a9f681002be5eafd97cbc5800aed2e27

I'm not sure if that's a problem or not, but something to be aware of in the design

Apr 6 2021, 4:48 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline
dduvall updated the task description for T279322: Design m8s deployment workflows and tooling.
Apr 6 2021, 4:46 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline

Apr 5 2021

mmodell awarded T279322: Design m8s deployment workflows and tooling a Love token.
Apr 5 2021, 7:39 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline
dduvall updated the task description for T279322: Design m8s deployment workflows and tooling.
Apr 5 2021, 7:30 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline
dduvall created T279340: Include MediaWiki testing in .pipeline/config.yaml execution graph.
Apr 5 2021, 7:22 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline
dduvall created T279324: Provide a development environment for testing m8s deployment tooling.
Apr 5 2021, 7:15 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline
dduvall updated subscribers of T279323: Profile single-version and multi-version image builds to identify bottlenecks.

@akosiaris I don't see hardware or VM specifications for releases1002 on wikitech. Do you know what our options are if we need to request more RAM or faster disks?

Apr 5 2021, 7:12 PM · Release-Engineering-Team (Doing), Release Pipeline
dduvall created T279323: Profile single-version and multi-version image builds to identify bottlenecks.
Apr 5 2021, 7:09 PM · Release-Engineering-Team (Doing), Release Pipeline
dduvall created T279322: Design m8s deployment workflows and tooling.
Apr 5 2021, 6:57 PM · Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline

Mar 31 2021

dduvall edited projects for T225335: Implement pipeline config validation, added: Release-Engineering-Team-TODO (Yak Shaving 🐃🪒 ); removed Release-Engineering-Team-TODO.

Reviving this as a Yak Shaving task involving 2-3 people over the course of a full day. Implementation idea:

Mar 31 2021, 5:59 PM · Release-Engineering-Team (Yak Shaving 🐃🪒), Release Pipeline
dduvall created T278988: Automate MediaWiki wmf branch image builds.
Mar 31 2021, 5:10 PM · Release-Engineering-Team (Doing), MW-on-K8s

Mar 30 2021

dduvall placed T267685: blubber: Execute python module as RUN command of dockerfile up for grabs.
Mar 30 2021, 8:10 PM · Release-Engineering-Team (Doing), Release Pipeline

Mar 25 2021

dduvall edited projects for T278492: Write PipelineLib user guide for setting up a GitOps-y workflow, added: Release-Engineering-Team; removed Release-Engineering-Team-TODO.
Mar 25 2021, 10:11 PM · Release-Engineering-Team (Next), Documentation, Release Pipeline
dduvall created T278492: Write PipelineLib user guide for setting up a GitOps-y workflow.
Mar 25 2021, 10:10 PM · Release-Engineering-Team (Next), Documentation, Release Pipeline
dduvall closed T277346: PipelineBuilder based jobs need to handle git tags and other ref updates as Resolved.
Mar 25 2021, 10:06 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline
dduvall closed T273676: Jenkins job exists for building patched MediaWiki image, a subtask of T271274: Security patch workflow for MediaWiki on k8s, as Resolved.
Mar 25 2021, 10:05 PM · Release-Engineering-Team (Doing), MW-on-K8s
dduvall closed T273676: Jenkins job exists for building patched MediaWiki image as Resolved.

The existing .pipeline/config.yaml was extended to patch the production image before publishing to the restricted repo. See T271274: Security patch workflow for MediaWiki on k8s and associated patchsets for implementation details.

Mar 25 2021, 10:05 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (Pipeline), MW-on-K8s

Mar 23 2021

dduvall updated the task description for T277777: Support copying files from run containers into the workspace for use in other stages.
Mar 23 2021, 5:30 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall triaged T277777: Support copying files from run containers into the workspace for use in other stages as Medium priority.
Mar 23 2021, 3:27 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline, Release-Engineering-Team (Pipeline)
dduvall claimed T277777: Support copying files from run containers into the workspace for use in other stages.
Mar 23 2021, 3:27 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline, Release-Engineering-Team (Pipeline)

Mar 18 2021

dduvall closed T277109: Containers on releases hosts cannot update apt cache from non-WMF sources as Resolved.
Mar 18 2021, 11:27 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall committed rGBLBR642f9d0a0113: apt: Implement merging `apt.proxies` config (authored by dduvall).
apt: Implement merging `apt.proxies` config
Mar 18 2021, 10:54 PM
dduvall committed rGBLBR7bb486915320: apt: Support configuration of http/https proxies (authored by dduvall).
apt: Support configuration of http/https proxies
Mar 18 2021, 6:43 PM
dduvall created T277777: Support copying files from run containers into the workspace for use in other stages.
Mar 18 2021, 4:45 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline, Release-Engineering-Team (Pipeline)

Mar 15 2021

dduvall closed T274182: Multi-version MediaWiki image is built and published, a subtask of T273676: Jenkins job exists for building patched MediaWiki image, as Resolved.
Mar 15 2021, 7:15 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (Pipeline), MW-on-K8s
dduvall closed T274182: Multi-version MediaWiki image is built and published as Resolved.
Mar 15 2021, 7:15 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (Pipeline), MW-on-K8s
dduvall removed a subtask for T274182: Multi-version MediaWiki image is built and published: T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.
Mar 15 2021, 7:15 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (Pipeline), MW-on-K8s
dduvall removed a parent task for T277109: Containers on releases hosts cannot update apt cache from non-WMF sources: T274182: Multi-version MediaWiki image is built and published.
Mar 15 2021, 7:15 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s

Mar 12 2021

dduvall claimed T277346: PipelineBuilder based jobs need to handle git tags and other ref updates.
Mar 12 2021, 10:51 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline
dduvall created T277346: PipelineBuilder based jobs need to handle git tags and other ref updates.
Mar 12 2021, 9:37 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release Pipeline
dduvall closed T273521: Create restricted docker-registry namespace for security patched images as Resolved.
Mar 12 2021, 5:37 PM · Patch-For-Review, serviceops, Release-Engineering-Team-TODO, Release Pipeline, MW-on-K8s
dduvall closed T273521: Create restricted docker-registry namespace for security patched images, a subtask of T271274: Security patch workflow for MediaWiki on k8s, as Resolved.
Mar 12 2021, 5:37 PM · Release-Engineering-Team (Doing), MW-on-K8s
dduvall added a comment to T273521: Create restricted docker-registry namespace for security patched images.

Success! I was able to build and push an image via docker-pusher on releases1002 to the restricted path (docker-registry.wikimedia.org/restricted/mediawiki-multiversion:2021-03-11-213947-production). Thanks for your hard work on this, @Legoktm

Mar 12 2021, 5:37 PM · Patch-For-Review, serviceops, Release-Engineering-Team-TODO, Release Pipeline, MW-on-K8s

Mar 11 2021

dduvall added a comment to T277236: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY'.

We hadn't yet merged a patchset to integration/config that was thought to be independent. However, the yaml and groovy job definition in integration/config is what defines the global whitelist which includes SONAR_API_KEY.

Mar 11 2021, 10:26 PM · Growth-Team, Release Pipeline, Add-Link, Continuous-Integration-Config
dduvall moved T277109: Containers on releases hosts cannot update apt cache from non-WMF sources from INBOX to New Work on the Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)) board.
Mar 11 2021, 7:02 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall claimed T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.
Mar 11 2021, 7:02 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall added a subtask for T274182: Multi-version MediaWiki image is built and published: T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.
Mar 11 2021, 7:01 PM · Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (Pipeline), MW-on-K8s
dduvall added a parent task for T277109: Containers on releases hosts cannot update apt cache from non-WMF sources: T274182: Multi-version MediaWiki image is built and published.
Mar 11 2021, 7:01 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall added a comment to T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.

Another option might just be to set an http_proxy environment variable in mediawiki-config's .pipeline/config.yaml. It seems apt-get will respect that for all of its http transfers.

Mar 11 2021, 6:57 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall updated subscribers of T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.

Another option might just be to set an http_proxy environment variable in mediawiki-config's .pipeline/config.yaml. It seems apt-get will respect that for all of its http transfers. @Legoktm does that seem reasonable?

Mar 11 2021, 6:55 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall added a project to T277109: Containers on releases hosts cannot update apt cache from non-WMF sources: Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).
Mar 11 2021, 12:11 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall added a comment to T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.

Notes from #mw-on-k8s for one possible solution.

Mar 11 2021, 12:11 AM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s

Mar 10 2021

dduvall created T277109: Containers on releases hosts cannot update apt cache from non-WMF sources.
Mar 10 2021, 11:06 PM · Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), serviceops, MW-on-K8s
dduvall closed T276869: Missing docker iptables nat rules for releases hosts as Resolved.

Yes! Thanks so much for the fix.

Mar 10 2021, 10:37 PM · serviceops, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), MW-on-K8s
dduvall added a comment to T276597: Blubber should fetch git submodules of its project if it has them.

Thanks, Dan; I think that PipelineLib's innate functionality works! Quick question: is there a way I can run PipelineLib locally the way I can run the Blubber binary? I'd like to be able to build a Docker image identical to the one that will be built in the pipeline.

Mar 10 2021, 5:38 PM · Release Pipeline (Blubber), Abstract Wikipedia team (Phase γ)
dduvall added a project to T277070: Beta cluster master switchover to deployment-db07: Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).
Mar 10 2021, 5:19 PM · User-zeljkofilipin, User-Majavah, User-Urbanecm, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Beta-Cluster-Infrastructure
dduvall added a comment to T276968: deployment-db05 needs replacing following disk corruption.

From @Marostegui in IRC: "To be honest, I would do it in different steps, set db06, make sure all is fine and the slave replicates just fine. And once that is fully ok, then proceed with marxarelli's plan"

Mar 10 2021, 4:46 PM · User-notice, DBA, Beta-Cluster-Infrastructure
dduvall added a comment to T276968: deployment-db05 needs replacing following disk corruption.

Forgot the UNLOCK TABLES on db07 :)

Mar 10 2021, 4:26 PM · User-notice, DBA, Beta-Cluster-Infrastructure
dduvall added a comment to T276968: deployment-db05 needs replacing following disk corruption.

Thanks for the help, everyone. I would still like to get off of db06 if possible at the end of this process since we have to finish the buster upgrade at some point anyhow. If we can get both db07 and db08 to reach the same point in the binlog from db06, can we simply:

Mar 10 2021, 4:25 PM · User-notice, DBA, Beta-Cluster-Infrastructure
dduvall claimed T276968: deployment-db05 needs replacing following disk corruption.

The current status of this is:

Mar 10 2021, 1:07 AM · User-notice, DBA, Beta-Cluster-Infrastructure
dduvall renamed T276968: deployment-db05 needs replacing following disk corruption from deployment-db05 disk issues to deployment-db05 needs replacing following disk corruption.
Mar 10 2021, 12:54 AM · User-notice, DBA, Beta-Cluster-Infrastructure

Mar 9 2021

dduvall updated the task description for T276869: Missing docker iptables nat rules for releases hosts.
Mar 9 2021, 12:03 AM · serviceops, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), MW-on-K8s
dduvall created T276869: Missing docker iptables nat rules for releases hosts.
Mar 9 2021, 12:02 AM · serviceops, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), MW-on-K8s

Mar 5 2021

dduvall added a comment to T276597: Blubber should fetch git submodules of its project if it has them.

Can you elaborate on your use case a bit? Blubber doesn't have any git-specific features currently (though you can run any ad hoc builder command you like). It sort of assumes the same context as does a docker build command, namely that you have some filesystem hierarchy already prepared locally and wish to copy files in and run commands against them to build artifacts and configure stuff.

Mar 5 2021, 6:08 PM · Release Pipeline (Blubber), Abstract Wikipedia team (Phase γ)

Mar 1 2021

dduvall added a comment to T273521: Create restricted docker-registry namespace for security patched images.

In other words, override the value of this secret for releases hosts with the new credentials.

Mar 1 2021, 5:45 PM · Patch-For-Review, serviceops, Release-Engineering-Team-TODO, Release Pipeline, MW-on-K8s
dduvall added a comment to T273521: Create restricted docker-registry namespace for security patched images.

The restricted/ namespace is now live. Where should I put the credentials for the ci-restricted user?

Mar 1 2021, 5:43 PM · Patch-For-Review, serviceops, Release-Engineering-Team-TODO, Release Pipeline, MW-on-K8s

Feb 25 2021

dduvall edited projects for T275826: L10n cache files building up on backup deploy hosts, added: Release-Engineering-Team; removed Release-Engineering-Team-TODO.
Feb 25 2021, 11:40 PM · Release-Engineering-Team (Doing), Deployments
dduvall created T275826: L10n cache files building up on backup deploy hosts.
Feb 25 2021, 11:39 PM · Release-Engineering-Team (Doing), Deployments

Feb 23 2021

dduvall committed rGBLBRdb3a5397f09a: copies: Allow copying directly from other images (authored by dduvall).
copies: Allow copying directly from other images
Feb 23 2021, 7:15 PM

Feb 19 2021

dduvall closed T271345: 1.36.0-wmf.31 deployment blockers as Resolved.
Feb 19 2021, 10:29 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall closed T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create as Resolved.
Feb 19 2021, 10:28 PM · MW-1.36-notes (1.36.0-wmf.31; 2021-02-16), Notifications, Growth-Team, Wikimedia-production-error
dduvall closed T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create, a subtask of T271345: 1.36.0-wmf.31 deployment blockers, as Resolved.
Feb 19 2021, 10:28 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall removed a subtask for T274936: 1.36.0-wmf.32 deployment blockers: T274589: No atomic section is open (got LocalFile::lockingTransaction) .
Feb 19 2021, 7:59 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall removed a parent task for T274589: No atomic section is open (got LocalFile::lockingTransaction) : T274936: 1.36.0-wmf.32 deployment blockers.
Feb 19 2021, 7:59 PM · Performance-Team (Radar), Wikimedia-production-error, Wikimedia-database-error, MediaWiki-File-management, Commons, Structured Data Engineering, Structured-Data-Backlog
dduvall removed a subtask for T271345: 1.36.0-wmf.31 deployment blockers: T274589: No atomic section is open (got LocalFile::lockingTransaction) .
Feb 19 2021, 7:50 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall added a subtask for T274936: 1.36.0-wmf.32 deployment blockers: T274589: No atomic section is open (got LocalFile::lockingTransaction) .
Feb 19 2021, 7:50 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall edited parent tasks for T274589: No atomic section is open (got LocalFile::lockingTransaction) , added: T274936: 1.36.0-wmf.32 deployment blockers; removed: T271345: 1.36.0-wmf.31 deployment blockers.
Feb 19 2021, 7:50 PM · Performance-Team (Radar), Wikimedia-production-error, Wikimedia-database-error, MediaWiki-File-management, Commons, Structured Data Engineering, Structured-Data-Backlog
dduvall reopened T262033: labswiki: Field 'notification_bundle_display_hash' doesn't have a default value (10.64.0.98) as "Open".

Reopening and categorizing as logspam since we're still seeing this albeit infrequently.

Feb 19 2021, 7:43 PM · DBA, Growth-Team, Notifications, Wikimedia-production-error

Feb 18 2021

dduvall triaged T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create as High priority.
Feb 18 2021, 9:47 PM · MW-1.36-notes (1.36.0-wmf.31; 2021-02-16), Notifications, Growth-Team, Wikimedia-production-error
dduvall added a subtask for T271345: 1.36.0-wmf.31 deployment blockers: T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create.
Feb 18 2021, 9:47 PM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
dduvall added a parent task for T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create: T271345: 1.36.0-wmf.31 deployment blockers.
Feb 18 2021, 9:47 PM · MW-1.36-notes (1.36.0-wmf.31; 2021-02-16), Notifications, Growth-Team, Wikimedia-production-error
dduvall created T275161: InvalidArgumentException: Invalid user parameter in EchoEvent::create.
Feb 18 2021, 9:19 PM · MW-1.36-notes (1.36.0-wmf.31; 2021-02-16), Notifications, Growth-Team, Wikimedia-production-error