dduvall (Dan Duvall)
Automation Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:24 PM (137 w, 2 d)
Availability
Available
IRC Nick
marxarelli
LDAP User
Dduvall
MediaWiki User
DDuvall (WMF)

Recent Activity

Yesterday

dduvall updated the task description for T166353: Fix Blubber variant expansion for boolean/int config properties.
Thu, May 25, 11:49 PM · Release Pipeline, Release-Engineering-Team (Kanban)
dduvall created T166353: Fix Blubber variant expansion for boolean/int config properties.
Thu, May 25, 11:47 PM · Release Pipeline, Release-Engineering-Team (Kanban)

Tue, May 23

dduvall committed rGBLBRd0266214e0da: Use Phabricator import URIs (authored by dduvall).
Use Phabricator import URIs
Tue, May 23, 9:14 PM
dduvall closed D657: Use Phabricator import URIs by committing rGBLBRd0266214e0da: Use Phabricator import URIs.
Tue, May 23, 9:14 PM · Release-Engineering-Team
dduvall updated the diff for D657: Use Phabricator import URIs.

Fixed docker/compiler import paths

Tue, May 23, 6:57 PM · Release-Engineering-Team
dduvall created D657: Use Phabricator import URIs.
Tue, May 23, 6:19 PM · Release-Engineering-Team

Thu, May 11

dduvall claimed T164964: Move blubber to phabricator repo.

Hmm... now to wrestle with import paths.

Thu, May 11, 6:21 PM · Release-Engineering-Team (Kanban), Release Pipeline
dduvall updated subscribers of T164965: ci-staging registry base image for mathoid PoC.

@mobrovac and I discussed Blubber requirements today regarding local development use cases. Among other things, we discussed the need for a Node base image with Wikimedia APT sources included and nodejs and related packages installed.

Thu, May 11, 5:41 PM · User-mobrovac, Release Pipeline

Apr 20 2017

dduvall added a comment to T158812: Gather requirements for build manifest specification.

Over the past few days I've been experimenting with a general build configuration format and wrote a build tool in Go—because, well, I unabashedly wanted to learn something new while experimenting; please don't judge my awful Go code :)—codenamed Blubber that spits out a Dockerfile.

Apr 20 2017, 6:39 PM · User-Joe, Services (designing), User-mobrovac, Release Pipeline

Apr 19 2017

dduvall added a comment to P5294 output of go build && ./blubber blubber.example.json production.

Result of:

Apr 19 2017, 6:52 PM
dduvall updated the title for P5294 output of go build && ./blubber blubber.example.json production from Masterwork From Distant Lands to output of go build && ./blubber blubber.example.json production.
Apr 19 2017, 6:50 PM
dduvall edited P5294 output of go build && ./blubber blubber.example.json production.
Apr 19 2017, 6:48 PM

Apr 17 2017

dduvall created T163159: Archive Malu project and repo.
Apr 17 2017, 9:48 PM · User-zeljkofilipin, Malu

Mar 29 2017

dduvall added a comment to T161659: Port existing `differential-docker-test` job to Jenkinsfile/Groovy.

Looks like there's a Docker Pipeline plugin that might provide what we need for this.

Mar 29 2017, 10:28 PM · Release Pipeline
dduvall added a comment to T161657: Set up Docker registry in ci-staging.

There's now a ci-staging-docker-registry instance running in ci-staging with the new role::ci::docker::registry applied. I've verified that it's working by pulling down the latest Debian image from dockerhub and pushing it to ci-staging-docker-registry.ci-staging.eqiad.wmflabs:

Mar 29 2017, 9:08 PM · Patch-For-Review, Release Pipeline
dduvall lowered the priority of T158812: Gather requirements for build manifest specification from Normal to Low.
Mar 29 2017, 7:42 PM · User-Joe, Services (designing), User-mobrovac, Release Pipeline
dduvall triaged T161661: Install Jenkins Pipeline Plugin for ci-staging and verify its basic functionality as Normal priority.
Mar 29 2017, 7:42 PM · Release Pipeline
dduvall moved T161661: Install Jenkins Pipeline Plugin for ci-staging and verify its basic functionality from Backlog to Build PoC on the Release Pipeline board.
Mar 29 2017, 7:42 PM · Release Pipeline

Mar 28 2017

dduvall added a parent task for T161659: Port existing `differential-docker-test` job to Jenkinsfile/Groovy: T161661: Install Jenkins Pipeline Plugin for ci-staging and verify its basic functionality.
Mar 28 2017, 11:08 PM · Release Pipeline
dduvall added a subtask for T161661: Install Jenkins Pipeline Plugin for ci-staging and verify its basic functionality: T161659: Port existing `differential-docker-test` job to Jenkinsfile/Groovy.
Mar 28 2017, 11:08 PM · Release Pipeline
dduvall created T161661: Install Jenkins Pipeline Plugin for ci-staging and verify its basic functionality.
Mar 28 2017, 11:08 PM · Release Pipeline
dduvall claimed T161657: Set up Docker registry in ci-staging.
Mar 28 2017, 11:02 PM · Patch-For-Review, Release Pipeline
dduvall moved T161657: Set up Docker registry in ci-staging from Backlog to Build PoC on the Release Pipeline board.
Mar 28 2017, 11:02 PM · Patch-For-Review, Release Pipeline
dduvall moved T159864: Build testing k8s setup in ci-staging from Build PoC to Backlog on the Release Pipeline board.
Mar 28 2017, 11:02 PM · Release-Engineering-Team (Kanban), Patch-For-Review, Release Pipeline
dduvall closed T153363: Spike: Evaluate containerized CI builds using Kubernetes as Declined.

This idea is essentially superseded by the Release Pipeline which will undoubtedly incorporate or interact with a k8s cluster in one way or another.

Mar 28 2017, 11:01 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall created T161659: Port existing `differential-docker-test` job to Jenkinsfile/Groovy.
Mar 28 2017, 10:57 PM · Release Pipeline
dduvall created T161657: Set up Docker registry in ci-staging.
Mar 28 2017, 10:47 PM · Patch-For-Review, Release Pipeline
dduvall added a comment to T159864: Build testing k8s setup in ci-staging.

Also needed add an annotation to the default namespace to satisfy our custom RegistryEnforcer admin controller:

Mar 28 2017, 6:49 PM · Release-Engineering-Team (Kanban), Patch-For-Review, Release Pipeline
dduvall added a comment to T159864: Build testing k8s setup in ci-staging.

So I think I got most of what was required for setup into the submitted puppet patch. The only other manual steps include:

Mar 28 2017, 6:43 PM · Release-Engineering-Team (Kanban), Patch-For-Review, Release Pipeline

Mar 16 2017

dduvall added a comment to T158812: Gather requirements for build manifest specification.

Also, it's probably worth pointing out at this point that 'manifest' is not really the right word for what we're talking about here; a manifest as it's defined in all of the existing specifications is actually a bundled filesystem image (agnostic to how it was built) and runtime metadata about how to set up the container and how to invoke some application binary. What we've been discussing is a slightly higher level abstraction that covers the metadata but also system level package dependencies. The manifest + image will be a product of our build step.

Mar 16 2017, 5:34 PM · User-Joe, Services (designing), User-mobrovac, Release Pipeline
dduvall added a comment to T158812: Gather requirements for build manifest specification.

Some related reading material on container standards (both de facto and burgeoning) that illustrates their separation of concerns and overlap.

Mar 16 2017, 5:10 PM · User-Joe, Services (designing), User-mobrovac, Release Pipeline
dduvall triaged T159864: Build testing k8s setup in ci-staging as Normal priority.
Mar 16 2017, 5:07 PM · Release-Engineering-Team (Kanban), Patch-For-Review, Release Pipeline
dduvall moved T159864: Build testing k8s setup in ci-staging from Backlog to Build PoC on the Release Pipeline board.
Mar 16 2017, 5:07 PM · Release-Engineering-Team (Kanban), Patch-For-Review, Release Pipeline

Feb 22 2017

dduvall created T158812: Gather requirements for build manifest specification.
Feb 22 2017, 9:40 PM · User-Joe, Services (designing), User-mobrovac, Release Pipeline

Feb 16 2017

dduvall abandoned D418: Use USER environment variable for default ssh user.
Feb 16 2017, 10:21 PM · Release-Engineering-Team

Feb 14 2017

dduvall closed D558: Fix `failure_rate` percentages by committing rMSCA292d89e6da9a: Fix `failure_rate` percentages.
Feb 14 2017, 12:41 AM · Release-Engineering-Team
dduvall committed rMSCA292d89e6da9a: Fix `failure_rate` percentages (authored by dduvall).
Fix `failure_rate` percentages
Feb 14 2017, 12:41 AM
dduvall committed rMSCAcc58ae9f2891: Exclude empty deploy groups (authored by dduvall).
Exclude empty deploy groups
Feb 14 2017, 12:41 AM
dduvall closed T157136: Error after "Finished deploy": <ValueError> xrange() arg 3 must not be zero as Resolved by committing rMSCAcc58ae9f2891: Exclude empty deploy groups.
Feb 14 2017, 12:41 AM · Scoring-platform-team, ORES, Deployment-Systems, Release-Engineering-Team, scap2, Scap
dduvall closed D566: Exclude empty deploy groups by committing rMSCAcc58ae9f2891: Exclude empty deploy groups.
Feb 14 2017, 12:41 AM · Release-Engineering-Team

Feb 13 2017

dduvall added a comment to D561: Adds check to prevent xrange related crash. Fixes T157136.

@Halfak: The Deployment Cabal discussed this a bit more in our meeting this morning and decided that disallowing/excluding empty groups may indeed be a better long-term fix since empty input could lead to further unexpected behavior if left unvalidated.

Feb 13 2017, 9:30 PM · Release-Engineering-Team
dduvall created D566: Exclude empty deploy groups.
Feb 13 2017, 9:09 PM · Release-Engineering-Team
dduvall added a revision to T157136: Error after "Finished deploy": <ValueError> xrange() arg 3 must not be zero: D566: Exclude empty deploy groups.
Feb 13 2017, 9:09 PM · Scoring-platform-team, ORES, Deployment-Systems, Release-Engineering-Team, scap2, Scap

Feb 9 2017

dduvall resigned from D561: Adds check to prevent xrange related crash. Fixes T157136.
Feb 9 2017, 8:45 PM · Release-Engineering-Team
dduvall added a comment to D561: Adds check to prevent xrange related crash. Fixes T157136.
In D561#11106, @dduvall wrote:
In D561#11102, @demon wrote:
In D561#11085, @Halfak wrote:

Not sure if we'd prefer this solution or to use max(self.size, 1) and skip the conditional.

I think this will be better, because it means we'll be doing an extra useless stage that we've already done. I'd like @thcipriani's input here though.

Perhaps put that in the constructor. A non-zero group size may cause other issues down the line and it would be best to sanitize the input early.

Feb 9 2017, 8:23 PM · Release-Engineering-Team
dduvall added a comment to D561: Adds check to prevent xrange related crash. Fixes T157136.
In D561#11102, @demon wrote:
In D561#11085, @Halfak wrote:

Not sure if we'd prefer this solution or to use max(self.size, 1) and skip the conditional.

I think this will be better, because it means we'll be doing an extra useless stage that we've already done. I'd like @thcipriani's input here though.

Feb 9 2017, 8:04 PM · Release-Engineering-Team
dduvall added a task to D561: Adds check to prevent xrange related crash. Fixes T157136: T123: Turn on "diffusion.allow-http-auth".

Oh and one little nit about the commit message: Please use the "Refs T123" or "Fixes T123" auto-close syntax to reference tasks. See https://secure.phabricator.com/book/phabricator/article/diffusion_autoclose/

Feb 9 2017, 8:02 PM · Release-Engineering-Team
dduvall added a revision to T123: Turn on "diffusion.allow-http-auth": D561: Adds check to prevent xrange related crash. Fixes T157136.
Feb 9 2017, 8:02 PM · Gerrit-Migration, Wikimedia Phabricator RfC
dduvall requested changes to D561: Adds check to prevent xrange related crash. Fixes T157136.

To clarify: I think your changes are beneficial and should stay as well. :) Can you please add a unit test to tests/scap/test_targets.py to verify the edge case? Thanks!

Feb 9 2017, 7:54 PM · Release-Engineering-Team
dduvall added a comment to D561: Adds check to prevent xrange related crash. Fixes T157136.

I think the correct fix here would be to ensure a non-zero group size in the first place. Any idea how that may have happened? I suspect a percentage failure_rate that was converted to an integer and floored to 0 perhaps...

Feb 9 2017, 7:52 PM · Release-Engineering-Team

Feb 8 2017

dduvall created D558: Fix `failure_rate` percentages.
Feb 8 2017, 9:06 PM · Release-Engineering-Team
dduvall committed rMSCAbf6443a7ba1b: Fix regression of deploy group continue prompt (authored by dduvall).
Fix regression of deploy group continue prompt
Feb 8 2017, 8:41 PM
dduvall closed T156839: Saying yes (y) continues to all groups as Resolved by committing rMSCAbf6443a7ba1b: Fix regression of deploy group continue prompt.
Feb 8 2017, 8:41 PM · Scap, Parsoid
dduvall closed D555: Fix regression of deploy group continue prompt by committing rMSCAbf6443a7ba1b: Fix regression of deploy group continue prompt.
Feb 8 2017, 8:40 PM · Release-Engineering-Team
dduvall added a comment to D555: Fix regression of deploy group continue prompt.
  • I can't remember what we decided about failure percentage for groups where group_size is defined. In D490#10152 you said:
In D490#10152, @dduvall wrote:
  • Stage execution methods now considers the failure_limit for the entire original group, not individual subgroups

but that doesn't seem to be the case when you use a string like '50%' because of the line: https://github.com/wikimedia/scap/blob/master/scap/targets.py#L257

I may file a task for that 2nd one...

Feb 8 2017, 8:40 PM · Release-Engineering-Team

Feb 7 2017

dduvall updated the diff for D555: Fix regression of deploy group continue prompt.

An additional follow up to make the prompt conditional more clear

Feb 7 2017, 7:29 PM · Release-Engineering-Team
dduvall updated the diff for D555: Fix regression of deploy group continue prompt.

Implemented suggested re-prompting following bad prompt input

Feb 7 2017, 7:19 PM · Release-Engineering-Team
dduvall created D555: Fix regression of deploy group continue prompt.
Feb 7 2017, 2:23 AM · Release-Engineering-Team
dduvall added a revision to T156839: Saying yes (y) continues to all groups: D555: Fix regression of deploy group continue prompt.
Feb 7 2017, 2:23 AM · Scap, Parsoid

Feb 6 2017

dduvall added a comment to T156839: Saying yes (y) continues to all groups.

Looks like I caused the regression rMSCA98247477db5d: Improve failure handling and rollback behavior and called it out in the commit message as a "behavior change". :)

Feb 6 2017, 7:42 PM · Scap, Parsoid
dduvall claimed T156839: Saying yes (y) continues to all groups.
Feb 6 2017, 6:15 PM · Scap, Parsoid

Jan 25 2017

dduvall edited P4809 Masterwork From Distant Lands.
Jan 25 2017, 11:24 PM

Jan 19 2017

dduvall archived P4770 Masterwork From Distant Lands.
Jan 19 2017, 11:38 PM
dduvall edited P4770 Masterwork From Distant Lands.
Jan 19 2017, 11:38 PM

Jan 10 2017

dduvall requested changes to D517: Unconditionally provision MW for testing.

Your changes look good but provision fails on account of checkoutMediaWiki not existing any longer:

Jan 10 2017, 6:43 PM · Release-Engineering-Team

Jan 4 2017

dduvall added a comment to T154612: Add DEPLOY_DIR env var to scap command checks.

Should we just make it the cwd during check execution?

Jan 4 2017, 9:05 PM · Scap

Dec 23 2016

dduvall committed rMSCA98247477db5d: Improve failure handling and rollback behavior (authored by dduvall).
Improve failure handling and rollback behavior
Dec 23 2016, 1:17 AM
dduvall closed T145512: Allow failures for a percentage of targets as Resolved by committing rMSCA98247477db5d: Improve failure handling and rollback behavior.
Dec 23 2016, 1:17 AM · User-mobrovac, Parsoid, Services, Scap
dduvall closed T145460: Rollback failed when target is down as Resolved by committing rMSCA98247477db5d: Improve failure handling and rollback behavior.
Dec 23 2016, 1:17 AM · Scap, Parsoid
dduvall closed D490: Improve failure handling and rollback behavior by committing rMSCA98247477db5d: Improve failure handling and rollback behavior.
Dec 23 2016, 1:17 AM · Release-Engineering-Team
dduvall closed T149008: Canary doesn't rollback if you don't continue as Resolved by committing rMSCA98247477db5d: Improve failure handling and rollback behavior.
Dec 23 2016, 1:17 AM · Scap, Parsoid
dduvall updated the diff for D490: Improve failure handling and rollback behavior.

Updating Diff summary to be consistent with most recent commit message. I'm probably doing this wrong! So weird.

Dec 23 2016, 1:14 AM · Release-Engineering-Team
dduvall added inline comments to D490: Improve failure handling and rollback behavior.
Dec 23 2016, 1:11 AM · Release-Engineering-Team
dduvall updated the diff for D490: Improve failure handling and rollback behavior.

Replace use of utils.ask with utils.confirm

Dec 23 2016, 1:08 AM · Release-Engineering-Team
dduvall committed rMSCAec8699367233: Rename test directory so nosetests will read it (authored by dduvall).
Rename test directory so nosetests will read it
Dec 23 2016, 12:56 AM
dduvall closed D523: Rename test directory so nosetests will read it by committing rMSCAec8699367233: Rename test directory so nosetests will read it.
Dec 23 2016, 12:56 AM · Release-Engineering-Team
dduvall updated the diff for D523: Rename test directory so nosetests will read it.

Edited Diff summary since it's not done automatically when you amend a commit message. :/

Dec 23 2016, 12:53 AM · Release-Engineering-Team

Dec 22 2016

dduvall committed rMSCA524ce660d613: Use `pip wheel` to manage CI pip cache (authored by dduvall).
Use `pip wheel` to manage CI pip cache
Dec 22 2016, 6:24 PM
dduvall closed D508: Use `pip wheel` to manage CI pip cache by committing rMSCA524ce660d613: Use `pip wheel` to manage CI pip cache.
Dec 22 2016, 6:23 PM · Release-Engineering-Team
dduvall updated the diff for D523: Rename test directory so nosetests will read it.

Implemented @hashar's recommended solution which is to simply add __init__.py files.

Dec 22 2016, 1:07 AM · Release-Engineering-Team
dduvall added a comment to D523: Rename test directory so nosetests will read it.
In D523#10364, @hashar wrote:

Thanks for the split! So the reason the tests under tests/scap is that it the paths are not python modules (they lack a __init__.py file).

TLDR:

touch tests/__init__.py
touch tests/scap/__init__.py`
Dec 22 2016, 12:58 AM · Release-Engineering-Team

Dec 21 2016

dduvall retitled D523: Rename test directory so nosetests will read it from to Rename test directory so nosetests will read it.
Dec 21 2016, 11:56 PM · Release-Engineering-Team
dduvall updated the diff for D508: Use `pip wheel` to manage CI pip cache.

Moving commit for rename of test directory to a separate Diff.

Dec 21 2016, 11:28 PM · Release-Engineering-Team
dduvall added a comment to D508: Use `pip wheel` to manage CI pip cache.
In D508#10330, @hashar wrote:

The rename of tests/scap to tests/scap_test is unrelated to the wheel/cache management. I would rather see that in a different differential. I am also wondering why nosetests would not find the tests, the default discovery filter is (?:^|[\b_\./-])[Tt]est which would match the test_*.py files

Dec 21 2016, 11:21 PM · Release-Engineering-Team

Dec 20 2016

dduvall updated the diff for D490: Improve failure handling and rollback behavior.

Yet another refactoring, simplification of rollback invocation using existing _execute_for_group method, and moved finalize stage to execute following primary stages for all groups which ensures rollback can actually function as expected.

Dec 20 2016, 10:53 PM · Release-Engineering-Team

Dec 15 2016

hashar awarded T153363: Spike: Evaluate containerized CI builds using Kubernetes a Barnstar token.
Dec 15 2016, 10:55 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall updated the task description for T153363: Spike: Evaluate containerized CI builds using Kubernetes.
Dec 15 2016, 10:00 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall updated subscribers of T153363: Spike: Evaluate containerized CI builds using Kubernetes.
Dec 15 2016, 9:59 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall updated the task description for T153363: Spike: Evaluate containerized CI builds using Kubernetes.
Dec 15 2016, 9:56 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall updated the task description for T153363: Spike: Evaluate containerized CI builds using Kubernetes.
Dec 15 2016, 9:53 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall created T153363: Spike: Evaluate containerized CI builds using Kubernetes.
Dec 15 2016, 9:47 PM · Release Pipeline, Release-Engineering-Team, Continuous-Integration-Infrastructure
dduvall updated subscribers of T150501: Spike: Evaluate experimental Docker based CI w/ scap builds.

From our notes for the 12/8 meeting with Ops regarding the direction of CI:

Dec 15 2016, 8:47 PM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
dduvall closed T150504: Define generic job that runs unit tests within a Docker container as Resolved.
Dec 15 2016, 8:45 PM · Continuous-Integration-Infrastructure
dduvall closed T150504: Define generic job that runs unit tests within a Docker container, a subtask of T150501: Spike: Evaluate experimental Docker based CI w/ scap builds, as Resolved.
Dec 15 2016, 8:45 PM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
dduvall requested review of D490: Improve failure handling and rollback behavior.

Requesting re-review after some minor changes.

Dec 15 2016, 7:11 PM · Release-Engineering-Team
dduvall updated the diff for D490: Improve failure handling and rollback behavior.

Fixed a small bug in the rollback message for when failure_limit is exceeded

Dec 15 2016, 12:31 AM · Release-Engineering-Team

Dec 14 2016

mmodell awarded D490: Improve failure handling and rollback behavior a Cookie token.
Dec 14 2016, 11:26 PM · Release-Engineering-Team
dduvall accepted D513: Use python2/3 octals, not just python2.
Dec 14 2016, 11:16 PM · Release-Engineering-Team
dduvall updated the diff for D490: Improve failure handling and rollback behavior.
  • Refactored targets.TargetsList to no longer split groups but return first-class instances of a new targets.DeployGroup class
  • Stage execution methods now considers the failure_limit for the entire original group, not individual subgroups
  • Some additional behavior changed slightly as a result of this refactoring (see the commit message)
Dec 14 2016, 11:11 PM · Release-Engineering-Team
dduvall accepted D511: Fix sha1 regex.
Dec 14 2016, 8:28 PM · Release-Engineering-Team