Page MenuHomePhabricator

thcipriani (Tyler Cipriani)
¯\_(ツ)_/¯Administrator

Projects (23)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Feb 9 2015, 10:04 PM (268 w, 2 d)
Roles
Administrator
Availability
Available
IRC Nick
thcipriani
LDAP User
Unknown
MediaWiki User
TCipriani (WMF) [ Global Accounts ]

Recent Activity

Today

thcipriani added a comment to T247284: Improve OCR: Move ws-google-ocr repository to Gerrit.

@thcipriani - Any idea how to move a repo from Phabricator to Gerrit (without losing the history)? Or who would know?

Same way you would with any new git repo - add the new remote, and push everything to it ;)
With gerrit, you might need to enable force push/forge committer rights temporarily though

Thu, Apr 2, 3:46 PM · Community-Tech
thcipriani awarded T249225: beta-publish-deb fails to find a .deb artifact from parent job a Barnstar token.
Thu, Apr 2, 3:34 PM · Beta-Cluster-Infrastructure, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Scap, Release-Engineering-Team (CI & Testing services)

Yesterday

thcipriani triaged T248351: Investigate wikimedia.biterg.io as possible replacement for people.wikimedia.org/~dduvall/jenkins as Low priority.
Wed, Apr 1, 5:47 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), wikimedia.biterg.io
thcipriani triaged T248927: Blubber policy should be verified after expansion as Medium priority.
Wed, Apr 1, 5:45 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Pipeline), Release Pipeline (Blubber)
thcipriani moved T248927: Blubber policy should be verified after expansion from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Apr 1, 5:45 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Pipeline), Release Pipeline (Blubber)
thcipriani triaged T248916: Document how to migrate a service to kubernetes as Medium priority.
Wed, Apr 1, 5:36 PM · Kubernetes, Documentation, Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4))
thcipriani moved T248916: Document how to migrate a service to kubernetes from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Apr 1, 5:36 PM · Kubernetes, Documentation, Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4))
thcipriani placed T200739: Upgrade to Gerrit 2.16.13 up for grabs.
Wed, Apr 1, 5:25 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Epic, Developer Productivity, Release-Engineering-Team (Development services), Patch-For-Review, Gerrit
thcipriani moved T243808: gerrit1002 running out of space from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Apr 1, 5:25 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Operations, Gerrit
thcipriani triaged T244892: PipelineLib needs a timeout as Medium priority.
Wed, Apr 1, 5:22 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release Pipeline, Release-Engineering-Team (Pipeline)
thcipriani edited projects for T244892: PipelineLib needs a timeout, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO.
Wed, Apr 1, 5:22 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release Pipeline, Release-Engineering-Team (Pipeline)
thcipriani triaged T245187: Automate rebuild l10n cache for Train as Medium priority.
Wed, Apr 1, 5:20 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Scap
thcipriani triaged T245575: Dockerize our Phabricator development environment as Medium priority.
Wed, Apr 1, 5:20 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Development services), User-brennen, Phabricator
thcipriani edited projects for T245575: Dockerize our Phabricator development environment, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO.
Wed, Apr 1, 5:19 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Development services), User-brennen, Phabricator
thcipriani claimed T245691: Should 'doc' machines (i.e. doc1001) have contint-roots as a group?.
Wed, Apr 1, 5:19 PM · Operations, Release-Engineering-Team-TODO
thcipriani moved T246793: Scap no longer sets 'level' in Logstash messages from INBOX to Watching/External on the Release-Engineering-Team-TODO board.

this looks like it's likely the culprit: rMSCA3ca8820b7a68: Support for logging json to syslog

Wed, Apr 1, 5:17 PM · Developer Productivity, observability, Regression, Release-Engineering-Team-TODO, Scap
thcipriani moved T245658: .mp4 build artifacts not viewable due to CSP in Chrome from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Apr 1, 5:15 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Developer Productivity, ContentSecurityPolicy, Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure
thcipriani triaged T245658: .mp4 build artifacts not viewable due to CSP in Chrome as Medium priority.
Wed, Apr 1, 5:15 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Developer Productivity, ContentSecurityPolicy, Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure
thcipriani edited projects for T245658: .mp4 build artifacts not viewable due to CSP in Chrome, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO.
Wed, Apr 1, 5:14 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Developer Productivity, ContentSecurityPolicy, Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure
thcipriani added a project to T243808: gerrit1002 running out of space: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)).
Wed, Apr 1, 4:21 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Operations, Gerrit
thcipriani closed T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos as Resolved.

This stopped abruptly 21 days ago:

Wed, Apr 1, 3:29 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)

Tue, Mar 31

thcipriani added a comment to T248922: Requesting access to mwmaint1002.eqiad.wmnet for holger.

Either @thcipriani or SRE team would be the right people to own mwmaint servers, requesting his ok. Do you know if the requested access "deployment" is required to run maintenance scripts, or does "restricted" suffice for that?

Tue, Mar 31, 1:22 PM · Operations, SRE-Access-Requests

Mon, Mar 30

thcipriani added a comment to T248927: Blubber policy should be verified after expansion.

Tentatively assigning to dduvall per IRC discussion

Mon, Mar 30, 10:37 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Pipeline), Release Pipeline (Blubber)
thcipriani created T248927: Blubber policy should be verified after expansion.
Mon, Mar 30, 10:37 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release-Engineering-Team (Pipeline), Release Pipeline (Blubber)
thcipriani added a comment to T248708: Design and document a workflow for accepting patches for techblog.

Is option 1 the only option that won't require some additional code that glues the moving parts together? That is, is a merge into master of the origin repo a deployment without any additional glue needed?

Mon, Mar 30, 10:07 PM · Technical blog

Fri, Mar 27

thcipriani renamed T247776: 1.35.0-wmf.29 deployment blockers from 1.36.0-wmf.2 deployment blockers to 1.35.0-wmf.29 deployment blockers.
Fri, Mar 27, 4:17 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani renamed T247775: 1.35.0-wmf.28 deployment blockers from 1.36.0-wmf.1 deployment blockers to 1.35.0-wmf.28 deployment blockers.
Fri, Mar 27, 4:16 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments

Thu, Mar 26

thcipriani added a comment to T248597: Grant "contint-roots" and "releasers-mediawiki" to user jforrester.

approved.

Thu, Mar 26, 5:12 PM · Operations, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, SRE-Access-Requests
thcipriani awarded T248597: Grant "contint-roots" and "releasers-mediawiki" to user jforrester a Like token.
Thu, Mar 26, 4:38 PM · Operations, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, SRE-Access-Requests

Wed, Mar 25

thcipriani edited projects for T244515: Underrun and CI capacity, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)).
Wed, Mar 25, 5:31 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, Release-Engineering-Team (CI & Testing services), User-greg
thcipriani moved T248351: Investigate wikimedia.biterg.io as possible replacement for people.wikimedia.org/~dduvall/jenkins from INBOX to New Work on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 25, 5:29 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), wikimedia.biterg.io
thcipriani moved T247426: Investigate duration of scap sync for 1.35.0-wmf.23 train deploy from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 25, 5:27 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, Release-Engineering-Team (Deployment services), Scap
thcipriani assigned T247005: Scap's link to Logstash for canary failure is not in sync with the monitoring query to Jdforrester-WMF.
Wed, Mar 25, 5:23 PM · Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Developer Productivity, Scap
thcipriani moved T247005: Scap's link to Logstash for canary failure is not in sync with the monitoring query from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 25, 5:23 PM · Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Developer Productivity, Scap
thcipriani moved T235807: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Mar 25, 5:22 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Continuous-Integration-Config, BlueSpice
thcipriani edited projects for T235807: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)).
Wed, Mar 25, 5:22 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Continuous-Integration-Config, BlueSpice
thcipriani moved T246942: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image from INBOX to New Work on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Mar 25, 5:21 PM · MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, MediaWiki-Docker
thcipriani assigned T246942: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image to brennen.
Wed, Mar 25, 5:21 PM · MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, MediaWiki-Docker
thcipriani moved T246935: Job queue runners for MediaWiki-Docker from INBOX to New Work on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Mar 25, 5:20 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, dev-images, MediaWiki-Docker
thcipriani assigned T246935: Job queue runners for MediaWiki-Docker to brennen.
Wed, Mar 25, 5:20 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, dev-images, MediaWiki-Docker
thcipriani triaged T247153: ffmpeg in MW-Docker lacks -row-mt option for improved multithreaded VP9 encoding as Medium priority.
Wed, Mar 25, 5:19 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, dev-images, MediaWiki-Docker, Release-Engineering-Team (Local Dev)
thcipriani assigned T247153: ffmpeg in MW-Docker lacks -row-mt option for improved multithreaded VP9 encoding to brennen.
Wed, Mar 25, 5:19 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-brennen, dev-images, MediaWiki-Docker, Release-Engineering-Team (Local Dev)
thcipriani lowered the priority of T248040: operations/mediawiki-config master branch failing tests from High to Medium.

We have this switched off linting for the moment, planning on moving these plugins back into scap core Soon™. Lowering priority for the time being.

Wed, Mar 25, 5:17 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), ci-test-error, User-DannyS712
thcipriani triaged T246959: Running "scap pull" on deploy1001 halts for 2 minutes, then reports a php-fpm error as Medium priority.

FWIW, the pausing and the error are two different pieces. That is, the pause is cdb rebuild IIRC. The error is sudo differences in beta vs prod.

Wed, Mar 25, 5:01 PM · Release-Engineering-Team-TODO, Scap

Mon, Mar 23

thcipriani triaged T247427: Consider what paging who should get for what in RelEng/EngProd? as Medium priority.
Mon, Mar 23, 9:31 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4))
thcipriani added a comment to T216483: Move scap-vagrant to Gerrit.

@thcipriani It's been a year since the last ping :-) Do you object moving this to Gerrit? If not, which repo name should we use for this one? Thanks.

Mon, Mar 23, 8:57 PM · User-MarcoAurelio, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Phabricator, Gerrit
thcipriani closed T238261: 2019 Tech Conf Unconference: New CI/Argo as Resolved.

Thank you @Aklapper

Mon, Mar 23, 3:29 PM · Wikimedia-Technical-Conference-2019

Fri, Mar 20

thcipriani closed T245545: Gerrit account got messed up as Resolved.

Hi @Soda sorry this took so long :(

Fri, Mar 20, 6:45 PM · Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Gerrit

Wed, Mar 18

thcipriani added a comment to T245185: Automate scap patch.

Same as T247075?

Wed, Mar 18, 7:17 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, Scap
thcipriani moved T236675: Investigate Docker slowness between 18.06.2 and 18.09.7 from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Wed, Mar 18, 5:44 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
thcipriani edited projects for T236675: Investigate Docker slowness between 18.06.2 and 18.09.7, added: Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)).
Wed, Mar 18, 5:44 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
thcipriani moved T246973: CI / Zuul not processing changes from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 18, 5:43 PM · Zuul, Gerrit, User-DannyS712, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Continuous-Integration-Infrastructure
thcipriani moved T234902: quibble + local-charts? from INBOX to Later / Need volunteer on the Release-Engineering-Team-TODO board.
Wed, Mar 18, 5:41 PM · Release-Engineering-Team-TODO, Patch-For-Review, Quibble
thcipriani moved T244544: Add a quick rollback command to scap that skips all unnecessary steps from INBOX to Soon-ish on the Release-Engineering-Team-TODO board.
Wed, Mar 18, 5:33 PM · User-brennen, Release-Engineering-Team-TODO, Wikimedia-Incident, Scap
thcipriani triaged T247005: Scap's link to Logstash for canary failure is not in sync with the monitoring query as Medium priority.
Wed, Mar 18, 5:30 PM · Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Developer Productivity, Scap
thcipriani claimed T247107: Temporarily disable 'scap update-interwiki-cache'.
Wed, Mar 18, 5:28 PM · Scap

Tue, Mar 17

thcipriani added a comment to T237033: Scap can't clear opcache on mw servers in Beta Cluster.

This one:

Tue, Mar 17, 10:49 PM · Release-Engineering-Team (Deployment services), serviceops, Operations, Beta-Cluster-Infrastructure, Scap

Mon, Mar 16

thcipriani added a comment to T233872: 1.35.0-wmf.24 deployment blockers.

Added wmf.23 as a blocker for wmf.24. If wmf.23 isn't fully deployed by the wmf.24 Tuesday train window then we should go ahead and cut the branch for wmf.24, but NOT deploy it to any wikis. This unblocks folks waiting to merge to master until after wmf.24 is cut, but spares us the confusion of 3 branches in production which I definitely want to avoid.

Mon, Mar 16, 11:07 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Release, Train Deployments
thcipriani added a parent task for T233871: 1.35.0-wmf.23 deployment blockers: T233872: 1.35.0-wmf.24 deployment blockers.
Mon, Mar 16, 11:06 PM · User-brennen, Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Release, Train Deployments
thcipriani added a subtask for T233872: 1.35.0-wmf.24 deployment blockers: T233871: 1.35.0-wmf.23 deployment blockers.
Mon, Mar 16, 11:06 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), Release, Train Deployments
thcipriani added a comment to T247562: Warning: Memcached::setMulti(): failed to set key global:segment:....

Also the things I see in the changelog that mention "cache" are...many things: https://www.mediawiki.org/wiki/MediaWiki_1.35/wmf.23/Changelog

Mon, Mar 16, 8:51 PM · User-notice, MW-1.35-notes (1.35.0-wmf.24; 2020-03-17), Core Platform Team Workboards (Clinic Duty Team), MediaWiki-Cache, User-brennen, Wikimedia-production-error
thcipriani added a comment to T247562: Warning: Memcached::setMulti(): failed to set key global:segment:....

I'm trying to figure out if this is an actual code change or a memcached server problem with bad timing. Or maybe it's a code change that causes memcached to have a bad time? Does anyone have a theory there?

Mon, Mar 16, 8:43 PM · User-notice, MW-1.35-notes (1.35.0-wmf.24; 2020-03-17), Core Platform Team Workboards (Clinic Duty Team), MediaWiki-Cache, User-brennen, Wikimedia-production-error
thcipriani moved T247776: 1.35.0-wmf.29 deployment blockers from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Mon, Mar 16, 4:53 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani assigned T247776: 1.35.0-wmf.29 deployment blockers to LarsWirzenius.
Mon, Mar 16, 4:53 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani moved T247775: 1.35.0-wmf.28 deployment blockers from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Mon, Mar 16, 4:52 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani assigned T247775: 1.35.0-wmf.28 deployment blockers to Jdforrester-WMF.
Mon, Mar 16, 4:52 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani moved T247773: 1.35.0-wmf.26 deployment blockers from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Mon, Mar 16, 4:51 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani moved T247774: 1.35.0-wmf.27 deployment blockers from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
Mon, Mar 16, 4:50 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani assigned T247774: 1.35.0-wmf.27 deployment blockers to jeena.
Mon, Mar 16, 4:50 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani assigned T247773: 1.35.0-wmf.26 deployment blockers to dduvall.
Mon, Mar 16, 4:49 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani created T247776: 1.35.0-wmf.29 deployment blockers.
Mon, Mar 16, 4:48 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani created T247775: 1.35.0-wmf.28 deployment blockers.
Mon, Mar 16, 4:48 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani created T247774: 1.35.0-wmf.27 deployment blockers.
Mon, Mar 16, 4:44 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments
thcipriani created T247773: 1.35.0-wmf.26 deployment blockers.
Mon, Mar 16, 4:44 PM · Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release, Train Deployments

Fri, Mar 13

thcipriani added a comment to T236104: Cache of wmf-config/InitialiseSettings often 1 step behind.

FWIW, ran some tests with apachebench today and the patchset on this task and it didn't seem to make requests significantly slower or load significantly higher in my rudimentary testing:

Fri, Mar 13, 6:50 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Patch-For-Review, Release-Engineering-Team (Deployment services), Wikimedia-Incident, Deployments, Performance-Team (Radar), serviceops
thcipriani awarded T247320: Order of magnitude for New CI hosting budget a Barnstar token.
Fri, Mar 13, 4:30 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3))

Thu, Mar 12

thcipriani closed T242237: Releng March priorities vs resources Chart as Resolved.

Updated chart, calling this complete.

Thu, Mar 12, 3:57 PM · Release-Engineering-Team (Other / Uncategorized), Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3))
thcipriani added a comment to T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos.

This hasn't happened today afaict:

Thu, Mar 12, 3:17 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)

Wed, Mar 11

thcipriani added a comment to T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos.
Wed, Mar 11, 9:08 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani claimed T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos.
Wed, Mar 11, 5:00 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani added a comment to T242237: Releng March priorities vs resources Chart.

Updated post team meeting with actual blockers.

Wed, Mar 11, 4:58 PM · Release-Engineering-Team (Other / Uncategorized), Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3))

Tue, Mar 10

thcipriani lowered the priority of T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos from Unbreak Now! to High.

Lowering priority since this hasn't happened once in the hour I've spent investigating according to the access logs.

Tue, Mar 10, 6:56 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani added a comment to T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos.

Looking specifically for upload-pack errors that happened over https in the past 30 days (this is what zuul would be using)

Tue, Mar 10, 5:54 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani added a comment to T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos.

Strangely, I'm not seeing the graphs in gerrit monitoring reporting a rise in error rate (this counts 4xx as well as 5xx so maybe it's lost in the noise). Logs are also not showing a massive increase in the error rate @Lucas_Werkmeister_WMDE did you run into this on one repo in particular? One job in particular?

Tue, Mar 10, 5:29 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani moved T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Tue, Mar 10, 5:24 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani edited projects for T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos, added: Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)); removed Release-Engineering-Team-TODO.
Tue, Mar 10, 5:24 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit, Wikimedia-production-error (Shared Build Failure)
thcipriani created T247320: Order of magnitude for New CI hosting budget.
Tue, Mar 10, 4:04 PM · Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3))

Sun, Mar 8

thcipriani closed T247182: gerrit-replica seems down as Resolved.

Back after restart. I didn't get an alert aside from this task. Filed as T247186

Sun, Mar 8, 5:22 AM · Gerrit
thcipriani triaged T247186: check gerrit-replica healthcheck alerts as High priority.
Sun, Mar 8, 5:21 AM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit
thcipriani created T247186: check gerrit-replica healthcheck alerts.
Sun, Mar 8, 5:21 AM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Gerrit

Thu, Mar 5

thcipriani created E1145: thcipriani out-of-office.
Thu, Mar 5, 3:05 PM · Release-Engineering-Team, events

Wed, Mar 4

thcipriani moved T246111: Command-line wrapper for interacting with core's docker-compose stack from INBOX to New Work on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 4, 6:58 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), User-zeljkofilipin, User-kostajh, Release-Engineering-Team, Developer Productivity, User-brennen, MediaWiki-Docker
thcipriani moved T246358: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 4, 6:54 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Quibble, Continuous-Integration-Config, Regression
thcipriani moved T246700: Implement golang directives for Blubber from INBOX to New Work on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 4, 6:52 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release Pipeline (Blubber)
thcipriani triaged T246700: Implement golang directives for Blubber as Medium priority.
Wed, Mar 4, 6:52 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Release Pipeline (Blubber)
thcipriani triaged T246903: Investigate Zuul crashing due to BlueSpice REL1_31 backlog as Medium priority.
Wed, Mar 4, 6:51 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), BlueSpice, Continuous-Integration-Infrastructure
thcipriani moved T246903: Investigate Zuul crashing due to BlueSpice REL1_31 backlog from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.
Wed, Mar 4, 6:50 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), BlueSpice, Continuous-Integration-Infrastructure
thcipriani added a project to T245614: Add an integration smoke test to scap: Scap.
Wed, Mar 4, 6:38 PM · Scap, Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3))
thcipriani added a project to T246025: Port scap to Python 3: Scap.
Wed, Mar 4, 6:38 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Python3-Porting, Scap
thcipriani added a project to T246455: Make scap release with --canary-wait-time and integration test changes: Scap.
Wed, Mar 4, 6:37 PM · Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Scap