Page MenuHomePhabricator

thcipriani (Tyler Cipriani)
¯\_(ツ)_/¯Administrator

Projects (19)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Feb 9 2015, 10:04 PM (248 w, 2 d)
Roles
Administrator
Availability
Available
IRC Nick
thcipriani
LDAP User
Unknown
MediaWiki User
TCipriani (WMF) [ Global Accounts ]

Recent Activity

Yesterday

thcipriani added a comment to T234657: Wikimedia Technical Conference 2019 Session: Platform Stewardship.

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Wed, Nov 13, 10:26 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani added a comment to T234643: Wikimedia Technical Conference 2019 Session: Quo Vadis Beta Cluster? Towards better testing and staging environments.

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Wed, Nov 13, 10:18 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani created T238261: 2019 Tech Conf Unconference: New CI/Argo.
Wed, Nov 13, 8:40 PM · Wikimedia-Technical-Conference-2019

Tue, Nov 12

thcipriani added a comment to T234641: Wikimedia Technical Conference 2019 Session: Continuous Delivery/Deployment in Wikimedia: The Future of the Deployment Pipeline.
Tue, Nov 12, 10:44 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani added a comment to T234641: Wikimedia Technical Conference 2019 Session: Continuous Delivery/Deployment in Wikimedia: The Future of the Deployment Pipeline.

Session slides: https://people.wikimedia.org/~thcipriani/pipeline-techconf/#/the-future-of-deployment-pipeline

Tue, Nov 12, 10:36 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani added a comment to T234641: Wikimedia Technical Conference 2019 Session: Continuous Delivery/Deployment in Wikimedia: The Future of the Deployment Pipeline.

^ same as above with a little formatting still in place.

Tue, Nov 12, 10:34 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019

Mon, Nov 11

thcipriani added a comment to T237807: gerrit: scoring/ores/editquality takes a long time to git gc.

FWIW, I ran git-sizer on a mirror of the repo. It's certainly a large repo with some large blobs:

Mon, Nov 11, 4:02 AM · Release-Engineering-Team-TODO, Release-Engineering-Team (Development services), Scoring-platform-team, Gerrit

Sat, Nov 9

thcipriani created T237807: gerrit: scoring/ores/editquality takes a long time to git gc.
Sat, Nov 9, 1:41 PM · Release-Engineering-Team-TODO, Release-Engineering-Team (Development services), Scoring-platform-team, Gerrit

Thu, Nov 7

thcipriani triaged T237449: wikimedia/security gerrit requests as Normal priority.

Can we update the wikimedia-security group to include @Dsharpe and @JFishback_WMF?

Thu, Nov 7, 3:09 PM · Release-Engineering-Team-TODO (201911), Gerrit, Security-Team, Release-Engineering-Team
thcipriani closed T229110: Upgrade Gerrit to 2.15.17 as Declined.

I think we're planning to go with 2.16 instead of doing this then 2.16. Since a tentative date has been decided. (Also this month is busy for releng).

Thu, Nov 7, 2:45 PM · Release-Engineering-Team-TODO (201911), Release-Engineering-Team (Development services), Gerrit

Wed, Nov 6

thcipriani added a comment to T235013: Use `git lfs` for large binary files of Design Style Guide.

Most of that 30GB is in packfiles. Those packfiles all seem to contain the same objects:

Wed, Nov 6, 9:38 PM · Release-Engineering-Team, Patch-For-Review, User-Ladsgroup, Wikimedia Design Style Guide
thcipriani edited P9518 deploy style guide.
Wed, Nov 6, 4:02 AM

Tue, Nov 5

thcipriani assigned T237450: Evaluate Airflow's suitability for CI to LarsWirzenius.

Assigning to @LarsWirzenius per discussion.

Tue, Nov 5, 7:39 PM · Release-Engineering-Team-TODO (201911), Continuous-Integration-Infrastructure
thcipriani created T237450: Evaluate Airflow's suitability for CI.
Tue, Nov 5, 7:38 PM · Release-Engineering-Team-TODO (201911), Continuous-Integration-Infrastructure

Fri, Nov 1

thcipriani added a comment to T235013: Use `git lfs` for large binary files of Design Style Guide.

@Volker_E: I agree, some clarity would be good.
Technically:

  • scap nominally supports git-lfs but we have very little experience with it in production.
  • Gerrit supports git-lfs but it's not the greatest implementation.

Beyond that, as far as institutional support, I really don't know where we stand.
cc: @thcipriani

Fri, Nov 1, 9:57 PM · Release-Engineering-Team, Patch-For-Review, User-Ladsgroup, Wikimedia Design Style Guide
thcipriani closed T236443: git review notes split brain, a subtask of T236114: check and fix some Gerrit revs, as Resolved.
Fri, Nov 1, 12:19 AM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani closed T236443: git review notes split brain as Resolved.
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaEvents.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/tools/api-testing.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/skins/Vector.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/vagrant.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/services/kartotherian.git: "refs/notes/review"
Fri, Nov 1, 12:19 AM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit

Thu, Oct 31

thcipriani added a comment to T236443: git review notes split brain.

Steps to fix

Thu, Oct 31, 11:29 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit

Tue, Oct 29

thcipriani added a comment to T236518: new deployment group and access for design site - Volker Eckl, Jan Drewniak, Amir Sarabadani, Mukunda Modell.

@thcipriani can you help with https://gerrit.wikimedia.org/r/c/operations/puppet/+/547014? I'm not entirely sure of what needs to happen there.

Tue, Oct 29, 8:18 PM · Operations, SRE-Access-Requests

Mon, Oct 28

thcipriani added a comment to T172333: Scap: keyholder Too many authentication failures.

This is happening to me on phab1001 now and phab's scap::target includes the requisite part:

scap::target { $deploy_target:
    deploy_user => $deploy_user,
    key_name    => 'phabricator',

Am I missing something? Does this need to be reopened? For some reason I never hit that error ("Too many authentication failures") with phabricator scap deployments until now.

Mon, Oct 28, 9:38 PM · RelEng-Archive-FY201718-Q1, Patch-For-Review, Scap

Fri, Oct 25

thcipriani added a comment to T201529: Document Quibble on wiki.

Code to look at in this case is available at: https://gerrit.wikimedia.org/g/integration/quibble/

Fri, Oct 25, 1:30 PM · Documentation, Quibble
thcipriani updated subscribers of T236017: Move blubberoid to use TLS only..
Fri, Oct 25, 1:28 PM · Release Pipeline (Blubber), Kubernetes, serviceops, Operations

Thu, Oct 24

thcipriani triaged T236443: git review notes split brain as Normal priority.
Thu, Oct 24, 11:08 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani created T236443: git review notes split brain.
Thu, Oct 24, 11:08 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani closed T236344: Some All-Users.git references are outdated after gerrit1001 migration, a subtask of T236114: check and fix some Gerrit revs, as Resolved.
Thu, Oct 24, 11:01 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani closed T236344: Some All-Users.git references are outdated after gerrit1001 migration as Resolved.

I've updated the All-Users.git refs that were out of date. Some I just left alone. The majority were out-of-date moving from "preferred download method" ssh to http or http-auth, etc.

Thu, Oct 24, 11:01 PM · Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani claimed T236344: Some All-Users.git references are outdated after gerrit1001 migration.
Thu, Oct 24, 7:21 PM · Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani added a comment to T236344: Some All-Users.git references are outdated after gerrit1001 migration.

Rerunning on all refs/* showed a few inconsistencies. For All-Users.git there appear to be a few forced merge changes that amount to someone changing download options: nothing serious there.

Thu, Oct 24, 7:21 PM · Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit

Wed, Oct 23

greg awarded T236114: check and fix some Gerrit revs a Barnstar token.
Wed, Oct 23, 12:31 AM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit

Tue, Oct 22

thcipriani closed T236114: check and fix some Gerrit revs, a subtask of T222391: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster), as Resolved.
Tue, Oct 22, 10:43 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
thcipriani closed T236114: check and fix some Gerrit revs as Resolved.

@thcipriani and @hashar paired again on a script to catch the last affected repositories/changes. We managed to get a list of them. Update is on the way.

Tue, Oct 22, 10:43 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani updated the task description for T236114: check and fix some Gerrit revs.
Tue, Oct 22, 10:35 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani created P9440 (An Untitled Masterwork).
Tue, Oct 22, 8:10 PM
thcipriani updated the task description for T236114: check and fix some Gerrit revs.
Tue, Oct 22, 6:59 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani added a comment to T236114: check and fix some Gerrit revs.

Should we be updating the description of this task with changes that need fixing, or will they all be fixed eventually regardless?

Tue, Oct 22, 5:34 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani added a comment to T236114: check and fix some Gerrit revs.

tl;dr: we ran a script, it should have fixed a lot. It may not have fixed everything. DO NOT fiddle with the broken patches; e.g., rebase or +2 or merge or whatever. Please add any problems that *still* exist as of now (post script run) to the task description.

Tue, Oct 22, 5:33 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani updated the task description for T236114: check and fix some Gerrit revs.
Tue, Oct 22, 5:11 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani added a comment to T236114: check and fix some Gerrit revs.

My suspicion is that after Gerrit got stopped on cobalt, a rsync has been done that DID NOT DELETE FILES. Thus we carried over a lot of metadata file that got prepopulated as part of the migration preparation?

Tue, Oct 22, 1:10 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani added a comment to T233851: 1.35.0-wmf.3 deployment blockers.

Started the branch cut. There are some possible issues with new Gerrit, but co-workers seem confident they don't affect this. We won't know if it works until we've tried it.

Tue, Oct 22, 12:52 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), Release, Train Deployments

Mon, Oct 21

thcipriani triaged T236114: check and fix some Gerrit revs as Normal priority.

I've updated all of these to match cobalt unless they had changes that were newer than cobalt's that looked normal.

Mon, Oct 21, 11:59 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani updated the task description for T236114: check and fix some Gerrit revs.
Mon, Oct 21, 11:58 PM · Wikimedia-Incident, Release-Engineering-Team-TODO (201910), Release-Engineering-Team (Development services), Gerrit
thcipriani created P9419 (An Untitled Masterwork).
Mon, Oct 21, 11:32 PM
thcipriani updated the task description for T222391: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster).
Mon, Oct 21, 8:45 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
thcipriani updated the task description for T222391: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster).
Mon, Oct 21, 8:40 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
thcipriani updated the task description for T222391: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster).
Mon, Oct 21, 8:38 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
thcipriani updated the task description for T222391: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster).
Mon, Oct 21, 6:55 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
thcipriani closed Restricted Task, a subtask of T218750: Re-enable use of Gerrit HTTP token to push patchsets, as Resolved.
Mon, Oct 21, 5:49 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T233852: 1.35.0-wmf.4 deployment blockers.

@LarsWirzenius as backup

Mon, Oct 21, 12:48 PM · Release, Train Deployments

Fri, Oct 18

thcipriani assigned T233852: 1.35.0-wmf.4 deployment blockers to brennen.

@LarsWirzenius as backup

Fri, Oct 18, 8:15 PM · Release, Train Deployments
thcipriani assigned T233851: 1.35.0-wmf.3 deployment blockers to LarsWirzenius.

Assigning to lars with @brennen as a backup

Fri, Oct 18, 7:39 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), Release, Train Deployments

Thu, Oct 17

thcipriani added a comment to T235677: Automatic pickup of Gerrit clone master doesn't happen due to missing git-lfs – new deployment env.

it looks like the design microsite is on bromine which is a jessie machine which means that the git::lfs module isn't going to install git lfs...not sure why.

Thu, Oct 17, 10:47 PM · Wikimedia Design Style Guide (Wikimedia Design Style Guide v1.1), Gerrit, Release-Engineering-Team, Operations
thcipriani added a comment to T235677: Automatic pickup of Gerrit clone master doesn't happen due to missing git-lfs – new deployment env.

It was indeed force-pushed by help of @20after4 as the Git history needed to be rewritten as a side-effect of the LFS change.

Thu, Oct 17, 10:37 PM · Wikimedia Design Style Guide (Wikimedia Design Style Guide v1.1), Gerrit, Release-Engineering-Team, Operations
thcipriani added a comment to T235674: Beta cluster doesn’t update since ca. 2019-10-15 21:00 UTC.

I found the thing I half remembered:

Thu, Oct 17, 9:34 PM · Patch-For-Review, Release-Engineering-Team-TODO (201910), Beta-Cluster-Infrastructure
thcipriani added a comment to T235674: Beta cluster doesn’t update since ca. 2019-10-15 21:00 UTC.

Summary of findings and actions re. eventlogging (mwdeploy was fixed by @Krenair yesterday):
I'd like first to thank @Dzahn for helping me through the process. This was caused by etc/keyholder.d/eventlogging private key not having a password set. keyholder arm seems to refuse not password-protected keypairs. The solution was to run sudo ssh-keygen -p -f eventlogging and set a password for the keypair, then sudo keyholder arm and when prompted by the keyholder service, enter the password for eventlogging. That caused keyholder to be happy as seen in the message above (scap did worked this time).
However puppet reverted the addition of the password to the etc/keyholder.d/eventlogging key so @Dzahn suggested that I add one in the labs/private repo so it does not get erased, and thus we don't have to repeat the same process for when the keyholder service is rebooted.
I must note that this same problem exists for all keys at etc/keyholder.d so I wonder if we should be doing this for all listed keys there, or just for this keypair. Pinging @thcipriani for this given that I've seen SAL entries from him dealing with this kind of stuff in the past.

Thu, Oct 17, 9:27 PM · Patch-For-Review, Release-Engineering-Team-TODO (201910), Beta-Cluster-Infrastructure
thcipriani added a comment to T235787: Missing annotations for sync-wikiversions.

These annotations are (TIL) coming from the counters that scap increments by sending udp datagrams to the statsd host. Scap does have logging in case sending a datagram fails (https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/log.py#673); however, that error message doesn't show up in the logs; that is, scap thinks it is sending the message (but it's over UDP so it's just throwing it over the wall, really).

Thu, Oct 17, 9:25 PM · Scap, serviceops, Release-Engineering-Team
thcipriani added a comment to T235677: Automatic pickup of Gerrit clone master doesn't happen due to missing git-lfs – new deployment env.

So from the looks of the puppet output, it looks like this repo is periodically pulled down by puppet to deploy

Thu, Oct 17, 8:47 PM · Wikimedia Design Style Guide (Wikimedia Design Style Guide v1.1), Gerrit, Release-Engineering-Team, Operations

Wed, Oct 16

thcipriani triaged T233850: 1.35.0-wmf.2 deployment blockers as Normal priority.
Wed, Oct 16, 3:29 PM · Patch-For-Review, Release, Train Deployments
thcipriani added a comment to T235338: "Currently active MediaWiki versions:" broken on noc/conf.

Current implementation:

<p>Currently active MediaWiki versions: <?php
        echo str_replace( ' ', ', ', exec( '/usr/bin/scap wikiversions-inuse' ) );
?></p>
<hr>

I'm guessing the exec for /usr/bin/scap wikiversions-inuse has broken with some PHP7 related migration?

Wed, Oct 16, 2:18 PM · User-jijiki, serviceops, Operations, Release-Engineering-Team, Scap, Wikimedia-General-or-Unknown

Oct 15 2019

thcipriani created P9356 (An Untitled Masterwork).
Oct 15 2019, 7:09 PM
thcipriani added a comment to T222472: Investigate gerrit session expiration.

As a data point, folks have complained of getting logged out between last night and this morning (although there were no gerrit restarts during that time). Interestingly, this was noted during our branch cut which is known to cause some strain on the garbage collector, currently (T231872)

Oct 15 2019, 6:43 PM · Patch-For-Review, Release-Engineering-Team-TODO (201907), Release-Engineering-Team (Development services), Gerrit
thcipriani assigned T233850: 1.35.0-wmf.2 deployment blockers to jeena.

Assigning to @jeena with @LarsWirzenius as a secondary

Oct 15 2019, 12:06 PM · Patch-For-Review, Release, Train Deployments

Oct 14 2019

zeljkofilipin awarded T222472: Investigate gerrit session expiration a Meh! token.
Oct 14 2019, 12:46 PM · Patch-For-Review, Release-Engineering-Team-TODO (201907), Release-Engineering-Team (Development services), Gerrit

Oct 13 2019

thcipriani updated subscribers of T222472: Investigate gerrit session expiration.

@Ladsgroup and @Niharika it would be helpful for me to understand how you're using Gerrit since you've reported session expiration recently (and I haven't been able to recreate this issue myself).

Oct 13 2019, 4:42 PM · Patch-For-Review, Release-Engineering-Team-TODO (201907), Release-Engineering-Team (Development services), Gerrit

Oct 11 2019

thcipriani added a comment to T235279: Create a new repository for kubernetes tool under a new directory in gerrit that doesn't exist yet.

What Paladox did :]

Oct 11 2019, 9:43 PM · Gerrit, cloud-services-team (Kanban)

Oct 10 2019

thcipriani added a comment to T234866: Set gerrit1001 master switch date.

I think some upcoming Monday would be best for this. We want to avoid making a big change in prod on a Friday. Tuesday–Thursday risks running into train/other deployments. Plus, my availability is better on Mondays as well -- although the success of this migration is probably more contingent on @Dzahn and @Paladox's availability than my own, really.

Oct 10 2019, 5:55 PM · serviceops, Release-Engineering-Team, Gerrit
thcipriani added a comment to P9308 (An Untitled Masterwork).
                             :Name           |Entries              |  AvgGet |Hit Ratio|
                             :               |   Mem   Disk   Space|         |Mem  Disk|
                             :---------------+---------------------+---------+---------+
2019-10-09T00:00:01+00:00.txt:D web_sessions |   276   4225   1.70m|         | 96%   0%|
2019-10-09T00:30:02+00:00.txt:D web_sessions |   276   4225   1.70m|         | 96%   0%|
2019-10-09T01:00:01+00:00.txt:D web_sessions |   276   4225   1.70m|         | 96%   0%|
2019-10-09T01:30:01+00:00.txt:D web_sessions |   276   4225   1.70m|         | 96%   0%|
2019-10-09T02:00:01+00:00.txt:D web_sessions |   278   4227   1.70m|         | 96%   0%|
2019-10-09T03:00:01+00:00.txt:D web_sessions |   279   4228   1.70m|         | 96%   0%|
2019-10-09T03:30:01+00:00.txt:D web_sessions |   279   4228   1.70m|         | 96%   0%|
2019-10-09T04:00:01+00:00.txt:D web_sessions |   279   4228   1.70m|         | 96%   0%|
2019-10-09T04:30:01+00:00.txt:D web_sessions |   279   4228   1.70m|         | 96%   0%|
2019-10-09T05:00:01+00:00.txt:D web_sessions |   279   4228   1.70m|         | 96%   0%|
2019-10-09T05:30:02+00:00.txt:D web_sessions |    11   4228   1.70m|         | 96%  41%|
2019-10-09T06:00:01+00:00.txt:D web_sessions |    21   4226   1.70m|         | 95%  22%|
2019-10-09T06:30:01+00:00.txt:D web_sessions |    27   4228   1.70m|         | 96%  25%|
2019-10-09T07:00:01+00:00.txt:D web_sessions |    32   4226   1.70m|         | 96%  15%|
2019-10-09T07:30:01+00:00.txt:D web_sessions |    42   4229   1.70m|         | 94%   8%|
2019-10-09T08:00:01+00:00.txt:D web_sessions |    45   4230   1.70m|         | 94%   6%|
2019-10-09T08:30:01+00:00.txt:D web_sessions |    53   4228   1.70m|         | 94%   5%|
2019-10-09T09:00:01+00:00.txt:D web_sessions |    57   4227   1.70m|         | 93%   3%|
2019-10-09T09:30:01+00:00.txt:D web_sessions |    64   4224   1.70m|         | 92%   3%|
2019-10-09T10:00:01+00:00.txt:D web_sessions |    69   4222   1.70m|         | 92%   2%|
2019-10-09T10:30:01+00:00.txt:D web_sessions |    71   4221   1.70m|         | 92%   2%|
2019-10-09T11:00:01+00:00.txt:D web_sessions |    75   4224   1.70m|         | 92%   2%|
2019-10-09T11:30:01+00:00.txt:D web_sessions |    77   4225   1.70m|         | 92%   2%|
2019-10-09T12:00:01+00:00.txt:D web_sessions |    80   4226   1.70m|         | 92%   1%|
2019-10-09T12:30:01+00:00.txt:D web_sessions |    85   4226   1.70m|         | 92%   1%|
2019-10-09T13:00:01+00:00.txt:D web_sessions |    86   4225   1.70m|         | 92%   1%|
2019-10-09T13:30:01+00:00.txt:D web_sessions |    91   4224   1.70m|         | 92%   1%|
2019-10-09T14:00:01+00:00.txt:D web_sessions |    95   4226   1.70m|         | 92%   1%|
2019-10-09T14:30:01+00:00.txt:D web_sessions |    98   4226   1.70m|         | 92%   1%|
2019-10-09T15:00:01+00:00.txt:D web_sessions |   105   4226   1.70m|         | 92%   1%|
2019-10-09T15:30:01+00:00.txt:D web_sessions |   114   4227   1.70m|         | 91%   1%|
2019-10-09T16:00:01+00:00.txt:D web_sessions |   116   4225   1.70m|         | 91%   1%|
2019-10-09T16:30:01+00:00.txt:D web_sessions |   120   4220   1.70m|         | 91%   1%|
2019-10-09T17:00:01+00:00.txt:D web_sessions |   128   4223   1.70m|         | 90%   0%|
2019-10-09T17:30:01+00:00.txt:D web_sessions |   131   4223   1.70m|         | 90%   0%|
2019-10-09T18:00:01+00:00.txt:D web_sessions |   134   4225   1.70m|         | 90%   0%|
2019-10-09T18:30:01+00:00.txt:D web_sessions |   135   4225   1.70m|         | 90%   0%|
2019-10-09T19:00:01+00:00.txt:D web_sessions |   137   4225   1.70m|         | 91%   0%|
2019-10-09T19:30:01+00:00.txt:D web_sessions |   138   4223   1.70m|         | 91%   0%|
2019-10-09T20:00:01+00:00.txt:D web_sessions |   140   4223   1.70m|         | 91%   0%|
2019-10-09T20:30:01+00:00.txt:D web_sessions |   142   4222   1.70m|         | 91%   0%|
2019-10-09T21:00:01+00:00.txt:D web_sessions |   146   4225   1.70m|         | 91%   0%|
2019-10-09T21:30:01+00:00.txt:D web_sessions |   146   4225   1.70m|         | 91%   0%|
2019-10-09T22:00:01+00:00.txt:D web_sessions |   148   4226   1.70m|         | 91%   0%|
2019-10-09T22:30:01+00:00.txt:D web_sessions |    34   4227   1.70m|         | 98% 100%|
2019-10-09T23:00:01+00:00.txt:D web_sessions |    39   4226   1.70m|         | 97%  56%|
2019-10-09T23:30:01+00:00.txt:D web_sessions |    47   4226   1.70m|         | 96%  60%|
2019-10-10T00:00:01+00:00.txt:D web_sessions |    50   4226   1.70m|         | 95%  27%|
2019-10-10T00:30:01+00:00.txt:D web_sessions |    50   4224   1.70m|         | 95%  25%|
2019-10-10T01:00:01+00:00.txt:D web_sessions |    53   4225   1.70m|         | 95%  21%|
2019-10-10T01:30:01+00:00.txt:D web_sessions |    54   4226   1.70m|         | 95%  21%|
2019-10-10T02:00:01+00:00.txt:D web_sessions |    54   4224   1.70m|         | 95%  15%|
2019-10-10T02:30:01+00:00.txt:D web_sessions |    54   4223   1.70m|         | 96%  12%|
2019-10-10T03:00:01+00:00.txt:D web_sessions |    54   4223   1.70m|         | 95%  11%|
2019-10-10T03:30:01+00:00.txt:D web_sessions |    54   4223   1.70m|         | 95%  10%|
2019-10-10T04:00:01+00:00.txt:D web_sessions |    54   4223   1.70m|         | 95%  10%|
2019-10-10T04:30:01+00:00.txt:D web_sessions |    55   4222   1.70m|         | 94%   9%|
2019-10-10T05:00:01+00:00.txt:D web_sessions |    57   4222   1.70m|         | 94%   8%|
2019-10-10T05:30:01+00:00.txt:D web_sessions |    59   4222   1.70m|         | 94%   7%|
2019-10-10T06:00:01+00:00.txt:D web_sessions |    61   4220   1.70m|         | 94%   7%|
2019-10-10T06:30:01+00:00.txt:D web_sessions |    64   4221   1.70m|         | 94%   6%|
2019-10-10T07:00:01+00:00.txt:D web_sessions |    69   4221   1.70m|         | 94%   5%|
2019-10-10T07:30:01+00:00.txt:D web_sessions |    73   4220   1.70m|         | 94%   5%|
2019-10-10T08:00:01+00:00.txt:D web_sessions |    76   4220   1.70m|         | 94%   5%|
2019-10-10T08:30:01+00:00.txt:D web_sessions |    81   4218   1.70m|         | 92%   3%|
2019-10-10T09:00:01+00:00.txt:D web_sessions |    88   4220   1.70m|         | 91%   3%|
2019-10-10T09:30:01+00:00.txt:D web_sessions |    93   4217   1.70m|         | 91%   2%|
2019-10-10T10:00:01+00:00.txt:D web_sessions |    97   4220   1.70m|         | 92%   2%|
2019-10-10T10:30:01+00:00.txt:D web_sessions |    98   4219   1.70m|         | 93%   2%|
2019-10-10T11:00:01+00:00.txt:D web_sessions |   100   4220   1.70m|         | 93%   2%|
2019-10-10T11:30:01+00:00.txt:D web_sessions |   103   4220   1.70m|         | 93%   2%|
2019-10-10T12:00:01+00:00.txt:D web_sessions |   105   4219   1.70m|         | 93%   2%|
2019-10-10T12:30:01+00:00.txt:D web_sessions |   107   4218   1.70m|         | 93%   2%|
2019-10-10T13:00:01+00:00.txt:D web_sessions |   108   4218   1.70m|         | 93%   1%|
2019-10-10T13:30:01+00:00.txt:D web_sessions |   111   4219   1.70m|         | 93%   1%|
2019-10-10T14:00:01+00:00.txt:D web_sessions |   115   4221   1.70m|         | 93%   1%|
2019-10-10T14:30:01+00:00.txt:D web_sessions |   115   4217   1.70m|         | 93%   1%|
2019-10-10T15:00:01+00:00.txt:D web_sessions |   118   4217   1.70m|         | 93%   1%|
2019-10-10T15:30:02+00:00.txt:D web_sessions |   123   4218   1.70m|         | 93%   1%|
2019-10-10T16:00:01+00:00.txt:D web_sessions |   127   4220   1.70m|         | 93%   1%|
2019-10-10T16:30:01+00:00.txt:D web_sessions |    43   4217   1.70m|         | 92%  29%|
Oct 10 2019, 4:41 PM
thcipriani created P9308 (An Untitled Masterwork).
Oct 10 2019, 4:34 PM
thcipriani added a comment to T224448: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely.

Mentioned in SAL (#wikimedia-operations) [2019-10-10T16:04:38Z] <thcipriani> restarting gerrit due to T224448

Oct 10 2019, 4:18 PM · Patch-For-Review, Upstream, Release-Engineering-Team-TODO, Release-Engineering-Team (Development services), serviceops-radar, Gerrit

Oct 9 2019

thcipriani updated subscribers of T234639: Wikimedia Technical Conference 2019 Session: WMF CI 2.0: Status and future.

Folks who attended the New CI Working Group meetings are sparsely represented at Tech Conf; however, I'd love to talk about the work of that group at a high level.

Oct 9 2019, 5:56 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani updated subscribers of T234641: Wikimedia Technical Conference 2019 Session: Continuous Delivery/Deployment in Wikimedia: The Future of the Deployment Pipeline.

I'm very interested in this topic as I've been involved with the Deployment Pipeline work along with @dduvall (who is not attending tech conf) and @akosiaris (who is attending afaik)

Oct 9 2019, 5:51 PM · International-Developer-Events, Wikimedia-Technical-Conference-2019
thcipriani edited projects for T228915: Update local-charts repository to use restbase chart from deployment-charts repo, added: Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO (201909).
Oct 9 2019, 12:06 PM · Release-Engineering-Team-TODO (201911), Patch-For-Review, Release-Engineering-Team (Local Dev), Developer Productivity, local-charts
thcipriani edited projects for T228910: Move restbase chart from local-charts to deployment-charts repository, added: Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO (201909).
Oct 9 2019, 12:06 PM · Release-Engineering-Team-TODO (201911), Patch-For-Review, RESTBase, Core Platform Team Workboards (Clinic Duty Team), Release-Engineering-Team (Local Dev), Developer Productivity, local-charts

Oct 8 2019

thcipriani triaged T234872: Scap silently ignores sync and service restart if hash is equal as Normal priority.

I'm neutral on whether this optimisation is useful, but at the very least I think it could be made more obvious to the operator that a restart did in fact not occur.

Oct 8 2019, 2:47 PM · Release-Engineering-Team, Performance-Team (Radar), Scap
thcipriani added a comment to T234866: Set gerrit1001 master switch date.

Run /usr/bin/java -jar review_site/bin/gerrit.war reindex -d review_site --threads 4 (this may take a while)

Oct 8 2019, 2:32 PM · serviceops, Release-Engineering-Team, Gerrit
thcipriani awarded Blog Post: Introducing Phatality a Barnstar token.
Oct 8 2019, 1:00 PM · Phabricator
thcipriani closed T234233: Reset 2FA for Phabricator account `Apap04` as Resolved.

@bd808 Done.

$ ssh bastion.wmflabs.org
$ ls -lh ~apap04/2fa-reset-request.txt
-rw-r--r-- 1 apap04 wikidev 56 Oct  5 21:19 /home/apap04/2fa-reset-request.txt
$ cat ~apap04/2fa-reset-request.txt
Hello, world! https://phabricator.wikimedia.org/T234233

@Aklapper, @mmodell, @greg, @thcipriani verification checks out for @Apap04 having control of linked Developer account. Can one of you take care of the Phabricator 2FA removal for them? (Requires a hat I have not collected.)

Oct 8 2019, 12:56 PM · Phabricator

Oct 4 2019

thcipriani edited P9243 (An Untitled Masterwork).
Oct 4 2019, 9:14 PM
thcipriani created P9243 (An Untitled Masterwork).
Oct 4 2019, 9:13 PM
thcipriani added a comment to T201261: Exclude secondary jenkins-bot/PipelineBot messages from Gerrit in Wikibugs on IRC.

@valhallasw Everything for Fresnel. For PipelineBot, I'm not sure. I've not seen it leave voting messages so probably everything as well, but T218442 suggests it can sometimes vote so in that case maybe only non-voting messages.
However from what I understand, it is no longer possible to distinguish a pre-existing V-2/V+2 from a new one with jenkins-bot leaves a comment. Or at least I got that impression because whenever I leave a comment, wikibugs shows it as if I left the same score again again. If it's possible to ignore all jenkins-bot/PipelineBot messages that don't change the score (or made it positive), that would be perfection :)

Oct 4 2019, 8:43 PM · Release-Engineering-Team, Wikibugs
thcipriani updated subscribers of T234691: Garbage-collect development (and other?) images from Docker registry as appropriate.
Oct 4 2019, 8:34 PM · Release Pipeline, Release-Engineering-Team-TODO, dev-images, Release-Engineering-Team (Local Dev)

Oct 3 2019

thcipriani closed T234533: gerrit shows merged patch as pending as Resolved.

The fix is: we should just delete the unmerged change from disk. This will trigger a reindex.

Clarification: delete the unmerged change using the REST api to remove it and trigger a reindex.

Oct 3 2019, 5:44 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

The fix is: we should just delete the unmerged change from disk. This will trigger a reindex.

Oct 3 2019, 4:36 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

the change numbers should increase, i.e., 534389 was pushed up before 534392. This is supported by the git timestamps for the changes first being created in Gerrit:

Oct 3 2019, 4:34 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

The URL https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389 is the one that just showed up on my Gerrit "Outgoing reviews", with that same patch that was merged a month ago.

Oct 3 2019, 3:50 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

OK. So this change was submitted twice.

Oct 3 2019, 3:50 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

The url is different in that @thcipriani (Reviewed-on: https://gerrit.wikimedia.org/r/534392) compared to the url in description https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389

Oct 3 2019, 3:37 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

hrm, yes, I see this in the review notes as well:

Oct 3 2019, 3:33 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T234533: gerrit shows merged patch as pending.

My question is: how was this merged? Gerrit doesn't seem to think it was merged through the gerrit interface.

Oct 3 2019, 3:20 PM · Release-Engineering-Team-TODO, Gerrit
thcipriani added a comment to T233989: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs.

The error is triggered when doing a query which is logged as well:

[2019-10-01 13:42:27,917] [SSH gerrit query limit:10 (status:open OR status:closed) AND NOT (502978 OR 478688 OR 266726 OR 266725 OR 256050 OR 235852 OR 143269 OR 102114 OR 101842 OR 99101 OR 58858 OR 32902 OR 13930 OR 225009 OR 225033 OR 225080 OR 225087 OR 225099) --all-approvals --comments --format JSON --start 66030 (owl)]
WARN  com.google.gerrit.server.query.change.ChangeIsVisibleToPredicate : Skipping change 342073 because the corresponding repository was not found
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable

That bot has since been deactivated and thus the warning no more show up. If I run the query manually, on the server side I get:

com.google.gerrit.server.permissions.PermissionBackendException: project 'analytics/wmde/Wiktionary/WD_percentUsageDashboard' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/shinken' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'analytics/wmde/Wiktionary/WD_percentUsageDashboard' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/pkg-php/php-ast' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/pkg-php/php-ast' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable

So the indexing has not completed :\

Oct 3 2019, 1:15 PM · Upstream, Gerrit

Oct 2 2019

thcipriani added a comment to T234328: biterg.io Gerrit crawling probably stresses the server too much.

Bitergia asked internally:

can you double-check that the replica server (https://gerrit-replica.wikimedia.org/r/) is working? We got a not found error.

Oct 2 2019, 4:57 PM · Release-Engineering-Team-TODO (201911), Developer-Advocacy, wikimedia.biterg.io, Release-Engineering-Team (Development services), Gerrit
thcipriani created P9229 (An Untitled Masterwork).
Oct 2 2019, 4:53 PM
thcipriani added a comment to T233316: Deployment Pipeline fails with CPS error for Kartotherian.
Oct 2 2019, 4:00 PM · Release-Engineering-Team-TODO (201910), Maps (Kartotherian), Release Pipeline, Release-Engineering-Team (Pipeline)
thcipriani moved T206358: Request Sauce Labs access for niedzielski from INBOX to Blocked externally on the Release-Engineering-Team-TODO board.
Oct 2 2019, 2:19 PM · Release-Engineering-Team-TODO
thcipriani assigned T206358: Request Sauce Labs access for niedzielski to Niedzielski.

@Niedzielski do you have the same problem as @Etonkovidova? Or do you get the error message all the time?

Oct 2 2019, 2:19 PM · Release-Engineering-Team-TODO
thcipriani triaged T232024: Branch REL1_34 for MediaWiki and deployed extensions as Normal priority.
Oct 2 2019, 2:16 PM · Release-Engineering-Team-TODO (201910), Core Platform Team, MediaWiki-Releasing, MW-1.34-notes, MW-1.34-release
thcipriani assigned T232024: Branch REL1_34 for MediaWiki and deployed extensions to dduvall.

Tentatively assigning to @dduvall following IRC discussion on Monday.

Oct 2 2019, 2:15 PM · Release-Engineering-Team-TODO (201910), Core Platform Team, MediaWiki-Releasing, MW-1.34-notes, MW-1.34-release
thcipriani moved T227562: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI from INBOX to Soon-ish on the Release-Engineering-Team-TODO board.
Oct 2 2019, 2:13 PM · Release-Engineering-Team-TODO, Documentation, Gerrit
thcipriani edited projects for T227562: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI, added: Release-Engineering-Team-TODO; removed Release-Engineering-Team-TODO (201909).
Oct 2 2019, 2:13 PM · Release-Engineering-Team-TODO, Documentation, Gerrit
thcipriani edited projects for T233644: SCAP python error on successful deploy, added: Release-Engineering-Team-TODO; removed Release-Engineering-Team-TODO (201909).
Oct 2 2019, 2:11 PM · Release-Engineering-Team-TODO, Scap
thcipriani moved T114488: Automate the recurring management of wikitech:Deployments and phab:#train_deployments from INBOX to Doing on the Release-Engineering-Team-TODO (201910) board.
Oct 2 2019, 2:11 PM · Release-Engineering-Team-TODO (201911), Deployments, User-MModell