Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

In T234641#5658582, @thcipriani wrote:

Session slides: https://people.wikimedia.org/~thcipriani/pipeline-techconf/#/the-future-of-deployment-pipeline

Session slides: https://people.wikimedia.org/~thcipriani/pipeline-techconf/#/the-future-of-deployment-pipeline

^ same as above with a little formatting still in place.

FWIW, I ran git-sizer on a mirror of the repo. It's certainly a large repo with some large blobs:

Can we update the wikimedia-security group to include @Dsharpe and @JFishback_WMF?

In T229110#5641121, @Paladox wrote:

I think we're planning to go with 2.16 instead of doing this then 2.16. Since a tentative date has been decided. (Also this month is busy for releng).

Most of that 30GB is in packfiles. Those packfiles all seem to contain the same objects:

Assigning to @LarsWirzenius per discussion.

In T235013#5627732, @mmodell wrote:

@Volker_E: I agree, some clarity would be good.
Technically:

• scap nominally supports git-lfs but we have very little experience with it in production.
• Gerrit supports git-lfs but it's not the greatest implementation.

Beyond that, as far as institutional support, I really don't know where we stand.
cc: @thcipriani

In T236443#5625312, @thcipriani wrote:

• ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaEvents.git: "refs/notes/review"
• ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/tools/api-testing.git: "refs/notes/review"
• ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/skins/Vector.git: "refs/notes/review"
• ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/vagrant.git: "refs/notes/review"
• ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/services/kartotherian.git: "refs/notes/review"

Steps to fix

In T236518#5617222, @Ottomata wrote:

@thcipriani can you help with https://gerrit.wikimedia.org/r/c/operations/puppet/+/547014? I'm not entirely sure of what needs to happen there.

In T172333#5607532, @mmodell wrote:

This is happening to me on phab1001 now and phab's scap::target includes the requisite part:

scap::target { $deploy_target:
    deploy_user => $deploy_user,
    key_name    => 'phabricator',

Am I missing something? Does this need to be reopened? For some reason I never hit that error ("Too many authentication failures") with phabricator scap deployments until now.

Code to look at in this case is available at: https://gerrit.wikimedia.org/g/integration/quibble/

I've updated the All-Users.git refs that were out of date. Some I just left alone. The majority were out-of-date moving from "preferred download method" ssh to http or http-auth, etc.

Rerunning on all refs/* showed a few inconsistencies. For All-Users.git there appear to be a few forced merge changes that amount to someone changing download options: nothing serious there.

In T236114#5596944, @hashar wrote:

@thcipriani and @hashar paired again on a script to catch the last affected repositories/changes. We managed to get a list of them. Update is on the way.

In T236114#5596069, @Mholloway wrote:

Should we be updating the description of this task with changes that need fixing, or will they all be fixed eventually regardless?

tl;dr: we ran a script, it should have fixed a lot. It may not have fixed everything. DO NOT fiddle with the broken patches; e.g., rebase or +2 or merge or whatever. Please add any problems that *still* exist as of now (post script run) to the task description.

In T236114#5594704, @hashar wrote:

My suspicion is that after Gerrit got stopped on cobalt, a rsync has been done that DID NOT DELETE FILES. Thus we carried over a lot of metadata file that got prepopulated as part of the migration preparation?

In T233851#5594700, @LarsWirzenius wrote:

Started the branch cut. There are some possible issues with new Gerrit, but co-workers seem confident they don't affect this. We won't know if it works until we've tried it.

I've updated all of these to match cobalt unless they had changes that were newer than cobalt's that looked normal.

In T233852#5587957, @thcipriani wrote:

@LarsWirzenius as backup

@LarsWirzenius as backup

Assigning to lars with @brennen as a backup

it looks like the design microsite is on bromine which is a jessie machine which means that the git::lfs module isn't going to install git lfs...not sure why.

In T235677#5585133, @Volker_E wrote:

It was indeed force-pushed by help of @20after4 as the Git history needed to be rewritten as a side-effect of the LFS change.

I found the thing I half remembered:

In T235674#5585184, @MarcoAurelio wrote:

Summary of findings and actions re. eventlogging (mwdeploy was fixed by @Krenair yesterday):
I'd like first to thank @Dzahn for helping me through the process. This was caused by etc/keyholder.d/eventlogging private key not having a password set. keyholder arm seems to refuse not password-protected keypairs. The solution was to run sudo ssh-keygen -p -f eventlogging and set a password for the keypair, then sudo keyholder arm and when prompted by the keyholder service, enter the password for eventlogging. That caused keyholder to be happy as seen in the message above (scap did worked this time).
However puppet reverted the addition of the password to the etc/keyholder.d/eventlogging key so @Dzahn suggested that I add one in the labs/private repo so it does not get erased, and thus we don't have to repeat the same process for when the keyholder service is rebooted.
I must note that this same problem exists for all keys at etc/keyholder.d so I wonder if we should be doing this for all listed keys there, or just for this keypair. Pinging @thcipriani for this given that I've seen SAL entries from him dealing with this kind of stuff in the past.

These annotations are (TIL) coming from the counters that scap increments by sending udp datagrams to the statsd host. Scap does have logging in case sending a datagram fails (https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/log.py#673); however, that error message doesn't show up in the logs; that is, scap thinks it is sending the message (but it's over UDP so it's just throwing it over the wall, really).

So from the looks of the puppet output, it looks like this repo is periodically pulled down by puppet to deploy

In T235338#5569953, @Reedy wrote:

Current implementation:

<p>Currently active MediaWiki versions: <?php
        echo str_replace( ' ', ', ', exec( '/usr/bin/scap wikiversions-inuse' ) );
?></p>
<hr>

I'm guessing the exec for /usr/bin/scap wikiversions-inuse has broken with some PHP7 related migration?

As a data point, folks have complained of getting logged out between last night and this morning (although there were no gerrit restarts during that time). Interestingly, this was noted during our branch cut which is known to cause some strain on the garbage collector, currently (T231872)

Assigning to @jeena with @LarsWirzenius as a secondary

@Ladsgroup and @Niharika it would be helpful for me to understand how you're using Gerrit since you've reported session expiration recently (and I haven't been able to recreate this issue myself).

In T235279#5567259, @hashar wrote:

What Paladox did :]

I think some upcoming Monday would be best for this. We want to avoid making a big change in prod on a Friday. Tuesday–Thursday risks running into train/other deployments. Plus, my availability is better on Mondays as well -- although the success of this migration is probably more contingent on @Dzahn and @Paladox's availability than my own, really.

In T224448#5563533, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2019-10-10T16:04:38Z] <thcipriani> restarting gerrit due to T224448

Folks who attended the New CI Working Group meetings are sparsely represented at Tech Conf; however, I'd love to talk about the work of that group at a high level.

I'm very interested in this topic as I've been involved with the Deployment Pipeline work along with @dduvall (who is not attending tech conf) and @akosiaris (who is attending afaik)

I'm neutral on whether this optimisation is useful, but at the very least I think it could be made more obvious to the operator that a restart did in fact not occur.

Run /usr/bin/java -jar review_site/bin/gerrit.war reindex -d review_site --threads 4 (this may take a while)

In T234233#5549503, @bd808 wrote:

In T234233#5549465, @Apap04 wrote:

@bd808 Done.

$ ssh bastion.wmflabs.org
$ ls -lh ~apap04/2fa-reset-request.txt
-rw-r--r-- 1 apap04 wikidev 56 Oct  5 21:19 /home/apap04/2fa-reset-request.txt
$ cat ~apap04/2fa-reset-request.txt
Hello, world! https://phabricator.wikimedia.org/T234233

@Aklapper, @mmodell, @greg, @thcipriani verification checks out for @Apap04 having control of linked Developer account. Can one of you take care of the Phabricator 2FA removal for them? (Requires a hat I have not collected.)

In T201261#5548264, @Krinkle wrote:

@valhallasw Everything for Fresnel. For PipelineBot, I'm not sure. I've not seen it leave voting messages so probably everything as well, but T218442 suggests it can sometimes vote so in that case maybe only non-voting messages.
However from what I understand, it is no longer possible to distinguish a pre-existing V-2/V+2 from a new one with jenkins-bot leaves a comment. Or at least I got that impression because whenever I leave a comment, wikibugs shows it as if I left the same score again again. If it's possible to ignore all jenkins-bot/PipelineBot messages that don't change the score (or made it positive), that would be perfection :)

In T234533#5544460, @thcipriani wrote:

In T234533#5544444, @thcipriani wrote:

The fix is: we should just delete the unmerged change from disk. This will trigger a reindex.

Clarification: delete the unmerged change using the REST api to remove it and trigger a reindex.

In T234533#5544444, @thcipriani wrote:

The fix is: we should just delete the unmerged change from disk. This will trigger a reindex.

the change numbers should increase, i.e., 534389 was pushed up before 534392. This is supported by the git timestamps for the changes first being created in Gerrit:

In T234533#5544228, @Marostegui wrote:

The URL https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389 is the one that just showed up on my Gerrit "Outgoing reviews", with that same patch that was merged a month ago.

OK. So this change was submitted twice.

In T234533#5544222, @Paladox wrote:

The url is different in that @thcipriani (Reviewed-on: https://gerrit.wikimedia.org/r/534392) compared to the url in description https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389

hrm, yes, I see this in the review notes as well:

My question is: how was this merged? Gerrit doesn't seem to think it was merged through the gerrit interface.

In T233989#5543982, @hashar wrote:

The error is triggered when doing a query which is logged as well:

[2019-10-01 13:42:27,917] [SSH gerrit query limit:10 (status:open OR status:closed) AND NOT (502978 OR 478688 OR 266726 OR 266725 OR 256050 OR 235852 OR 143269 OR 102114 OR 101842 OR 99101 OR 58858 OR 32902 OR 13930 OR 225009 OR 225033 OR 225080 OR 225087 OR 225099) --all-approvals --comments --format JSON --start 66030 (owl)]
WARN  com.google.gerrit.server.query.change.ChangeIsVisibleToPredicate : Skipping change 342073 because the corresponding repository was not found
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable

That bot has since been deactivated and thus the warning no more show up. If I run the query manually, on the server side I get:

com.google.gerrit.server.permissions.PermissionBackendException: project 'analytics/wmde/Wiktionary/WD_percentUsageDashboard' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/shinken' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'analytics/wmde/Wiktionary/WD_percentUsageDashboard' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/pkg-php/php-ast' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'operations/debs/pkg-php/php-ast' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/kartotherian' is unavailable
com.google.gerrit.server.permissions.PermissionBackendException: project 'maps/tilerator' is unavailable

So the indexing has not completed :\

In T234328#5541545, @Aklapper wrote:

Bitergia asked internally:

can you double-check that the replica server (https://gerrit-replica.wikimedia.org/r/) is working? We got a not found error.

In T233316#5536374, @Mathew.onipe wrote:

Post merge builds seems to fail.
https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/539209

In T206358#5467853, @zeljkofilipin wrote:

@Niedzielski do you have the same problem as @Etonkovidova? Or do you get the error message all the time?

Tentatively assigning to @dduvall following IRC discussion on Monday.