The output of the shallow clone from the "git-http-backend" is:

Similar to T242298: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)

Assigning per brief discussion in IRC. @dancy feel free to loop in others (@LarsWirzenius will be helpful for actual release) as needed.

In T243009#6527973, @Joe wrote:

In T243009#6519176, @Krinkle wrote:

Open questions (I guess for @Joe):

• Would "simply" restarting php-fpm directly abort on-going requests? I assume so, and we're fine with this. Just checking so that we can document this as expected.

Yes, restarting php-fpm will kill all in-flight requests

In T263008#6565092, @hashar wrote:

Gerrit tuning infos: https://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf

We probably want to look at jgit config:

modules/gerrit/templates/gerrit.config.erb

[core]
    packedGitOpenFiles = <%= @git_open_files %>  # 20000
    packedGitLimit = 4g

Happening on hrwiki as of today (2020-10-14) with wmf.11: Failed to load data blob from tt:1677927: Bad data in text row 1677927. Use findBadBlobs.php to remedy.. If this problem persist, use the findBadBlobs maintenance script to investigate the issue and mark bad blobs.

Still seen in production today for wmf.11:

There was a spike in events related to "Could not enqueue jobs from stream" for various streams at ~12pm UTC: https://logstash.wikimedia.org/goto/7bbe517376de0be509c6a07dd8d027bc

Week of data center switchover

In T261403#6540180, @kostajh wrote:

if you help me with requesting a gerrit-repo that would be great (no experience with that yet). also

Pinging Release-Engineering-Team and serviceops with some questions about this process:

First, as I understand it, since we are using the Deployment pipeline we just need a single gerrit repository that will have the .pipeline directory (and not a separate deploy repo as it appears ORES has), is that right? (Question for serviceops I think)

In T263179#6538423, @DannyS712 wrote:

Per https://wikitech.wikimedia.org/wiki/Deployments, 8 hours from now 1.36.0-wmf.13 will be deployed to group 0. But, T263177: 1.36.0-wmf.11 deployment blockers hasn't been reattempted yet, and everything is currently on wmf.10. Is there a plan for rolling out .11 and then .13, or are we going straight to .13?

In T248180#6525280, @kostajh wrote:

None of the patches for GrowthExperiments pass CI because of this. Does anyone have ideas on what caused this problem?

In T256367#6523330, @CDanis wrote:

I also support this proposal. @DannyS712 has been very helpful in all sorts of production firefighting recently -- of particular note, T264369

@greg @thcipriani @faidon @mark Can one of you please help move this along?

hrm, seems like we ran out of disk space. Investigating.

One other thing to note: there are two purges that happen as part of the scap sync-world and one seems to happen way too soon. @dduvall pointed this out to me recently that rebuildLocalisationCache.php does a purge:

In T252124#6494323, @Zbyszko wrote:

@thcipriani I have the change in puppet (https://gerrit.wikimedia.org/r/630081). What else would I need to do to guarantee that generated lists in /etc/dsh/groups are picked up? I rather leave everything else configured as it is right now (https://github.com/wikimedia/wikidata-query-deploy/blob/master/scap/scap.cfg)

Approved as Release-Engineering-Team manager

In T257978#6490003, @dancy wrote:

Hi @DannyS712. @thcipriani instructed me to remove the non-blocking tasks from the subtasks list. An empty subtasks lists is the indication that there are no outstanding train blockers. Hope that helps!

In T263682#6489090, @hashar wrote:

I deleted a repository on disk (for T257775) which seemed to work fine but Gerrit internal caches were not invalidated as a result.

Solved it by flushing the project_list cache.

Added as a train blocker

Requests are generating a lot of noise here. For example the 1 request-id in this task generated 164 notices https://logstash.wikimedia.org/goto/0aa65e4883b273a1497897615ab6a081

train blockers are UBNs

train blockers are UBNs

In T262867#6481940, @srodlund wrote:

@thcipriani Fixed!

One quick thing:

After a few rounds of spikes, messages seem to have subsided, keeping an eye on logs for the time being.

In T263132#6475784, @jeena wrote:

Various jobenqueue errors happened today in the past 6 hours with spikes of over 10k. I didn't see any k8s deployments correlated with them.

Couple of photos appeal to me:

In T262867#6474912, @srodlund wrote:

@thcipriani do you have any thoughts about an image you would like to include for the featured image?

Here is some info about picking an image: https://www.mediawiki.org/wiki/Wikimedia_technical_blog_editorial_guidelines#Images_used_in_your_post