We've been using this task for the dawiki issue as it was the same trace (although it may be a different cause as you've now pointed out - we were advised it wasn't to do with legacy encoding earlier)

Notice given at https://da.wikipedia.org/w/index.php?diff=11489388

Adding User-notice, A deployer took a Sysadmin action to delete 2 pages to mitigate. Notice coming on wiki.

Happening on even login and preferences so not LT related

Possible dupe: T337700: Exception: "Malformed UTF-8 characters" in Parser\MagicWordArray (via LqtVIew)

Also reported in #wikimedia-tech, see https://da.wikipedia.org/wiki/Wikipedia:Teknisk_forum#Intern_fejl

Thanks for the quick response and for all the hard work. Please enjoy some of your Memorial Day weekend. :)

There's also https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Step_5:_setting_indexes which I'm not sure if that's been / needs running.

In T335851#8823395, @Tgr wrote:

The explainer says "If your RPs are SameParty, you may be better served by First-Party Sets." Which would be nice if there actually was a first-party sets standard, but it doesn't seem to be making much progress.

In T337121#8868286, @RhinosF1 wrote:

@Nux: do you have a WMF sponsor for this?

@Nux: do you have a WMF sponsor for this?

Employees being off-boarded from the WMF may wish to continue in some roles as a volunteer.

Phorge is definitely not on life support which is the community fork of Phab that formed since then.

@Ladsgroup: is run.php your work?

@Samwilson: assuming this is following https://wikitech.wikimedia.org/wiki/Volunteer_NDA, please get your manager to comment and approve

Why can't this be tested on any other beta wiki or patch demo?

https://gerrit.wikimedia.org/r/c/operations/dns/+/919346

See also T295221: Clarify Puppet tag

Herald rule is {H389} from T285143: Create project tag for SRE Infrastructure Foundations

No longer needed (see parent task and #wikimedia-sre)

Hardware log from @Dzahn:

In T292505#8816010, @valerio.bozzolan wrote:

As already mentioned by Aklapper I think this is too much controversial in Wikimedia but I see this could be useful.

The main problem I see here, is that Wikimedia disabled the possibility to change "Editable By".

So if a Task has massive spam from trusted contributors, we cannot just Edit → Editable To → Task Subscribers (4 clicks, 6 seconds maximum).

In T335626#8815416, @Krinkle wrote:

This bugfix would be important enought to backport. But, if I'm seeing this right, the root cause (change 902537) has not yet made it into any release branches, as such resolving without backport. Feel free to re-open to create backport cherry-picks if I missed something.

Note: Extension loads fine in 1.40 so it's something not on 1.40

It's also possible to create new groups I believe, but have never done that.

The hardest part is coming up with the name

@Dzahn: is this expected? The phorge install doesn’t seem open so I can’t tell if it was migrated.

Someone with access who will watch the tag

TNT isn’t the only one who can handle this

Should this be updated to say 1.41 now (or a realistic target)?

This is being investigated. Deployers are aware.

Fallout of recent deployment

@Cmjohnson: @nfraison said to wait on the other task.

This task was split from T330971: Degraded RAID on an-worker1132

Per other task, doesn’t seem actually needed.

I’ve had a look and I don’t see any privileged access nor am I sure the user was created by ITS and there’s no -ctr email.

A combination of ITS (mainly for WMF staff) and SRE process will result in it. A member of SRE will be alerted when the date SRE were given to expire the access (or when they are informed otherwise) and they’ll run a script that disables accounts and helps them removes access (some of it automatic, some of it prompts).

In T328361#8571672, @mpopov wrote:

I suppose we could also have aliases for the bastion hosts so instead of connecting to bast3006 users can specify bast-esams (which would actually be a huge improvement) but geolocating would support changes to data centers.

You’d then get a scary warning about a key mismatch when the server was changed.

This caused an error - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cargo/+/884344

Wrong task

Why do you need grafana access?

If you are seeing session related errors, there is no need to add drive-by comments with the message or saying you are.

Can someone from SRE or releng please look to facilitate an emergency deploy?

Lowering to deal with on Monday/Tuesday and updated description. Thanks everyone for the response.

P43154 has a draft IR if needed, collates times and actionables to save any SRE time.

Following hosts are down:
cp2031
ms-be2046
elastic2041
kafka-logging2002
mc2043
thanos-fe2002
elastic2063
cp2032
elastic2046
elastic2057
lvs2008
elastic2077
elastic2078
mc2042
ms-fe2010
ms-be2041
ml-cache2002
elastic2042

Not my say but I very much doubt they’ll be an issue, just helping speed up the process by asking the questions that the Phabricator team will ask.

Is it likely to be something you only do once or do you see a need to do it in future?

One thing I will point out is that the batch edit tool on the UI is not silenced. You require shell access for that.

Can you please give a reason?

@Mutante: adding as IC, can you please let people know when the incident report from last night is ready?

In T301505#8507048, @AlexisJazz wrote:

I just received one trying to look up https://en.wiktionary.org/wiki/stuff. Not reproducible it seems.

See T326495: 2023-01-08 Wikimedia Connectivity Issues

In T326495#8507147, @SD_hehua wrote:

Emmm not a duplicate?

This was resolved, SRE are aware of an issue around this time.

Reclosing to correct error