I'm just putting this here for future knowledge (or future onlookers) and to explain somewhere why I took the action to delete the elwiki page.

In T357225#9622429, @abi_ wrote:

You are right JSON is supported, but certain elements like Laravel pluralization format (https://laravel.com/docs/10.x/localization#pluralization) is not supported.

@abi_ yes my intention is to get them switched over, but I will need some time to do so. Due to multiple factors it will take time - possibly a week or two depending on when I can find time. That said, the banana-i18n format would be a complete departure from the existing setup and require more man hours. I would prefer to switch to json as I think it is natively supported, and I'm reading translatewiki also supports it. I will move forward with that as soon as possible, though it also may not be on the main branch until the end of the month given a number of other features being deployed alongside in UTRS 3.0.

@Wangombe Sorry, I forgot to mention, but yes, that's the intention and has been when I read the requirements. I was just holding off on till I knew if this was a full go unless you need it switched now. It's an easy fix, just let me know.

Grid engine tasks have been disabled.

@nskaggs @komla I remember an early on notification about this, but I was not tagged/subscribed here. Only found out this past weekend about this by a fellow steward. I will need to take time this weekend to resolve this and have added it to my must-action board, if you could not include me in the Dec 14 service disruption please. I will see any notifications of replies here as my email is attached.

Verified working. Thank you

@Dreamy_Jazz As this may have been related to a recent patch

I'm disappointed that things were reverted here without followup with something in it's path, or just dealing with the analytics concern separately. Now we are at the point of starting to see the user agent reduction and it's going to impact us very hard and very fast. We will likely now how to do mitigation range blocks as we don't know who the heck is on our ranges. Beyond that, I was waiting for what WMF was going to do with client hints in order to adapt UTRS, and I think @stwalkerster was along the same lines for ACC. Now we have to see if we can get someone's attention before all the LTAs (especially a certain few) pick up that this is deploying. Going to be a rough time while we wait for re-deployment.

In T329251#8603276, @bd808 wrote:

@AmandaNP, does the custom contentadmin group on wikitech have all of the sysop related rights that would be needed? I believe that contentadmin grants everything that sysop does except editinterface although there may be some rights drift if sysop has gained new global permissions since we invented contentadmin.

It's been a few days, and I've only gotten 1 error from an LTA related case...but I've found a way to debug more, so I will follow up again in a few days.

Unless I'm really out to lunch - which I don't think I am - this bug has reappeared.
I tried to remove importer and transwiki-importer from Addshore@testwikidatawiki via Meta.

I have filled https://github.com/UTRS2/utrs/pull/609 which seems to be the sum of the issue of async vs sync
I'm going to pull the debug config especially since it didn't even seem to work.
I'll see if any more issues of this occur in the coming days.

Should this need to be reverted in an emergency, please feel free to have someone go on utrs-production and navigate to /var/www/html/vendor/addwiki/mediawiki-api-base/src/Guzzle/ClientFactory.php and remove the header from line 53.

I should be able to modify the vendor files for addwiki's API without affecting the end user to enable the debug header. I will also see what I can insert in terms of variable debugging locally. This only happens a maximum of 3 times per day, if that, as it is intermittent - so it will only spam if I leave it for a few days. I will try to work on this in the next few days and get back this ticket when it's on, and then again after 3-6 failures.

@TheresNoTime @Urbanecm_WMF
Can I get a copy of what you can from logstash for 2023-01-17 07:49:36 on meta to try and figure this issue out. I'm continually getting Bad CSRF token, but can't diagnose it because I can't reproduce it anywhere other than the automated job that runs this.

I checked and verified doesn't matter if the user was actually previously blocked or not. Issue might also only occur when globally blocked as mine is both. Still occurs in all forms. Still waiting for my usual enwiki checks to come up to provide an example for CU.

So I did a deeper diver now that I have access to the error logs. The Email errors have been showing up at least since October, that's as far as the logs I have back go.
The AssertBot was only for the time period mentioned, so ignoring that until it returns.

So this, unfortunately, has started again for the 2nd part (CSRF token).
First occurrence on: [2022-12-27 18:23:30]
and is occurring regularly again

I just tried to add CU to myself at testwiki, and I can confirm that I get the same error.

Sorry, I was dealing with medical stuff at the time you pinged. Yes, the NFS server/volume can be deleted.

I apologize for missing this when it happened. I talked a bit with bd808 and I don't need the NFS volume.

I have finished deleting the instances. Closing this task.

Our instances are already shut down, just were being kept in case we needed a backup of things. It seems like things went smoothly, so we should just be able to delete them in another week or so to be safe.

*!*@ wikimedia/bot/wmopbot +Aeiortv (OP) [modified 8h 22m 19s ago]

Full rotation of all 5 BotPasswords has been done too as a preventative measure.

The OAuth grant has just been revoked. It has not been renewed since it's not in use.
Session log out has forced the two cookies to be reset.
I'm going to proceed with a full rotation of logins in an abundance of caution.

In T274050#6862681, @Josve05a wrote:

@DeltaQuad Seems to have blocked the IP on enwp https://en.wikipedia.org/w/index.php?title=Special:Log/block&page=User%3A185.15.56.1

@Prtksxna @Niharika How important is highlight on keyboard focus vs the ability to select text?

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

I thought I would mention there is now an Upstream request for mwclient
https://github.com/mwclient/mwclient/issues/256

In T249408#6030438, @Reedy wrote:

Bot username, if applicable (The "@whatever")

This is already in the subject, right? Just not in the body text?

As mentioned above, UTRS does need this also on both:
utrs-beta.wmflabs.org
utrs.wmflabs.org

Everything except for the home one can go. We are only using that one.

Thanks, this worked!

@Krenair I don't see where we would require it, but I'd like to have things copied over from current home directories at minimum to make sure nothing crashes.

@Krenair I have finished moving them onto 2 small instances instead of mediums and deleted them. You can close this when you are done re floating ip.

Yes, I just didn't see that I had to switch projects. I see it now.
We are just waiting for the floating IP to re associate to utrs-production2 instead of utrs-production. Not sure how long that will take.

@Krenair So to be clear, there is no more ubuntu available? And therefore the entire instance needs rebuilding?

Fair enough that it's scope creep to add it to that. I meant more just working along at the same time. But I get that foundation priorities are on the wishlist and we have to make due with what we got. I just hope this doesn't totally drop of the radar because one part of the blocking thing has been fixed, and then this waits again.

It's really sad that this can't be integrated within T2674 as its the same form. You don't understand that as a CheckUser, we could be blocking millions less users from editing the site with this feature, allowing more people to edit. That and it would help reduce backlogs at our account creation tool ACC, where checkuser review hovers around 100 on a daily basis because we can't do this.

@jcrespo Ya I waited a bit last night, and it still wasn't going. It works fine this morning. Like you say it might just need a proper error message, but figured if i'm hitting it at 10 in the morning UTC, for a long while that something seemed a little off.

I apologize for the delay on this. It was scheduled a while back, but other critical errors and my work schedule prohibited this from happening. Right now i'm making the transfer.

In T143349#2965177, @chasemp wrote:

In T143349#2962963, @chasemp wrote:

Greetings @Crazycomputers @DeltaQuad @TParis from https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs

Do you have plans to convert this instance to Ubuntu Trusty or Debian Jessie? We will leave it as long as we can but by the end of March our hands are tied on shutting this down for security reasons. We can bump up project quotas if that helps to migrate configuration and such. Please let us know :)

Utrs-primary.utrs.eqiad.wmflabs

In T143349#2963953, @TParis wrote:

I'm in the middle of a move from Hawaii to Texas, Deltaquad would be the
best poc. But if it isn't moved by March 1st then I'll take care of it.

Vr
TParis

Thanks @TParis, I'll mark it as such

@Andrew It appears recreation was successful, although the number of notices on the console output is crazy. Is that to be expected?

As mentioned on IRC, I have no clue. The option was not presented in the setup options as far as I know.