Got it. It wasn't always the default, but it has been for at least a decade. I haven't been able to find why (and what would be the harms of enabling it), but I documented this practice/issue here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Caveats

Just tried a restart while monitoring the following query:

Hey there! Sorry for the inconvenience, going to have to plead ignorance on this one. Just curious, how long has this policy been around?

Great, thank you for the resources and tips! Will circle back if there are more reports of issues. Thanks again!

Yeah, after a few days of monitoring, there's definite improvement.

Yeah, the Python 3 backend is already deployed over on the montage-dev tool. Peak traffic isn't an ideal time to migrate.

Actually, I just did a closer read of the docs, and Montage isn't using all of its allocated pods. I just bumped it up myself and will monitor performance.

Odd. L2W is back up after a couple restarts. Kiwi I'm less sure about; I just picked it because that's what I saw being used/linked from https://meta.wikimedia.org/wiki/IRC/Channels. Figured it was the blessed successor to webirc.

Ah, that makes sense. Maybe the help strings can describe that format, the default, and/or link over to that page?

You're tellin me! 😭 I've got a branch or five dotted about, we'll get to 3 soon.

Hi there, just wanted to share that I worked around this issue in the py2 web situation by switching to PyOpenSSL, which brings along a newer version of OpenSSL. The changes were pretty minimal and can be seen here: https://github.com/hatnote/montage/commit/1be5d09ff5b80e2a57eb71802096fc1fcb98e60f

Hmm, so I guess I could see some users relying on this behavior to get very rudimentary batch processing. I.e., refresh their browser until all the processing is complete. Is it maybe possible to enable it for one app's db at a time? Or does it really have to be all of ToolsDB?

Hey there, been about a month since the last update. Was curious if @Marostegui or anyone else had given some more thought to maybe increasing the ACIDity here? We're still getting pretty regular integrity errors on those Wiki Loves campaigns.

Hey there, just wanted to drop a note that we've (cc @Slaporte) started on migration work for Hatnote projects (montage, monumental), but didn't get around to finishing it. The June 11 mention was the first I heard of it (thanks @Lokal_Profil otherwise I might have heard from a user), and we put it on the backlog for this past weekend.

Hey @dr0ptp4kt! Not sure if we've met, either, but nice to e-meet you here. This is for _a_ Hatnote project, but not one of the well-known ones. Here's a permalink to some Python in our Pacetrack project. I guess the main thing @Slaporte and I could hope for now is another straightforward endpoint where we could get similar citations?

Seems like this deprecation already went all the way through? We found out through breakage (e.g., https://en.wikipedia.org/api/rest_v1/page/references/The_Register-Guard/957644254 )

Ah, sorry, yes, ToolsDB, provided ToolsDB is what the developer-writable databases provided on toolforge are called. It may affect other databases, but we only write to ToolsDB, so we couldn't ask for any more. :)

Hey all, circling back again. Any news? Hope you're well!

Hi all, just checking in to see if there are any updates. Let me know if there's something I can help with.

Well, because it's an interactive page, if we can't get a lock within a reasonable amount of time, we don't have much choice but to roll back the transaction and request that the user try again. A partial commit really isn't an option for us.

Interesting that they count against the same quota, I thought accounting was per-session. But even so, I'd expect the limit would at least reliably allow for one shell + at least one kubectl get pods. And it's still kinda spotty, I wonder why it's nondeterministic.

I just reproduced the issue by doing webservice python2 shell in one session and while true ; do kubectl get pods ; done, in the other. I don't have pstree on tools-sgebastion-07, so maybe other things are different?

Ah, it worked the first few days, but I just ran into this:

Cool. I'll try this for a few days and let you know if I see the issue.

I understand completely. It's a shared resource and such a system makes a lot of sense. That's one reason why I avoid customization on such environments. I'm using a bog-standard workflow of

$ ssh mahmoud@login.tools.wmflabs.org
$ become montage
$ webservice python2 shell

Still getting this fairly regularly. Anything we can do on our end? Not sure I have reproduction steps, except this happens about half the times I try to shell into montage.

Ah sorry, to be clear, using GOMAXPROCS appeared to fix the kubectl crash I
was getting, but only at first. Eventually, kubectl would crash no matter
what env vars or ulimit I set.

Running into this, GOMAXPROCS=1 worked at first, but I kept needing to adjust it higher and higher (GOMAXPROCS=2, etc.), and eventually no more dice.

Ah, makes sense, thanks @Anomie, @Nirmos, and @Ciencia_Al_Poder! Not sure if you want to add that either to the rc_bot part of the recentchanges table docs, but that's good info! Thanks again!

Hey all, just refactored a couple tools to work with the new schema, wasn't too bad. Sad about the performer thing, though I have to admit I'm more used to "actor" from other codebases.

I've got a version of the code going off of the rest_api now. The code's not running exactly the same, but that could be on me.

Looks like a combination of techniques on the linked pages has got me unblocked. FWIW, I also had the "version: 2" manifest.

It would be nice to have both execute time and combined execute + fetch time, as the latter more accurately represents the an application would spend waiting (which would really help with prototyping). Also, thanks for this!

fwiw @bd808 I put the -once in. Not sure if that's enough info to close this ticket, but hopefully we won't see this happen again. Thanks!

Sorry for the delay in replying, wedding planning has done a number on my open software contribs.

Yeah, sorry about that. This was due to us attempting to re-enable the hashtag search on IFTTT. Turns out it's a very popular feature! Too popular unfortunately. We've wired that off again, so if we can go back to having a regular limit that would be nice.

That's an interesting idea. It would really help to have answers to my questions if we are forced to register an account per language.

It's hard to say what can be done about this without more information. There's user-generated load from visits to http://tools.wmflabs.org/hashtags but there's also batch traffic as well. Both are necessary for editathon folks to get their data.

Ah, yes, I expected that the issue was related to the Ubuntu upgrade, I just didn't expect it to manifest as datetime being unimportable. I seem to recall the email saying a service restart was all that was necessary to upgrade, but things are never that easy :) I'll make a note to check the rest of our virtualenvs, too. Thank you everyone!

Meant to post this earlier, but great work @MusikAnimal! I'm eager to see this codified into some sort of anti-spam correction, but I'm concerned by articles like "Oxford Manifesto", which also have <0.1% mobile. Though on second thought, the page does look a bit anomalous to be ranking so highly.

@yuvipanda Thanks man :)

Hey all, just got back to my laptop, the service appears to be running, no 500s for me. Maybe it was something transient related to maintenance, or @chasemp's restarts fixed it with some delay. Is it working ok for you Ciell?

Are there any other details whatsoever? Like, do they get to the Wikimedia screen? Are they redirected back? Can we get a screenshot of where they get stuck?

Wait, so is it 50 or more than 50?

Also, yesno round -> rating round advancement is tested working in the backend. I'm onto ranking stuff now, so I think we have a healthy head start.

The logic is largely the same. Montage's streamlined approach is that the winners of one round are the entrants into the next round. In addition to the simpler user experience, this avoids all sorts of process issues that could result if there is a phase between rounds. We couldn't think of any interactions that Montage could provide between rounds that it also shouldn't want during rounds (adjusting jurors, etc.).

(Not sure if I added the right tags, sorry for the spam.)

So the "name" parameter here is "توضیح" which technically means "description" if you ask me.

So it seems like we have a fix for T139258, both in Lua and Python, but we haven't seen any changes on the Unused images page. Has the fix been deployed and/or has the job been run?

The Lua looks pretty good, too, but @Lokal_Profil is there a reason the Arabic numbers are written rtl while Persian is ltr? For all intents and purposes, to the best of my knowledge, both languages have identical number semantics.

Reviewed the Python, looks good! I'll take a stab at reviewing that Lua script too, but I doubt I'll come out unscathed.

In response to @LilyOfTheWest's issue, this does seem like something that might be caused by an incomplete fix to T139258: Figure out improved matching of monuments for Iran. Other examples include 944 aka ۹۴۴. The unused images page has an image, even though it's filled in on the associated monuments page.

Cookies have expiration times built right in, so you can easily set a sane default of, say, 48 hours. Hashtags are also short, clear, and trivially deletable. Ideally we get this launched, and if there are reports or if it seems to cause confusion, a priority can be set for additional UIs.

So, did we make it onto the backlog? The #artandfeminism editathon is back again this year and their efforts reminded me. Hope we can have something for the next editathon organizers!

Hey again @jcrespo, we made some code changes and would request that you or another DBA throttle the number of database connections up to 20 or so max so we can confirm the fix.

Cool, I'm also up at 2:20am still looking at this, so I think you can assume I take this seriously, despite my attempts at a friendly and community-driven tone. Our tools also have hundreds of downstream users.

Also, is there an idle timeout or recommended connection keep alive time?

Haha, I'm sure some would argue 338 is small potatoes, but I'm not one to argue.

OK, Stephen and I both have work tomorrow and nothing has jumped out at us, bug wise.

Ah, if it's just that pattern, it's probably the same issue just magnified across all the languages. Stephen and I will look into it, thanks!

Also what are the headings to these columns in the issue description?

Can you expand upon the issue at hand? Do you know when it started or how similar the in-flight queries are/were?

+1 ! :)

Oh, sure, so the design as I understand it is to make it so that editathon coordinators can distribute a special link that would associate the clicking user to an editing group. Users associated with that group would, for the duration of the browser session, have a hashtag prepopulated in their edit summary.

Hey all! Glad to see this discussion made some good progress without me, so thanks to everyone for getting the ball rolling.