Ive reverted the unilateral change in policy about source code publication. Providing a OSI license with the code has met the toolserver and labs/cloud requirement for the last 18 years. There are reasons to not publicly publish code, if you want to change the policy this is not the correct way. And Im not going to debate it in a ticket either.

The code should be publicly accessible

Per RFC 2119

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

I have reviewed publishing my code and there are a number of valid reasons for not making it accessible to the world. I am not going to re-hash them.

1. I dont use Git, I use SVN
2. The code is hosted in a private repo to avoid leaking of credentials/non-public information.
3. None of the listed WSGI/Flask/FastAPI are the current CGI format that is used on labs ATM

@Tgr Not sure if cleanupUsersWithNoId.php will actually do anything, but I figured Id document everything.

I am not on 1.39 yet, I would NOT call this already documented. the upgrade.php should either invoke the cleanupUsersWithNoId.php or should not run. failing to do so is a destructive upgrade, that multiple users have encountered.

Spent the better part of the day rolling back those changes, after restoring both the original database, and directory structure, I updated git checkout to REL1_35, updated composer, ran cleanupUsersWithNoId.php, then followed that by update.php.

You are correct I was referring to the meta repos. Normally you just set the REL correct and check out the entire repo, then enable extensions on an as needed basis. Avoids having to do 10+ separate checkouts.

rel1_39 was only branched on CORE, not skins or extensions. and there hasn't been a REL branch for those since REL1_35. When attempting to update to the most current LTS it is not possible to match CORE checkout and extension/skin checkouts. This will cause compatibility issues at some point.

I tried moving to K8's when they were first introduced. It went about as well as a cat in a room full of rocking chairs.

Either way, commons does need a reparse for LARGE number of pages. I took a look at anything with a date before 20151106000000 and it returned 1,652,571 pages and going back just a year resulted in 10,722,335 pages. With the improvement of the parser and introduction of a number of tracking categories we should be really parsing pages at least 1 a year if not every 6 months at a minimum.

Ive been following this for a while. How does this interact with SUL? is the login action only tracked on the home wiki, or all wikis you visit?

Also I think you have to do it as a tool, not as yourself

In T216208#4959184, @Tagishsimon wrote:

Day 2 of no Quickstatements, no Listeria, no ETA for a resumption of service ... no comment from WMF on why what is a critical service - if we're taking this whole wikidata business at all seriously - is fritzed.

Looks like this is still happening.

Not sure why T117801 was merged. It was about exposing the registered email address for an account, not user to user emails. (useful for cases where the same email address is used to register multiple accounts)

In T183647#3861456, @matmarex wrote:

The real bug here is that WikEd converts normal non-breaking spaces to their HTML entities. This behavior has been known for years. I'm actually surprised no one has gotten WikEd removed from en.wp for "breaking" thousands of pages like this over the years.

In most wikis the use of non-visible non-breaking spaces is discouraged. What I think the real issue here is the spike in non-breaking spaces being added in the first place. I think all wikied is doing is highlighting the root issue, not that wikied is the problem.

In T183090#3843370, @zhuyifei1999 wrote:

In T183090#3843358, @Betacommand wrote:

there is an issue with sslv3 in older versions of python which has been fixed in .8 or .9

You mean PEP 0466?

there is an issue with sslv3 in older versions of python which has been fixed in .8 or .9 which is why I was investigating k8s

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Using_virtualenv_with_webservice_shell

The other thing you can do when working with external links is to use the linksearch feature (Be sure to check for both http and https versions of the URL) Ive been using it for years and should be a lot more reliable than using the search index.

Ive done some large scale processing but needing more than 4GB of ram is a sign of bad programming. fix that first, dont try to come back later to try and reduce usage then, its too late

Would love to see this feature implemented. With large watchlists people disable the bolding/unbolding for readability purposes. Being able to set a timestamp and only get changes new than that also helps when edits are grouped by page. As of now its manually checking the load timestamp of your watchlist, refreshing and then finding the previous time and reviewing changes since then.

In T162884#3182781, @Paladox wrote:

@Betacommand hi, could you try removing the ones that is blocking you and give a full list of the ones to be removed please?

PipVideoJs

why not just keep a continuous thread going with a sleep(60) that way you minimize the number of jobs and don't loose functionality.

and PhpTagsDebugger

and Notifications

looks like DetectLanguage is also throwing the same issue, and I suspect there will be a half dozen or so others that need fixed too.

In T160916#3115802, @JEumerus wrote:

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

@BurstPower here is a list of all pages in enwiki's mainspace minus redirects: http://tools.wmflabs.org/betacommand-dev/reports/en_articles.txt

Sorry if that wasn't clear in the initial post. The reason I was thinking of a combined total was to include the JobQueue as part of the value so that it was a representation of the current server lag. Returning the max of either would also work.

I havent looked at the actual code, but the formula looks correct. My thought would be in getMaxLag() to call that formula and if it has a positive value greater than 1 add it to the maxlag total value. So that the final Maxlag value would be ServerLag + JObQueueLag = MaxLag

Most bots use a maxlag check of 5-10 seconds, if we pick a value for each wiki based off average jobqueue size and for every X over that value add a few seconds to the lag param. We would want to scale the check based on wiki size. Take enwiki (These are random numbers not fact checked) has an average JobQueue of 1 million items. for every 100,000 over that add 1 second to the value of maxlag so that when the JobQueue reaches 150% of normal they back off until its down to normal levels.

In T159618#3080947, @Legoktm wrote:

In T159618#3079361, @Esc3300 wrote:

I think it might be worth attempting to determine the factors that lead to the rapid raise.

• The edit rate didn't seem that high and we could easily have several bots that resulted in the same rate.

I disagree. According to https://wikipulse.herokuapp.com/, the edit rate on Wikidata right now is anywhere from 150-220 epm. Adding an extra 600 epm on top of that is what caused the problem.

Suggestion, when the job queue gets too high set the maxlag parameter to a higher value, most bots use that as a throttle.

I would say this is functioning as intended all /wiki urls are in the short url form. Short urls do not handle any URL params. index.php is the correct access point for URLs that need params passed.

I know jsub already has an option to log to a specific directory (Ive been using this for a while to avoid sge logs flooding my $HOME perhaps setting a global option using a username variable?

It was every six months and happened via a login/script going to 4 times a year would be kind of obnoxious for most. I have things configured so it is mostly hands off for a reason.

Quite, would help tracking down the remaining non-wikidata entries and addressing/fixing those.

I saw a note that commons was working which is why I asked

Which wikis are currently affected?

@jcrespo Ok, looks like we have been on two different pages. I thought you where trying to make this a labs only "feature", where as you just want a proof of concept/viability. I know you are doing quite a bit to improve labs quality of service, and this wasn't a dig on that. Rather it was planning for the issues that labs has historically had, and trying to limit the long term impact of those issues. If you want to get it implemented on labs go ahead. Ill reach out to my Wayback Machine contacts and see what the best process moving forward would be for mass archivals.

In T74931#2514759, @jcrespo wrote:

You also say:

that would be to keep an offline duplicate of the el table and compare it during each run. (which eats a LOT of disk space, CPU, and time)

But you suggest that keeping that in real time on >100 production redundant instances will be more optimal than keeping it only on labs, or on a subset of production separate instances, away from the main dbs?

Production issues may occur but getting those fixed is fairly easy, either purging or null editing the page in question is enough. Labs drift on the other hand requires a full re-import (which would loose all of the custom timestamps added over time).

@jcrespo Nothing against labs, but T138967 is a constant and persistent issue which would make the data useless upon the next re-import. Keeping this in mediawiki allows more consistent data and fewer chances of corruption. Yes page blanking vandalism would cause minor variances but isn't a significant issue in the big picture. Yes having a complete historical reference might be nice, its just not practical at this time. The category links table already has the same feature that I'm looking to add to the el table and allows tools like http://tools.wmflabs.org/betacommand-dev/reports/CATCSD.html for patrolling admins

One of the most useful use cases is for having a bot that feeds URLs to archiving services such as the wayback or webcite in a manor that is actually feasible. Right now the only way to do a task like that would be to keep an offline duplicate of the el table and compare it during each run. (which eats a LOT of disk space, CPU, and time) Having a incremental based reference point would mean that only those links newer than the last run would need to be processed, and that when retrieving copies of the archived version of a link we have a time reference to work with without needing to parse the history of the article.

Wont happen, the amount of cases where sensitive information is contained in deleted revisions prevents mass automatic access.

In T138684#2407282, @ArielGlenn wrote:

Is the template itself in enwiki-20160601-pages-articles-multistream.xml.bz2? If it is then you should look at whether wikidata is used as the source for the de page or whether it's in the wikitext. If the templat eitself is missing entirely from the pages articles dump then the next step would be to see if the stub for it is present in the stubs dump.

Recommend closing as invalid. the information that the user wants is being pulled from wikidata via module see https://en.wikipedia.org/wiki/Template:Authority_control

Not sure how I got listed as a maintainer, I am not involved with this project.

In T49505#2212659, @Glaisher wrote:

Is this for "Get users" only? IMO, Special:CheckUser is badly in need of a UI revamp.

I would say 2 total messages, one for IPs one for users. If further customization is needed the messages can be updated on wiki to tweak the output.

In T49505#2203209, @MarcoAurelio wrote:

Is there an easy way to do this?

In T126607#2182557, @Krenair wrote:

In T126607#2182537, @Kharkiv07 wrote:

Can we keep automatic promotions out of the user rights log (similarly to the way that autoconfirmed is silent); it'll be a far too common event that spams the log and watchlists.

No, that would need a change to MediaWiki core itself.

In T124172#1948028, @jcrespo wrote:

Replication lag of 10 minutes

Are you talking about the production replica databases or something else?

Yeah, sorry for the delay in getting back to this, I have a dump from a few months after this, but it doesnt look like the revision is in it.

I might have a few tricks for recovering this revision, give me a day or two Ill see what I can do.

In T118932#1846871, @demon wrote:

In T118932#1846869, @Betacommand wrote:

I think that 1.27 should ship with 5.3 or 5.4, and then move to 5.6 after the LTS, Given how difficult it is to get third party hosting providers to update php we shouldn't use our LTS to move to a cutting edge version. Lets use 1.28 for that.

I would hardly call PHP 5.5 "a cutting edge version." It's been out for 2 years and 5.6 is already out and 7.0 is just days away.

I think that 1.27 should ship with 5.3 or 5.4, and then move to 5.6 after the LTS, Given how difficult it is to get third party hosting providers to update php we shouldn't use our LTS to move to a cutting edge version. Lets use 1.28 for that.

`

tools.betacommand-dev@tools-bastion-01:~$ mysql --defaults-file="${HOME}"/replica.my.cnf -h labsdb1001.eqiad.wmnet plwikisource_p -e "SELECT count(page_id) FROM page WHERE page_namespace=100 AND page_is_redirect=0"
+----------------+
| count(page_id) |
+----------------+
|         241003 |
+----------------+
tools.betacommand-dev@tools-bastion-01:~$ mysql --defaults-file="${HOME}"/replica.my.cnf -h labsdb1002.eqiad.wmnet plwikisource_p -e "SELECT count(page_id) FROM page WHERE page_namespace=100 AND page_is_redirect=0"
+----------------+
| count(page_id) |
+----------------+
|         241003 |
+----------------+
tools.betacommand-dev@tools-bastion-01:~$ mysql --defaults-file="${HOME}"/replica.my.cnf -h labsdb1003.eqiad.wmnet plwikisource_p -e "SELECT count(page_id) FROM page WHERE page_namespace=100 AND page_is_redirect=0"
+----------------+
| count(page_id) |
+----------------+
|         240973 |
+----------------+

I know this is more work, but I think the community would prefer that you do a fresh import of the data. Having an unknown volume of data corruption in a table is not a good idea. Starting with a fresh copy would ensure the reliability of the data in question.

Yeah, http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag is showing 28047 behind

I must agree with the others, the "improvements" that where made are not good. Depending on activity I can switch between the 1,2,6 Hour options or even 12-24 on a regular basis. What was 1 click is now 3 each time..... Why are you making the UI more complex to use?