Turns out flickr retired the secure.flickr.com domain without forwarding. Removed the "secure" but still using https, should work again.

Also, doesn't seem to work for "has a birth date"?
https://www.wikidata.org/w/index.php?search=haswbstatement%3AP21%3DQ6581072++haswbstatement%3AP569&title=Special:Search&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns120=1

In T179879#6859731, @So9q wrote:

Both of these can be done using CirrusSearch API and can't they?

Restarted wikidata-todo.

It does work, however, since there is currently not a single page on euwiki where the latest revision has a wp10 score, the results will be empty.

Thanks, I think I fixed it now.

Should restart now

On it

I would also like to point out https://tabernacle.toolforge.org/ which ATM is for editing existing items, but I could easily create a new blank item to edit.

Just saw this one. I think fixing wikibase to add to watchlist is the way to go. I could update the OAuth permissions but that would break everyone's server-based edits as they would use an old OAuth, correct?

Should be fixed now

I'm on it.

Worked with 10K batches. I still think this rapid disconnect in the middle of a query is a bug.

Trying 1M chunks, first two worked, but then:

mysqlimport: Error: 2013, Lost connection to MySQL server during query, when using table: osopenuprn_202006

I'll try even smaller ones, but it's getting a bit ridiculous. Why disconnect the server in the middle of a query?

I have switched the configuration to use the petscan db connections only.

Just to add, if some queries take too long I can likely just change a parameter to fix that. Someone with access to the stats, let me know.

The reason I use that many connections is precisely to avoid long-running ones, as I had in a previous version; they would time out or lose database connection, so I re-wrote the code to use more but shorter queries.

In T243848#6251382, @Aklapper wrote:

@Magnus: Does that mean that https://phabricator.wikimedia.org/source/tool-quickstatements/browse/master/ will get out of sync and should be archived?

Yes.

I have "forked" it into github. Using diffusion was a mistake in the first place. Now at https://github.com/magnusmanske/quickstatements

Tried again with ssh://vcs@git-ssh.wikimedia.org/source/tool-quickstatements.git, no joy:

Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.

tools.quickstatements@tools-sgebastion-07:~$ git remote remove origin
tools.quickstatements@tools-sgebastion-07:~$ git remote add origin ssh://phabricator.wikimedia.org/diffusion/2010/tool-quickstatements.git/
tools.quickstatements@tools-sgebastion-07:~$ git push fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

FWIW, I believe all my (active) tools now use the new canonical URL.

As I wrote five years ago, I added a limit to upload only two files concurrently.
No further comment has been made since, so I consider this issue to be solved.

Thanks, that seems perfect!

In T254640#6202698, @bd808 wrote:

This config is part of the default configuration generated by webservice. That means we need to fix it upstream of the usage by this particular tool.

OK thanks

As a horrible short term fix for reasonator, I did:

$ ln -s $HOME/public_html/reasonator_types.js $HOME/public_html/_types.js

In T254640#6201620, @LucasWerkmeister wrote:

I don’t understand why this is an UBN when as far as I understand we know a workaround (restart without --canonical)?

In T194953#6185511, @bd808 wrote:

Is there a movement towards Rust in any significant segment of the Wikimedia movement? Are there clear benefits for Toolforge tools to be written and maintained in Rust?

I think this is a blocker to moving all tools to the canonical URL. T234617 ?

I cleared the error log on reasonator and switched to the previous URL schema, then back to the canonical one. No server errors in the log (some unrelated PhP ones, now cleared again)

@LucasWerkmeister thanks that worked, for now

Same with https://geohack.toolforge.org/geohack.php . Geohack is kinda central for Wikipedia so please help!

Thanks @bd808 !

Other files load fine. Haven't checked all though

URL is https://reasonator.toolforge.org/reasonator_types.js

Restarted it now.
I am also working on a watcher script to restart it automatically when it goes down.

It's my dev server, should work for any query size. Feel free to use when the main site is down.

@Aklapper actually the repo is https://github.com/magnusmanske/petscan_rs (the bitbucket one is the old C++ version).

There are some PetScan outages, reason unknown so far, but outside those, the above queries all work fine.

Still happening.

Maybe this helps:

Yes, still seeing it. Runs for ~20sec so probably not a cache. Changed the JSONP callback(s) and still same error, so definitely no content-level cache.

It appears that the third-party bot framework quietly swallows this and just throws an exception with the "info" field :-(

In the error I mean

Hmm, I think what tripped my code is that there was no code:'maxlag' in the response?

Seems to clear up now

But it's all kinds of wikis that show up. Something with the dispatcher?

QuickStatements edits as the user who submitted the job, using OAuth, as is the WMF-preferred method for tools.

Should now be fixed with https://bitbucket.org/magnusmanske/listeria/commits/5b53c94ffb10

Getting this error for mix-n-match tool.

@bd808 Thanks, now running quickstatements as two repilcas

Update: Setting server.max-request-size = 1000000 in $HOME/.lighttpd.conf and restarting the webservice did not help.

In T240316#5728232, @Lea_Lacroix_WMDE wrote:

@Bugreporter That seems unlikely. QuickStatementsBot is blocked since October 28th, and daily users of the tool only started reporting this issue yesterday.

For debugging:

It's not a stale cache on my side. If I get different properties from https://www.wikidata.org/w/api.php?action=help&modules=query%2Buserinfo I get the requested properties but still the bogus block.

If I check the API directly in the browser, it doesn't show. Maybe because OAuth login?

I seem to be getting the block for [[User:Doqume]] on Wikidata, but for every user

I think I found the reason. Using MW API to get user info, I get:

{
  "id": 4420,
  "name": "Magnus Manske",
  "blockid": 15320,
  "blockedby": "Mahir256",
  "blockedbyid": 203574,
  "blockreason": "আপনাকে স্বয়ংক্রিয়ভাবে বাধা দেওয়া হয়েছে, কারণ আপনার আইপি ঠিকানাটি সম্প্রতি \"[[User:Doqume|Doqume]]\" ব্যবহার করেছেন। Doqume-কে বাধাদানের কারণ \"Freebald-ish behavior\"",
  "blockedtimestamp": "2019-12-09T16:59:04Z",
  "blockexpiry": "2019-12-10T16:59:04Z",
  "groups": [
    "rollbacker",
    "*",
    "user",
    "autoconfirmed"
  ],
  "rights": [
    "autopatrol",
    "editsemiprotected",
    "move",
    "autoconfirmed",
    "skipcaptcha",
    "abusefilter-log-detail",
    "suppressredirect",
    "read",
    "edit",
    "createpage",
    "createtalk",
    "writeapi",
    "translate",
    "item-term",
    "property-term",
    "item-merge",
    "item-redirect",
    "abusefilter-view",
    "abusefilter-log",
    "flow-hide",
    "reupload-own",
    "move-rootuserpages",
    "move-categorypages",
    "minoredit",
    "purge",
    "applychangetags",
    "changetags",
    "reupload",
    "upload",
    "flow-edit-post"
  ]
}

But I am not blocked; editing works fine.

I can't create batches either, so good for testing!

I did not touch QS for weeks. No idea why this is happening.

Fixed now.

In T239036#5715194, @ROOTxDEAD wrote:

Hello Team,

I am still able to have a XSS vulnerability in the same website. Should I paste the details over here or make a new report. Waiting for your response

@sbassett @Reedy

Done.

This should get them all:

on it

To add another use case (and to ping the issue):

deleted access and error log files

As of today, QuickStatements supports MediaInfo items (Mxxx).
For now, you'll have to supply the IDs manually, which is a pain.
I am working on a QS syntax parser in Rust, which will support

• ranks
• page/filename => ID conversion on-the-fly

This will require some more testing

Bot code patched, deployed, someone please test

+1

That's true, but a reload of the batch page should return the STOP button, as its state is only read from the database. The bot, in turn, only checks the database (or should, I suspect it doesn't).

Another idea came to me:
What is it's not just "page lists", but any (general, of one of pre-defined types) tables?
One table type would be "page title/page namespace", giving us the above lists.
Others could be, say, Mix'n'match catalogs ("external ID/url/name/description/instance of").

In T231891#5468371, @Astinson wrote:

In T231891#5467967, @Magnus wrote:

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

Ooooh, thank you Magnus, that's a really great first pass at thinking about that. For the reuse by something like Listeria or tabernacle would it make sense to store the associated Wikidata item (or also commons MediaInfo id?) with the page ? (so that you wouldn't have to query those pages for them to do things like add properties?) One thing that kindof "fails" for me in the user experience of the current Petscan->Pagepile->Tabernacle workflows (and I am thinking this might be true in other workflows as well) is that the endtool expects _only_ Wikidata items, so if I don't generate a Wikidata list first, the tool either needs to have code to retrieve that or you have to generate a new list. If there was a second column with the optional Wikidata id, it would probably make lists made with one wiki in mind more portable.

In T231891#5468418, @Astinson wrote:

In T231891#5467967, @Magnus wrote:

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

For the List data structure, in addition too or as part of the Description: would it make sense to require a field for "Source" of the data (i.e. Petscan id, shorturl for query, etc) so that anyone "seeing" the pile could go to it, recreate the query/input, and modify it? (kindof like how folks use the Listeriabot lists).

In T231891#5469814, @LucasWerkmeister wrote:

Minimum viable product

• Import from various sources
  • All sources offered in PagePile
• Export to various places
  • All consumers offered in PagePile

How is this supposed to work? As far as I can tell, these imports and exports would have to go through the PagePile tool in some form, so to me these read like requirements that can only be fulfilled by one person: the PagePile maintainer.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

OK, some initial thoughts and remarks on this:

• I have actually rewritten Listeria in Rust, to use the Commons Data: namespace (aka .tab files) to store the lists, and use Lua to display them.
• I think the Commons Data: namespace would technically work for a generalized "list storage", thugh it seems to be a bit of abandonware (will this feature be long-term supported by the WMF?)
• Commons Data: namespace, if supported, would also have the proper scaling, caching etc. that PagePile is lacking
• It should, in principle, be possible to change PagePile to write new piles to the Commons Data: namespace, and return queries from there. That would give the new list storage a running start. We can replace PagePile later.
• Drawbacks of Commons Data: namespace are (a) cell size limit (400 characters, so should work for simple page lists), and (b) total page size (thus limiting the max list length)
• If Labs were to offer a scalable, backed-up object store for tools, that might be better suited for general list management
• Much of the "average Wikimedian" integration will have to come from (user-supplied) JavaScript, such as "snapshot this category tree" or something. I doubt waiting for WMF would be a timely solution.
• Short term, we (I?) could write a slim web API on Labs that abstracts the implementation away, offering a to-be-discussed set of functions (create/amend/remove list etc). Initially, this could run on PagePile in the background, or Commons Data: namespace, or even both (large lists go to pagepile, short ones into a MySQL database or Commons Data: namespace, etc.)