Page MenuHomePhabricator

Magnus (Magnus Manske)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2014, 3:02 PM (309 w, 3 d)
Availability
Available
LDAP User
Magnus Manske
MediaWiki User
Unknown

Recent Activity

Jul 13 2020

Magnus closed T257555: Lost connection to MySQL server while trying to import 39M rows of csv data (toolsdb) as Resolved.

Worked with 10K batches. I still think this rapid disconnect in the middle of a query is a bug.

Jul 13 2020, 7:23 PM · Data-Services, Toolforge
Magnus added a comment to T257555: Lost connection to MySQL server while trying to import 39M rows of csv data (toolsdb).

Trying 1M chunks, first two worked, but then:

mysqlimport: Error: 2013, Lost connection to MySQL server during query, when using table: osopenuprn_202006

I'll try even smaller ones, but it's getting a bit ridiculous. Why disconnect the server in the middle of a query?

Jul 13 2020, 8:51 AM · Data-Services, Toolforge

Jul 9 2020

Magnus created T257555: Lost connection to MySQL server while trying to import 39M rows of csv data (toolsdb).
Jul 9 2020, 10:13 AM · Data-Services, Toolforge

Jun 30 2020

Magnus added a comment to T255730: Request increased quota for petscan Toolforge tool for database access.

I have switched the configuration to use the petscan db connections only.

Jun 30 2020, 8:36 AM · Data-Services (Quota-requests)

Jun 26 2020

Magnus added a comment to T255730: Request increased quota for petscan Toolforge tool for database access.

Just to add, if some queries take too long I can likely just change a parameter to fix that. Someone with access to the stats, let me know.

Jun 26 2020, 7:46 AM · Data-Services (Quota-requests)
Magnus added a comment to T255730: Request increased quota for petscan Toolforge tool for database access.

The reason I use that many connections is precisely to avoid long-running ones, as I had in a previous version; they would time out or lose database connection, so I re-wrote the code to use more but shorter queries.

Jun 26 2020, 7:45 AM · Data-Services (Quota-requests)

Jun 24 2020

Magnus added a comment to T243848: Can't push to diffusion.

@Magnus: Does that mean that https://phabricator.wikimedia.org/source/tool-quickstatements/browse/master/ will get out of sync and should be archived?

Yes.

Jun 24 2020, 8:45 AM · Diffusion
Magnus changed the status of T243848: Can't push to diffusion from Declined to Resolved.

I have "forked" it into github. Using diffusion was a mistake in the first place. Now at https://github.com/magnusmanske/quickstatements

Jun 24 2020, 8:03 AM · Diffusion
Magnus added a comment to T243848: Can't push to diffusion.

Tried again with ssh://vcs@git-ssh.wikimedia.org/source/tool-quickstatements.git, no joy:

Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.
Jun 24 2020, 7:34 AM · Diffusion
Magnus added a comment to T243848: Can't push to diffusion.

tools.quickstatements@tools-sgebastion-07:~$ git remote remove origin
tools.quickstatements@tools-sgebastion-07:~$ git remote add origin ssh://phabricator.wikimedia.org/diffusion/2010/tool-quickstatements.git/
tools.quickstatements@tools-sgebastion-07:~$ git push fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

Jun 24 2020, 7:32 AM · Diffusion

Jun 22 2020

Magnus added a comment to T234617: Toolforge. introduce new domain toolforge.org.

FWIW, I believe all my (active) tools now use the new canonical URL.

Jun 22 2020, 3:08 PM · Patch-For-Review, Goal, Toolforge, cloud-services-team (Kanban), Kubernetes

Jun 19 2020

Magnus closed T100973: Flickr2Commons tool: Throttle needed as Resolved.

As I wrote five years ago, I added a limit to upload only two files concurrently.
No further comment has been made since, so I consider this issue to be solved.

Jun 19 2020, 9:30 AM · Tools, Commons

Jun 18 2020

Magnus added a comment to T255730: Request increased quota for petscan Toolforge tool for database access.

Thanks, that seems perfect!

Jun 18 2020, 5:18 PM · Data-Services (Quota-requests)

Jun 8 2020

Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

This config is part of the default configuration generated by webservice. That means we need to fix it upstream of the usage by this particular tool.

OK thanks

As a horrible short term fix for reasonator, I did:

$ ln -s $HOME/public_html/reasonator_types.js $HOME/public_html/_types.js
Jun 8 2020, 6:57 PM · cloud-services-team (Kanban), Tools, Toolforge
Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

I don’t understand why this is an UBN when as far as I understand we know a workaround (restart without --canonical)?

Jun 8 2020, 6:55 PM · cloud-services-team (Kanban), Tools, Toolforge
Magnus added a comment to T194953: Support hosting Rust tools on Toolforge.

Is there a movement towards Rust in any significant segment of the Wikimedia movement? Are there clear benefits for Toolforge tools to be written and maintained in Rust?

Jun 8 2020, 7:16 AM · cloud-services-team (Kanban), Toolforge

Jun 7 2020

Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

I think this is a blocker to moving all tools to the canonical URL. T234617 ?

Jun 7 2020, 4:04 PM · cloud-services-team (Kanban), Tools, Toolforge
Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

I cleared the error log on reasonator and switched to the previous URL schema, then back to the canonical one. No server errors in the log (some unrelated PhP ones, now cleared again)

Jun 7 2020, 3:34 PM · cloud-services-team (Kanban), Tools, Toolforge

Jun 6 2020

Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

@LucasWerkmeister thanks that worked, for now

Jun 6 2020, 3:10 PM · cloud-services-team (Kanban), Tools, Toolforge
Magnus raised the priority of T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical` from High to Unbreak Now!.

Same with https://geohack.toolforge.org/geohack.php . Geohack is kinda central for Wikipedia so please help!

Jun 6 2020, 12:43 PM · cloud-services-team (Kanban), Tools, Toolforge
Magnus added a comment to T254361: Tool `most-wanted` is 502.

Thanks @bd808 !

Jun 6 2020, 9:23 AM · cloud-services-team (Kanban), Toolforge
Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

Other files load fine. Haven't checked all though

Jun 6 2020, 8:57 AM · cloud-services-team (Kanban), Tools, Toolforge
Magnus added a comment to T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.

URL is https://reasonator.toolforge.org/reasonator_types.js

Jun 6 2020, 8:56 AM · cloud-services-team (Kanban), Tools, Toolforge
Magnus triaged T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical` as High priority.
Jun 6 2020, 8:54 AM · cloud-services-team (Kanban), Tools, Toolforge
Magnus created T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.
Jun 6 2020, 8:54 AM · cloud-services-team (Kanban), Tools, Toolforge

Jun 3 2020

Magnus created T254361: Tool `most-wanted` is 502.
Jun 3 2020, 3:07 PM · cloud-services-team (Kanban), Toolforge

May 28 2020

Magnus added a comment to T251567: PetScan problem: 504 Gateway Time-out.

Restarted it now.
I am also working on a watcher script to restart it automatically when it goes down.

May 28 2020, 8:27 AM · Cloud-VPS
Magnus added a comment to T215035: 502 /504 Bad Gateway issue on Petscan.

It's my dev server, should work for any query size. Feel free to use when the main site is down.

May 28 2020, 8:26 AM · VPS-Projects

May 22 2020

Magnus created T253373: Commons API search/imageinfo generator not working.
May 22 2020, 8:30 AM · MediaWiki-API, Commons

May 21 2020

Magnus placed T181062: Adapt QuickStatements2 to be able to work with structured data on Commons as well up for grabs.
May 21 2020, 7:24 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, GLAM-Tech, SDC General, Wikidata

Apr 29 2020

Magnus added a comment to T215035: 502 /504 Bad Gateway issue on Petscan.

@Aklapper actually the repo is https://github.com/magnusmanske/petscan_rs (the bitbucket one is the old C++ version).

Apr 29 2020, 1:37 PM · VPS-Projects

Apr 22 2020

Addshore awarded T249587: Wikidata API wbgetentities: Allowed memory size exceeded a Hungry Hippo token.
Apr 22 2020, 11:08 PM · Wikidata-Campsite, Wikidata

Apr 20 2020

Magnus added a comment to T215035: 502 /504 Bad Gateway issue on Petscan.

There are some PetScan outages, reason unknown so far, but outside those, the above queries all work fine.

Apr 20 2020, 7:43 AM · VPS-Projects

Apr 14 2020

Magnus added a comment to T249587: Wikidata API wbgetentities: Allowed memory size exceeded.

Still happening.

Apr 14 2020, 9:43 AM · Wikidata-Campsite, Wikidata

Apr 7 2020

Magnus added a comment to T249587: Wikidata API wbgetentities: Allowed memory size exceeded.

Maybe this helps:

Apr 7 2020, 2:26 PM · Wikidata-Campsite, Wikidata
Magnus added a comment to T249587: Wikidata API wbgetentities: Allowed memory size exceeded.

Yes, still seeing it. Runs for ~20sec so probably not a cache. Changed the JSONP callback(s) and still same error, so definitely no content-level cache.

Apr 7 2020, 2:25 PM · Wikidata-Campsite, Wikidata
Magnus created T249587: Wikidata API wbgetentities: Allowed memory size exceeded.
Apr 7 2020, 7:20 AM · Wikidata-Campsite, Wikidata

Mar 18 2020

bd808 awarded T246096: Tools cdnjs returns blank page a Yellow Medal token.
Mar 18 2020, 7:57 PM · Tools, cloud-services-team (Kanban)

Mar 17 2020

Magnus updated subscribers of T246096: Tools cdnjs returns blank page.
Mar 17 2020, 10:56 AM · Tools, cloud-services-team (Kanban)

Mar 13 2020

Magnus closed T247596: Strange wikibase API error as Resolved.

It appears that the third-party bot framework quietly swallows this and just throws an exception with the "info" field :-(

Mar 13 2020, 2:18 PM · Wikidata
Magnus added a comment to T247596: Strange wikibase API error.

In the error I mean

Mar 13 2020, 1:24 PM · Wikidata
Magnus added a comment to T247596: Strange wikibase API error.

Hmm, I think what tripped my code is that there was no code:'maxlag' in the response?

Mar 13 2020, 1:23 PM · Wikidata
Magnus added a comment to T247596: Strange wikibase API error.

Seems to clear up now

Mar 13 2020, 1:22 PM · Wikidata
Magnus added a comment to T247596: Strange wikibase API error.

But it's all kinds of wikis that show up. Something with the dispatcher?

Mar 13 2020, 1:17 PM · Wikidata
Magnus created T247596: Strange wikibase API error.
Mar 13 2020, 11:46 AM · Wikidata

Mar 4 2020

Magnus added a comment to T246746: Bot edits on Wikidata should be hidden in other project's recent changes.

QuickStatements edits as the user who submitted the job, using OAuth, as is the WMF-preferred method for tools.

Mar 4 2020, 1:27 PM · Upstream, MediaWiki-extensions-WikibaseClient, Wikidata, MediaWiki-Recent-changes, Edit-Review-Improvements-RC-Page, Growth-Team

Feb 25 2020

Magnus created T246096: Tools cdnjs returns blank page.
Feb 25 2020, 12:32 PM · Tools, cloud-services-team (Kanban)

Feb 19 2020

Magnus closed T232917: XSS in listeria as Resolved.

Should now be fixed with https://bitbucket.org/magnusmanske/listeria/commits/5b53c94ffb10

Feb 19 2020, 9:35 AM · Security, Vuln-XSS, Tools

Feb 4 2020

Magnus added a comment to T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter`.

Getting this error for mix-n-match tool.

Feb 4 2020, 3:52 PM · Patch-For-Review, Kubernetes, Tools-Kubernetes, Toolforge

Jan 28 2020

Magnus added a project to T243848: Can't push to diffusion: Diffusion.
Jan 28 2020, 3:55 PM · Diffusion
Magnus created T243848: Can't push to diffusion.
Jan 28 2020, 3:55 PM · Diffusion

Jan 26 2020

Magnus added a comment to T214278: Quickstatements, "backend is overloaded".

@bd808 Thanks, now running quickstatements as two repilcas

Jan 26 2020, 9:28 AM · TestMe, Tools, Toolforge

Jan 24 2020

Magnus added a comment to T243580: "413 Request Entity Too Large" / Kubernetes 2020?.

Update: Setting server.max-request-size = 1000000 in $HOME/.lighttpd.conf and restarting the webservice did not help.

Jan 24 2020, 9:29 AM · cloud-services-team (Kanban), Tools, Kubernetes, Toolforge
Magnus created T243580: "413 Request Entity Too Large" / Kubernetes 2020?.
Jan 24 2020, 9:22 AM · cloud-services-team (Kanban), Tools, Kubernetes, Toolforge

Jan 14 2020

Magnus created T242720: Regression in URL path handling on 2020 (?) kubernetes.
Jan 14 2020, 9:39 AM · Cloud-Services

Dec 10 2019

Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

@Bugreporter That seems unlikely. QuickStatementsBot is blocked since October 28th, and daily users of the tool only started reporting this issue yesterday.

Dec 10 2019, 3:06 PM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

For debugging:

Dec 10 2019, 12:32 PM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

It's not a stale cache on my side. If I get different properties from https://www.wikidata.org/w/api.php?action=help&modules=query%2Buserinfo I get the requested properties but still the bogus block.

Dec 10 2019, 11:30 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

If I check the API directly in the browser, it doesn't show. Maybe because OAuth login?

Dec 10 2019, 11:26 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

I seem to be getting the block for [[User:Doqume]] on Wikidata, but for every user

Dec 10 2019, 11:25 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

I think I found the reason. Using MW API to get user info, I get:

{
  "id": 4420,
  "name": "Magnus Manske",
  "blockid": 15320,
  "blockedby": "Mahir256",
  "blockedbyid": 203574,
  "blockreason": "আপনাকে স্বয়ংক্রিয়ভাবে বাধা দেওয়া হয়েছে, কারণ আপনার আইপি ঠিকানাটি সম্প্রতি \"[[User:Doqume|Doqume]]\" ব্যবহার করেছেন। Doqume-কে বাধাদানের কারণ \"Freebald-ish behavior\"",
  "blockedtimestamp": "2019-12-09T16:59:04Z",
  "blockexpiry": "2019-12-10T16:59:04Z",
  "groups": [
    "rollbacker",
    "*",
    "user",
    "autoconfirmed"
  ],
  "rights": [
    "autopatrol",
    "editsemiprotected",
    "move",
    "autoconfirmed",
    "skipcaptcha",
    "abusefilter-log-detail",
    "suppressredirect",
    "read",
    "edit",
    "createpage",
    "createtalk",
    "writeapi",
    "translate",
    "item-term",
    "property-term",
    "item-merge",
    "item-redirect",
    "abusefilter-view",
    "abusefilter-log",
    "flow-hide",
    "reupload-own",
    "move-rootuserpages",
    "move-categorypages",
    "minoredit",
    "purge",
    "applychangetags",
    "changetags",
    "reupload",
    "upload",
    "flow-edit-post"
  ]
}

But I am not blocked; editing works fine.

Dec 10 2019, 11:14 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

I can't create batches either, so good for testing!

Dec 10 2019, 11:02 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata
Magnus added a comment to T240316: Issue with QuickStatements "you are blocked on Wikidata".

I did not touch QS for weeks. No idea why this is happening.

Dec 10 2019, 11:01 AM · User-Ladsgroup, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata

Dec 6 2019

Magnus closed T239817: Cross Site Scripting on tools.wmflabs.org/multidesc as Resolved.
Dec 6 2019, 12:37 PM · Security, Security-Team, Tools, Vuln-XSS
Magnus added a comment to T239817: Cross Site Scripting on tools.wmflabs.org/multidesc.

Fixed now.

Dec 6 2019, 8:39 AM · Security, Security-Team, Tools, Vuln-XSS

Dec 5 2019

Magnus added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

Hello Team,

I am still able to have a XSS vulnerability in the same website. Should I paste the details over here or make a new report. Waiting for your response

@sbassett @Reedy

Dec 5 2019, 1:46 PM · Security, Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team

Dec 4 2019

Magnus added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

Done.

Dec 4 2019, 11:34 AM · Security, Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team

Dec 3 2019

Magnus added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

This should get them all:

Dec 3 2019, 2:42 PM · Security, Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team
Magnus added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

on it

Dec 3 2019, 2:33 PM · Security, Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team

Nov 12 2019

Magnus added a comment to T236974: Establish a process for increasing a toolforge tool's connections to the wiki replicas.

To add another use case (and to ping the issue):

Nov 12 2019, 9:58 AM · cloud-services-team (Kanban), Data-Services

Sep 25 2019

Magnus closed T233730: Reducing geohack/access.log size (58GB), a subtask of T233120: 2019-09-17: tools share cleanup (high usage), as Resolved.
Sep 25 2019, 8:30 AM · Data-Services, cloud-services-team (Kanban)
Magnus closed T233730: Reducing geohack/access.log size (58GB) as Resolved.

deleted access and error log files

Sep 25 2019, 8:30 AM · Data-Services, cloud-services-team (Kanban)

Sep 24 2019

Magnus added a comment to T181062: Adapt QuickStatements2 to be able to work with structured data on Commons as well.

As of today, QuickStatements supports MediaInfo items (Mxxx).
For now, you'll have to supply the IDs manually, which is a pain.
I am working on a QS syntax parser in Rust, which will support

  • ranks
  • page/filename => ID conversion on-the-fly

This will require some more testing

Sep 24 2019, 10:04 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, GLAM-Tech, SDC General, Wikidata

Sep 17 2019

Magnus closed T232960: QuickStatements batch running in background not stopping with the stop button as Resolved.
Sep 17 2019, 9:25 AM · Tools, Wikidata

Sep 16 2019

Magnus added a comment to T232960: QuickStatements batch running in background not stopping with the stop button.

Bot code patched, deployed, someone please test

Sep 16 2019, 1:11 PM · Tools, Wikidata
Magnus added a comment to T232986: Enable all possible language pairs in cxserver, apply wiki specific configuration in the wiki.

+1

Sep 16 2019, 10:46 AM · Language-Team (Language-2019-October-December), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), ContentTranslation
Magnus added a comment to T232960: QuickStatements batch running in background not stopping with the stop button.

That's true, but a reload of the batch page should return the STOP button, as its state is only read from the database. The bot, in turn, only checks the database (or should, I suspect it doesn't).

Sep 16 2019, 8:09 AM · Tools, Wikidata

Sep 8 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Another idea came to me:
What is it's not just "page lists", but any (general, of one of pre-defined types) tables?
One table type would be "page title/page namespace", giving us the above lists.
Others could be, say, Mix'n'match catalogs ("external ID/url/name/description/instance of").

Sep 8 2019, 9:07 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request

Sep 6 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

Ooooh, thank you Magnus, that's a really great first pass at thinking about that. For the reuse by something like Listeria or tabernacle would it make sense to store the associated Wikidata item (or also commons MediaInfo id?) with the page ? (so that you wouldn't have to query those pages for them to do things like add properties?) One thing that kindof "fails" for me in the user experience of the current Petscan->Pagepile->Tabernacle workflows (and I am thinking this might be true in other workflows as well) is that the endtool expects _only_ Wikidata items, so if I don't generate a Wikidata list first, the tool either needs to have code to retrieve that or you have to generate a new list. If there was a second column with the optional Wikidata id, it would probably make lists made with one wiki in mind more portable.

Sep 6 2019, 6:24 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

For the List data structure, in addition too or as part of the Description: would it make sense to require a field for "Source" of the data (i.e. Petscan id, shorturl for query, etc) so that anyone "seeing" the pile could go to it, recreate the query/input, and modify it? (kindof like how folks use the Listeriabot lists).

Sep 6 2019, 6:23 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Minimum viable product

  • Import from various sources
    • All sources offered in PagePile
  • Export to various places
    • All consumers offered in PagePile

How is this supposed to work? As far as I can tell, these imports and exports would have to go through the PagePile tool in some form, so to me these read like requirements that can only be fulfilled by one person: the PagePile maintainer.

Sep 6 2019, 6:22 AM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request

Sep 5 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

Sep 5 2019, 1:33 PM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

OK, some initial thoughts and remarks on this:

  • I have actually rewritten Listeria in Rust, to use the Commons Data: namespace (aka .tab files) to store the lists, and use Lua to display them.
  • I think the Commons Data: namespace would technically work for a generalized "list storage", thugh it seems to be a bit of abandonware (will this feature be long-term supported by the WMF?)
  • Commons Data: namespace, if supported, would also have the proper scaling, caching etc. that PagePile is lacking
  • It should, in principle, be possible to change PagePile to write new piles to the Commons Data: namespace, and return queries from there. That would give the new list storage a running start. We can replace PagePile later.
  • Drawbacks of Commons Data: namespace are (a) cell size limit (400 characters, so should work for simple page lists), and (b) total page size (thus limiting the max list length)
  • If Labs were to offer a scalable, backed-up object store for tools, that might be better suited for general list management
  • Much of the "average Wikimedian" integration will have to come from (user-supplied) JavaScript, such as "snapshot this category tree" or something. I doubt waiting for WMF would be a timely solution.
  • Short term, we (I?) could write a slim web API on Labs that abstracts the implementation away, offering a to-be-discussed set of functions (create/amend/remove list etc). Initially, this could run on PagePile in the background, or Commons Data: namespace, or even both (large lists go to pagepile, short ones into a MySQL database or Commons Data: namespace, etc.)
Sep 5 2019, 1:09 PM · WMSE-Tools-for-Partnerships-2019-Blueprinting, Technical-Tool-Request
Magnus added a comment to T231916: tool-quickstatements V2 batch mode: "Qualifier to" not more working.

I believe I fixed the issue in the Rust bot. I had a successful test, but please try it yourself.

Sep 5 2019, 12:23 PM · Tools

Sep 4 2019

Magnus reopened T231916: tool-quickstatements V2 batch mode: "Qualifier to" not more working as "Open".

Actually, that bitbucket repo is for the _really old version_ (pre-1.0).

Sep 4 2019, 1:22 PM · Tools

Sep 2 2019

Magnus added a comment to T229917: Add tags parameter to Wikibase APIs that edit entities.

https://www.mediawiki.org/wiki/Manual:Tags says "A complete list of all the available tags is displayed on Special:Tags. Users with the managechangetags user right – administrators by default – can use this special page to create and delete tags (see Help:Tags)."

Does this help?

Sep 2 2019, 9:39 AM · MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Patch-For-Review, Wikidata-Bridge-Sprint-3, MediaWiki-extensions-WikibaseRepository, MediaWiki-Change-tagging, Wikidata
Magnus added a comment to T229917: Add tags parameter to Wikibase APIs that edit entities.

Added it for most of my tools, centrally. Works fine for distributed-game. for wdfist I get:

E1:The tag "wdfist" is not allowed to be manually applied

Now rolling the change back, until I know what tags I am allowed to use where and when.

Sep 2 2019, 8:46 AM · MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Patch-For-Review, Wikidata-Bridge-Sprint-3, MediaWiki-extensions-WikibaseRepository, MediaWiki-Change-tagging, Wikidata

Jul 24 2019

Magnus added a comment to T205549: book2scroll is not working (due to jquery 404?).

Because toolforge forgot the replica.conf again, see T166949. Webservice restarted, manually, yet again, works. For the next few minutes, probably.

Jul 24 2019, 10:02 PM · Tools, Wikisource

Jul 2 2019

Magnus added a comment to T223548: Item namespace issues in Factgrid.

Everyone, I own Reasonator, including the experimental version 2 which is used here (and should be better suited than the dated V1).

Jul 2 2019, 12:36 PM · Wikidata-Campsite, Wikidata, FactGrid

Jun 27 2019

Magnus created T226714: Special:EntityData and wbgetentities report missing entities differently.
Jun 27 2019, 1:24 PM · Wikidata

Jun 26 2019

Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

Changed back the topic. This is a huge scope change and derailing things. As far as I see everywhere in the api we use "claims", not "statements" (also in the functions). The only inconsistency right now is mediainfo, that should be fixed. If you want to change the everything in the Wikibase API to use statements instead of claims (wbgetclaims -> wbgetstatements, etc.), file a new task so I can down vote that one as a huge waste of resources.

OK, then I can just Decline this task? As established above, when Wikimedia DE wrote WBMI in early 2016 they used "statements" because all new code should use that and not "claims", but haven't gone back to fix Wikidata to use the modern language.

Jun 26 2019, 10:55 AM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 25 2019

Magnus updated subscribers of T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

@Jdforrester-WMF Is that an official design decision (claims=>statements)? Where was this fundamentally breaking change announced to the public?

Jun 25 2019, 7:42 AM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 22 2019

Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

FWIW, I have already changed my code to work with either claims or statements. Quick thoughts:

Jun 22 2019, 1:00 PM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 21 2019

Magnus added a comment to T226084: Wikidata server slow.

On another note, the Reasonator example in my original post seems to load now. I'll check if the Rust code works as well now.

Jun 21 2019, 7:24 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.

@Magnus It is well-known currently that MediaWiki exposes many powerful API that we do not support to perform well, but allow regardless as a convenience service. If we were stricter about response times for all features, we'd probably just turn many of them off and limit the capabilities of those APIs until and unless the amount of resources required to make them work reliably fast is justified.

I expect the maintainers of this API to have tested the supported and encouraged use cases and to know whether they are fast. I haven't personally looked at the p99 for this particular API, but from experience in other endpoints, it tends to be extreme cases that we'd be very unlikely to support with fast responses.

But if they haven't in a while, it's certainly worth looking at those again from time to time.

Jun 21 2019, 7:23 AM · TestMe, Wikidata-Campsite, Wikidata

Jun 20 2019

Magnus added a comment to T226084: Wikidata server slow.

May I humbly suggest to have a look at the consistent 2min response time of the p99 server (in grafana), before deciding it's a problem outside WMFs control, no matter how convenient that may seem?

Jun 20 2019, 9:39 AM · TestMe, Wikidata-Campsite, Wikidata

Jun 19 2019

Magnus added a comment to T226084: Wikidata server slow.

No, sorry, issue remains.

Jun 19 2019, 5:26 PM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.
GET /w/api.php?callback=jQuery21303406678877236998_1560936691744&action=wbgetentities&ids=P2508%7CP2631%7CP2509%7CP4276%7CP272%7CP4529%7CP5032%7CP4947%7CP5786%7CP6145%7CP1609%7CP1230%7CP2896%7CP4730%7CP2093%7CP1844%7CP1813%7CP5396%7CQ1199348%7CP435%7CP3959%7CP747%7CP1274%7CP1085%7CP5331%7CP4839%7CP4969%7CP103%7CQ49088%7CP1648%7CQ19045189%7CP3793%7CP2847%7CP3035%7CP4389%7CP5062%7CP5508%7CP4264%7CP6698%7CP6617%7CP2241%7CQ44374960%7CQ4644021%7CQ839097%7CP1268%7CQ9624%7CQ8055775%7CQ210152%7CQ4642661%7CQ635616&props=info%7Caliases%7Clabels%7Cdescriptions%7Cclaims%7Csitelinks%7Cdatatype&format=json&_=1560936691745 HTTP/1.1
Host: www.wikidata.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: */*
Accept-Language: en-GB,en;q=0.7,de;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://tools.wmflabs.org/reasonator/?q=Q350
DNT: 1
Connection: keep-alive

Cookies redacted

Jun 19 2019, 2:15 PM · TestMe, Wikidata-Campsite, Wikidata
Magnus created T226088: DNS lookups for nih.gov hosts failing from Cloud VPS/Toolforge, services (citoid).
Jun 19 2019, 10:10 AM · Citoid, Services, cloud-services-team (Kanban), Toolforge
Magnus added a comment to T226084: Wikidata server slow.

Timing details of that slow request:

Jun 19 2019, 10:01 AM · TestMe, Wikidata-Campsite, Wikidata