Page MenuHomePhabricator

Magnus (Magnus Manske)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2014, 3:02 PM (259 w, 4 d)
Availability
Available
LDAP User
Magnus Manske
MediaWiki User
Unknown

Recent Activity

Wed, Sep 25

Magnus closed T233730: Reducing geohack/access.log size (58GB), a subtask of T233120: 2019-09-17: tools share cleanup (high usage), as Resolved.
Wed, Sep 25, 8:30 AM · Data-Services, cloud-services-team (Kanban)
Magnus closed T233730: Reducing geohack/access.log size (58GB) as Resolved.

deleted access and error log files

Wed, Sep 25, 8:30 AM · Data-Services, cloud-services-team (Kanban)

Tue, Sep 24

Magnus added a comment to T181062: Adapt QuickStatements2 to be able to work with structured data on Commons as well.

As of today, QuickStatements supports MediaInfo items (Mxxx).
For now, you'll have to supply the IDs manually, which is a pain.
I am working on a QS syntax parser in Rust, which will support

  • ranks
  • page/filename => ID conversion on-the-fly

This will require some more testing

Tue, Sep 24, 10:04 AM · GLAM-Tech, SDC General, Wikidata

Tue, Sep 17

Magnus closed T232960: QuickStatements batch running in background not stopping with the stop button as Resolved.
Tue, Sep 17, 9:25 AM · Tools, Wikidata

Mon, Sep 16

Magnus added a comment to T232960: QuickStatements batch running in background not stopping with the stop button.

Bot code patched, deployed, someone please test

Mon, Sep 16, 1:11 PM · Tools, Wikidata
Magnus added a comment to T232986: Enable all possible language pairs in cxserver, apply wiki specific configuration in the wiki.

+1

Mon, Sep 16, 10:46 AM · Language-Team (Language-2019-October-December), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), ContentTranslation
Magnus added a comment to T232960: QuickStatements batch running in background not stopping with the stop button.

That's true, but a reload of the batch page should return the STOP button, as its state is only read from the database. The bot, in turn, only checks the database (or should, I suspect it doesn't).

Mon, Sep 16, 8:09 AM · Tools, Wikidata

Sep 8 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Another idea came to me:
What is it's not just "page lists", but any (general, of one of pre-defined types) tables?
One table type would be "page title/page namespace", giving us the above lists.
Others could be, say, Mix'n'match catalogs ("external ID/url/name/description/instance of").

Sep 8 2019, 9:07 AM · Technical-Tool-Request

Sep 6 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

Ooooh, thank you Magnus, that's a really great first pass at thinking about that. For the reuse by something like Listeria or tabernacle would it make sense to store the associated Wikidata item (or also commons MediaInfo id?) with the page ? (so that you wouldn't have to query those pages for them to do things like add properties?) One thing that kindof "fails" for me in the user experience of the current Petscan->Pagepile->Tabernacle workflows (and I am thinking this might be true in other workflows as well) is that the endtool expects _only_ Wikidata items, so if I don't generate a Wikidata list first, the tool either needs to have code to retrieve that or you have to generate a new list. If there was a second column with the optional Wikidata id, it would probably make lists made with one wiki in mind more portable.

Sep 6 2019, 6:24 AM · Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

For the List data structure, in addition too or as part of the Description: would it make sense to require a field for "Source" of the data (i.e. Petscan id, shorturl for query, etc) so that anyone "seeing" the pile could go to it, recreate the query/input, and modify it? (kindof like how folks use the Listeriabot lists).

Sep 6 2019, 6:23 AM · Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Minimum viable product

  • Import from various sources
    • All sources offered in PagePile
  • Export to various places
    • All consumers offered in PagePile

How is this supposed to work? As far as I can tell, these imports and exports would have to go through the PagePile tool in some form, so to me these read like requirements that can only be fulfilled by one person: the PagePile maintainer.

Sep 6 2019, 6:22 AM · Technical-Tool-Request

Sep 5 2019

Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

Started some design notes of such a product: https://meta.wikimedia.org/wiki/Gulp

Sep 5 2019, 1:33 PM · Technical-Tool-Request
Magnus added a comment to T231891: Create a Generic List-building tool that can meet and exceed the applications of Pagepile.

OK, some initial thoughts and remarks on this:

  • I have actually rewritten Listeria in Rust, to use the Commons Data: namespace (aka .tab files) to store the lists, and use Lua to display them.
  • I think the Commons Data: namespace would technically work for a generalized "list storage", thugh it seems to be a bit of abandonware (will this feature be long-term supported by the WMF?)
  • Commons Data: namespace, if supported, would also have the proper scaling, caching etc. that PagePile is lacking
  • It should, in principle, be possible to change PagePile to write new piles to the Commons Data: namespace, and return queries from there. That would give the new list storage a running start. We can replace PagePile later.
  • Drawbacks of Commons Data: namespace are (a) cell size limit (400 characters, so should work for simple page lists), and (b) total page size (thus limiting the max list length)
  • If Labs were to offer a scalable, backed-up object store for tools, that might be better suited for general list management
  • Much of the "average Wikimedian" integration will have to come from (user-supplied) JavaScript, such as "snapshot this category tree" or something. I doubt waiting for WMF would be a timely solution.
  • Short term, we (I?) could write a slim web API on Labs that abstracts the implementation away, offering a to-be-discussed set of functions (create/amend/remove list etc). Initially, this could run on PagePile in the background, or Commons Data: namespace, or even both (large lists go to pagepile, short ones into a MySQL database or Commons Data: namespace, etc.)
Sep 5 2019, 1:09 PM · Technical-Tool-Request
Magnus added a comment to T231916: tool-quickstatements V2 batch mode: "Qualifier to" not more working.

I believe I fixed the issue in the Rust bot. I had a successful test, but please try it yourself.

Sep 5 2019, 12:23 PM · Tools

Sep 4 2019

Magnus reopened T231916: tool-quickstatements V2 batch mode: "Qualifier to" not more working as "Open".

Actually, that bitbucket repo is for the _really old version_ (pre-1.0).

Sep 4 2019, 1:22 PM · Tools

Sep 2 2019

Magnus added a comment to T229917: Add tags parameter to Wikibase APIs that edit entities.

https://www.mediawiki.org/wiki/Manual:Tags says "A complete list of all the available tags is displayed on Special:Tags. Users with the managechangetags user right – administrators by default – can use this special page to create and delete tags (see Help:Tags)."
Does this help?

Sep 2 2019, 9:39 AM · MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Patch-For-Review, Wikidata-Bridge-Sprint-3, MediaWiki-extensions-WikibaseRepository, MediaWiki-Change-tagging, Wikidata
Magnus added a comment to T229917: Add tags parameter to Wikibase APIs that edit entities.

Added it for most of my tools, centrally. Works fine for distributed-game. for wdfist I get:

E1:The tag "wdfist" is not allowed to be manually applied

Now rolling the change back, until I know what tags I am allowed to use where and when.

Sep 2 2019, 8:46 AM · MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Patch-For-Review, Wikidata-Bridge-Sprint-3, MediaWiki-extensions-WikibaseRepository, MediaWiki-Change-tagging, Wikidata

Jul 24 2019

Magnus added a comment to T205549: book2scroll is not working (due to jquery 404?).

Because toolforge forgot the replica.conf again, see T166949. Webservice restarted, manually, yet again, works. For the next few minutes, probably.

Jul 24 2019, 10:02 PM · Tools, Wikisource

Jul 2 2019

Magnus added a comment to T223548: Item namespace issues in Factgrid.

Everyone, I own Reasonator, including the experimental version 2 which is used here (and should be better suited than the dated V1).

Jul 2 2019, 12:36 PM · Wikidata-Campsite, Wikidata, FactGrid

Jun 27 2019

Magnus created T226714: Special:EntityData and wbgetentities report missing entities differently.
Jun 27 2019, 1:24 PM · Wikidata

Jun 26 2019

Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

Changed back the topic. This is a huge scope change and derailing things. As far as I see everywhere in the api we use "claims", not "statements" (also in the functions). The only inconsistency right now is mediainfo, that should be fixed. If you want to change the everything in the Wikibase API to use statements instead of claims (wbgetclaims -> wbgetstatements, etc.), file a new task so I can down vote that one as a huge waste of resources.

OK, then I can just Decline this task? As established above, when Wikimedia DE wrote WBMI in early 2016 they used "statements" because all new code should use that and not "claims", but haven't gone back to fix Wikidata to use the modern language.

Jun 26 2019, 10:55 AM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 25 2019

Magnus updated subscribers of T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

@Jdforrester-WMF Is that an official design decision (claims=>statements)? Where was this fundamentally breaking change announced to the public?

Jun 25 2019, 7:42 AM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 22 2019

Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".

FWIW, I have already changed my code to work with either claims or statements. Quick thoughts:

Jun 22 2019, 1:00 PM · StructuredDataOnCommons, Wikidata, WikibaseMediaInfo

Jun 21 2019

Magnus added a comment to T226084: Wikidata server slow.

On another note, the Reasonator example in my original post seems to load now. I'll check if the Rust code works as well now.

Jun 21 2019, 7:24 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.

@Magnus It is well-known currently that MediaWiki exposes many powerful API that we do not support to perform well, but allow regardless as a convenience service. If we were stricter about response times for all features, we'd probably just turn many of them off and limit the capabilities of those APIs until and unless the amount of resources required to make them work reliably fast is justified.
I expect the maintainers of this API to have tested the supported and encouraged use cases and to know whether they are fast. I haven't personally looked at the p99 for this particular API, but from experience in other endpoints, it tends to be extreme cases that we'd be very unlikely to support with fast responses.
But if they haven't in a while, it's certainly worth looking at those again from time to time.

Jun 21 2019, 7:23 AM · TestMe, Wikidata-Campsite, Wikidata

Jun 20 2019

Magnus added a comment to T226084: Wikidata server slow.

May I humbly suggest to have a look at the consistent 2min response time of the p99 server (in grafana), before deciding it's a problem outside WMFs control, no matter how convenient that may seem?

Jun 20 2019, 9:39 AM · TestMe, Wikidata-Campsite, Wikidata

Jun 19 2019

Magnus added a comment to T226084: Wikidata server slow.

No, sorry, issue remains.

Jun 19 2019, 5:26 PM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.
GET /w/api.php?callback=jQuery21303406678877236998_1560936691744&action=wbgetentities&ids=P2508%7CP2631%7CP2509%7CP4276%7CP272%7CP4529%7CP5032%7CP4947%7CP5786%7CP6145%7CP1609%7CP1230%7CP2896%7CP4730%7CP2093%7CP1844%7CP1813%7CP5396%7CQ1199348%7CP435%7CP3959%7CP747%7CP1274%7CP1085%7CP5331%7CP4839%7CP4969%7CP103%7CQ49088%7CP1648%7CQ19045189%7CP3793%7CP2847%7CP3035%7CP4389%7CP5062%7CP5508%7CP4264%7CP6698%7CP6617%7CP2241%7CQ44374960%7CQ4644021%7CQ839097%7CP1268%7CQ9624%7CQ8055775%7CQ210152%7CQ4642661%7CQ635616&props=info%7Caliases%7Clabels%7Cdescriptions%7Cclaims%7Csitelinks%7Cdatatype&format=json&_=1560936691745 HTTP/1.1
Host: www.wikidata.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: */*
Accept-Language: en-GB,en;q=0.7,de;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://tools.wmflabs.org/reasonator/?q=Q350
DNT: 1
Connection: keep-alive

Cookies redacted

Jun 19 2019, 2:15 PM · TestMe, Wikidata-Campsite, Wikidata
Magnus created T226088: DNS lookups for nih.gov hosts failing from Cloud VPS/Toolforge, services (citoid).
Jun 19 2019, 10:10 AM · Citoid, Services, cloud-services-team (Kanban), Toolforge
Magnus added a comment to T226084: Wikidata server slow.

Timing details of that slow request:

Jun 19 2019, 10:01 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.

Response header from one of the slow requests:

HTTP/2.0 200 OK
date: Wed, 19 Jun 2019 09:31:43 GMT
content-type: text/javascript; charset=utf-8
server: mw1341.eqiad.wmnet
x-powered-by: HHVM/3.18.6-dev
mediawiki-login-suppressed: true
cache-control: private, must-revalidate, max-age=0
content-disposition: inline; filename=api-result.js
x-content-type-options: nosniff
x-frame-options: DENY
backend-timing: D=1129973 t=1560936702635082
vary: Accept-Encoding,Treat-as-Untrusted,X-Forwarded-Proto,Cookie,Authorization,X-Seven
content-encoding: gzip
x-varnish: 774984440, 505592114, 724292664
via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)
accept-ranges: bytes
age: 0
x-cache: cp1081 pass, cp3032 pass, cp3041 pass
x-cache-status: pass
server-timing: cache;desc="pass"
strict-transport-security: max-age=106384710; includeSubDomains; preload
x-analytics: ns=-1;special=Badtitle;loggedIn=1;WMF-Last-Access=19-Jun-2019;WMF-Last-Access-Global=19-Jun-2019;https=1
x-client-ip: 2001:630:206:6204:cc46:3ce1:27e1:3062
X-Firefox-Spdy: h2
Jun 19 2019, 10:00 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus triaged T226084: Wikidata server slow as High priority.
Jun 19 2019, 9:50 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus added a comment to T226084: Wikidata server slow.

Jun 19 2019, 9:46 AM · TestMe, Wikidata-Campsite, Wikidata
Magnus created T226084: Wikidata server slow.
Jun 19 2019, 9:46 AM · TestMe, Wikidata-Campsite, Wikidata

Jun 6 2019

Magnus created T225191: Backed-up large storage?.
Jun 6 2019, 10:30 AM · Data-Services, Toolforge
Magnus created T225190: Object store?.
Jun 6 2019, 10:17 AM · Toolforge

May 31 2019

Magnus committed R2010:7830824139e1: remove site notice (authored by Magnus).
remove site notice
May 31 2019, 4:16 PM
Magnus committed R2010:a56829012473: separate DB fixes (authored by Magnus).
separate DB fixes
May 31 2019, 4:16 PM
Magnus committed R2010:6a2fb4cba2a4: changing to new auth DB (authored by Magnus).
changing to new auth DB
May 31 2019, 1:09 PM
Magnus committed R2010:b97e8a6e7ed2: JSON oauth (authored by Magnus).
JSON oauth
May 31 2019, 12:26 PM

May 23 2019

Bstorm awarded T224155: Reduce size of referee db on toolsdb if at all possible a Party Time token.
May 23 2019, 4:38 PM · Data-Services
Magnus closed T224155: Reduce size of referee db on toolsdb if at all possible as Resolved.

Done.

May 23 2019, 11:45 AM · Data-Services
Magnus closed T224155: Reduce size of referee db on toolsdb if at all possible, a subtask of T224152: toolsdb replica is running low on space -- cleanup large tables if possible, as Resolved.
May 23 2019, 11:45 AM · cloud-services-team (Kanban), Data-Services

May 14 2019

Magnus closed T223282: wbeditentity error as Resolved.

Never mind, it's the multilingual string!

May 14 2019, 3:11 PM · Wikidata
Magnus created T223282: wbeditentity error.
May 14 2019, 3:06 PM · Wikidata

Mar 29 2019

Magnus committed R2010:6eedbf852913: maxlag for bot mode (authored by Magnus).
maxlag for bot mode
Mar 29 2019, 4:07 PM

Mar 8 2019

Magnus added a comment to T217894: Toolforge Stretch bastion occasionally closes SSH connection just before authentication succeeds.

Happened to me as well, yesterday (2019-03-08, 08:23UTC)

Mar 8 2019, 1:06 PM · cloud-services-team (Kanban), Toolforge

Mar 7 2019

Magnus added a comment to T217141: ‏ListeriaBot appear in recent changes although human (not bot) filter is active in arwiki.

Fixed Listeria.
Don't know anything about ASammourBot.

Mar 7 2019, 9:06 AM · Growth-Team, MediaWiki-Recent-changes

Mar 5 2019

Magnus closed T215380: Content too big! Entity: Q27972199 as Resolved.
Mar 5 2019, 9:59 AM · User-Noarave, Performance Issue, Wikidata-Campsite, Wikidata, Wikimedia-production-error
Magnus committed R2010:a1086af9c050: fixing slow start of new batches (authored by Magnus).
fixing slow start of new batches
Mar 5 2019, 9:19 AM
Magnus committed R2010:a5aea85f12e0: Edit Groups UUID fix (authored by Magnus).
Edit Groups UUID fix
Mar 5 2019, 8:54 AM
Magnus committed R2010:534cd562e8cc: formatting Edit Groups link (authored by Magnus).
formatting Edit Groups link
Mar 5 2019, 8:54 AM
Magnus committed R2010:b03d1a2e0505: formatting Edit Groups link (authored by Magnus).
formatting Edit Groups link
Mar 5 2019, 8:54 AM

Feb 28 2019

Magnus added a comment to T215380: Content too big! Entity: Q27972199.

Removal is running.

Feb 28 2019, 3:41 PM · User-Noarave, Performance Issue, Wikidata-Campsite, Wikidata, Wikimedia-production-error
Magnus added a comment to T215380: Content too big! Entity: Q27972199.

Update: Will remove them with QuickStatements now

Feb 28 2019, 3:34 PM · User-Noarave, Performance Issue, Wikidata-Campsite, Wikidata, Wikimedia-production-error
Magnus added a comment to T215380: Content too big! Entity: Q27972199.

So here is what happens: I create(d) lots of gene/protein items (example) for various species. For many statements, I can create references, as I get them from the upstream source. That paper is one of the often-cited ones, about a determination method.

Feb 28 2019, 3:29 PM · User-Noarave, Performance Issue, Wikidata-Campsite, Wikidata, Wikimedia-production-error

Feb 27 2019

Magnus added projects to T217246: cronjob error mails: Tools, Cloud-VPS.
Feb 27 2019, 2:42 PM · Tools
Magnus created T217246: cronjob error mails.
Feb 27 2019, 2:41 PM · Tools

Feb 19 2019

Magnus changed the status of T216153: ListeriaBot appears as a non bot despite of having a bot flag in euwiki recent changes from Invalid to Resolved.

I have added euwiki to the list of wikis where the bot flag is to be used.

Feb 19 2019, 12:35 PM

Feb 12 2019

Magnus closed T215620: Tool "autodesc" - can't start webservice as Resolved.

That did the trick, thanks!

Feb 12 2019, 3:45 PM · Wikidata, Toolforge

Feb 11 2019

Sjoerddebruin awarded T215620: Tool "autodesc" - can't start webservice a The World Burns token.
Feb 11 2019, 7:54 PM · Wikidata, Toolforge
Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

Tried that, also on login.tools.wmflabs.org (just to be sure). Both say "webservice is not running". Still won't start kubernetes.

Feb 11 2019, 12:47 PM · Wikidata, Toolforge
Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

Thanks, I have rebuild and updated via npm on the kubernetes shell.

Feb 11 2019, 9:48 AM · Wikidata, Toolforge

Feb 9 2019

Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

I have run npm update in the kubernetes shell, but no joy.

Feb 9 2019, 8:45 AM · Wikidata, Toolforge

Feb 8 2019

Magnus created T215620: Tool "autodesc" - can't start webservice.
Feb 8 2019, 3:09 PM · Wikidata, Toolforge

Feb 1 2019

Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Try it now...

Feb 1 2019, 2:40 PM · VPS-Projects
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Testing on dev now. Looks like frequent Lost connection to MySQL server during query errors from the DB replicas.

Feb 1 2019, 1:10 PM · VPS-Projects
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Also, I just clicked on the two examples. They took 133 and 173 seconds, and returned 50 and 1 results, respectively. No 502s, though I have seen those occasionally.

Feb 1 2019, 9:51 AM · VPS-Projects
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

OK, so what appears to happen is that SQL queries timeout and take PetScan with them. Note:

  • I wrote some code that re-arranges certain large queries into smaller ones, which cuts down on the timeouts; that code has been live for weeks
  • That works fine on the dev machine but not reliably on the production machine
  • The dev machine has less resources than production, but is otherwise identical (OS etc)

As this does not fail reproducibly, it's either some odd bug in my code, or some situation on the DB replicas.

Feb 1 2019, 9:49 AM · VPS-Projects

Jan 21 2019

Tagishsimon awarded T214278: Quickstatements, "backend is overloaded" a Heartbreak token.
Jan 21 2019, 2:56 PM · TestMe, Tools, Toolforge
Magnus added a comment to T214278: Quickstatements, "backend is overloaded".

As far as I can see, this is not limited to QuickStatements - when I'm logged out (as just happened again - my "usual" time is at 21 min into the hour), I cannot access SourceMD either. Access normally comes back around 37 min into the hour, even if I press the "log in" button immediately after being logged out.

Jan 21 2019, 2:54 PM · TestMe, Tools, Toolforge
Magnus created T214278: Quickstatements, "backend is overloaded".
Jan 21 2019, 8:25 AM · TestMe, Tools, Toolforge

Jan 17 2019

Magnus added a comment to T193560: Tool keeps falling into permanent 500 error.

Looks like it's back to normal at the moment.

Jan 17 2019, 1:32 PM · cloud-services-team (Kanban), Toolforge

Jan 14 2019

Magnus committed R2010:779f5692926e: bugfix (authored by Magnus).
bugfix
Jan 14 2019, 8:24 PM
Magnus committed R2010:756382987e1a: fixes for Commons, also requires fixed wikidata.php! (authored by Magnus).
fixes for Commons, also requires fixed wikidata.php!
Jan 14 2019, 11:19 AM
Magnus committed R2010:574b1e48ccfe: bugfix (authored by Magnus).
bugfix
Jan 14 2019, 10:20 AM
Magnus committed R2010:aa343fe8abeb: trying to add Commons access by making things more generic (authored by Magnus).
trying to add Commons access by making things more generic
Jan 14 2019, 10:20 AM

Dec 4 2018

Magnus committed R2010:84fb87586d23: typo (authored by Magnus).
typo
Dec 4 2018, 11:04 AM
Magnus committed R2010:c5953eeeafa0: quite option (authored by Magnus).
quite option
Dec 4 2018, 10:58 AM
Magnus committed R2010:68f49e60870e: misc (authored by Magnus).
misc
Dec 4 2018, 9:43 AM
Magnus committed R2010:ed7c36eb5be4: one batch per job (authored by Magnus).
one batch per job
Dec 4 2018, 9:31 AM

Nov 30 2018

Magnus awarded T210836: +2 for Lucas_Werkmeister_WMDE in mediawiki/* a Like token.
Nov 30 2018, 2:24 PM · User-Ladsgroup, Repository-Ownership-Requests

Nov 9 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

I have repeatedly asked if there is a "max concurrency" setting for jsub, as there is for other grid engines. I would consider it rather silly to force every user with such jobs to implement that on their own. For example, I could record it in the database when a job starts, but how do I know it is still running, and hasn't failed? Guess based on last action time? It makes vastly more sense to do that kind of thing in the job scheduler.

Nov 9 2018, 8:23 AM · cloud-services-team (Kanban), Toolforge

Nov 8 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

Checked out sourcemd. That is actually correct; one job for each of the "TODO" ones here:
https://tools.wmflabs.org/sourcemd/?action=batches

Nov 8 2018, 9:48 PM · cloud-services-team (Kanban), Toolforge
Magnus added a comment to T208940: Grid slow on Toolforge.

Looks like the grid caught up with them. One job type seems to be problematic, not sure if it's an aftereffect.

Nov 8 2018, 9:44 PM · cloud-services-team (Kanban), Toolforge
Magnus added a comment to T208940: Grid slow on Toolforge.

Using mix-n-match as an example, two of the cronjob commands:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 8g -N as_import -cwd ./autoscrape_import_bot.php
4,14,24,34,44,54 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 6g -cwd  -N  mnm-microsync ./microsync.php random

Yes, they run often, but usually not for long, and have never had such an issue in the years before.

Nov 8 2018, 3:46 PM · cloud-services-team (Kanban), Toolforge
Magnus added a comment to T208940: Grid slow on Toolforge.

Up to 95 "qw" now...

Nov 8 2018, 8:25 AM · cloud-services-team (Kanban), Toolforge

Nov 7 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

(I picked mix-n-match as an example. Other tools are affected as well)

Nov 7 2018, 10:54 AM · cloud-services-team (Kanban), Toolforge
Magnus created T208940: Grid slow on Toolforge.
Nov 7 2018, 10:53 AM · cloud-services-team (Kanban), Toolforge

Nov 2 2018

Magnus added a comment to T199228: Define an SLO for Wikidata Query Service public endpoint and communicate it.

In many cases, especially bot/background tasks (e.g. Listeria), a lag of hours is not critical. This is also true for many interactive tools, where the user gets some items matching certain criteria.

Nov 2 2018, 2:28 PM · Operations, Wikidata, Wikidata-Query-Service

Nov 1 2018

Magnus closed T208460: wikidata-todo is using 959GB in shared NFS area as Resolved.
Nov 1 2018, 11:01 AM · Data-Services, Toolforge
Magnus closed T208460: wikidata-todo is using 959GB in shared NFS area, a subtask of T206239: 2018-10-04: tools and NFS share cleanup (high usage), as Resolved.
Nov 1 2018, 11:01 AM · cloud-services-team (Kanban)
Magnus added a comment to T208460: wikidata-todo is using 959GB in shared NFS area.

I have removed unused files /shared/dumps and altered the update code to only keep recent ones around.

Nov 1 2018, 10:58 AM · Data-Services, Toolforge

Oct 25 2018

Magnus committed R2010:d4bbd816e688: This patch set, along with an accompanying series for magnustools, adds… (authored by Magnus).
This patch set, along with an accompanying series for magnustools, adds…
Oct 25 2018, 5:13 PM

Oct 23 2018

Magnus added a comment to T207579: imagecopy_tests failing with IndexError.

Try it now please.

Oct 23 2018, 7:27 PM · Patch-For-Review, Tools, Pywikibot-tests, Pywikibot

Oct 22 2018

Magnus added a comment to T207579: imagecopy_tests failing with IndexError.

This should be fixed now in CommonsHelper.

Oct 22 2018, 6:40 PM · Patch-For-Review, Tools, Pywikibot-tests, Pywikibot

Oct 20 2018

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Thanks @Jkbr, that seems to do the trick!

Oct 20 2018, 9:51 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 19 2018

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Can I give you (or can you just get) access to my VM? mixnmatch in the mix-n-match project.

Oct 19 2018, 1:57 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

As a sidenote, this may also contain the solution to T207132 ...

Oct 19 2018, 8:32 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

docker-compose.yml:

# Wikibase with Query Service
#
# This docker-compose example can be used to pull the images from docker hub.
#
# Examples:
#
# Access Wikibase via "http://localhost:8181"
#   (or "http://$(docker-machine ip):8181" if using docker-machine)
#
# Access Query Service via "http://localhost:8282"
#   (or "http://$(docker-machine ip):8282" if using docker-machine)
version: '3'
Oct 19 2018, 8:28 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers