hoo (Marius Hoch)
User

Projects (20)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 12:09 PM (172 w, 20 h)
Availability
Available
LDAP User
Hoo man
MediaWiki User
Hoo man

Recent Activity

Yesterday

Steinsplitter awarded T128661: Don't add other projects links to commons on commons a Pterodactyl token.
Fri, Jan 19, 2:59 PM · Structured-Data-Commons, Commons, Wikidata, Wikimedia-Site-requests

Tue, Jan 16

hoo added a comment to T177550: Only dump up to N entities in each maintenance script run.

How is this coming along?

Tue, Jan 16, 4:35 PM · Wikidata, Datasets-General-or-Unknown
hoo added a comment to T155060: Get some statistics for dump downloads.

That's great for the wikidata dumps. How about the xml/sql dumps? It would sure be nice to have.

Tue, Jan 16, 1:55 PM · Datasets-General-or-Unknown
hoo added a comment to T155060: Get some statistics for dump downloads.

Re-upping this. I know the logs get sucked into a pile of data because https://phabricator.wikimedia.org/T118739 Does anyone process them?

Tue, Jan 16, 12:05 PM · Datasets-General-or-Unknown
hoo closed T162245: Enable GC for HHVM CLI (at least for dump runners) as Resolved.

Snapshot hosts are going directly to php7/stretch, bypassing this issue. See T181029.

Tue, Jan 16, 4:04 AM · Patch-For-Review, HHVM, Operations, Datasets-General-or-Unknown, Wikidata
hoo added a comment to T151876: Consider using pigz (Zopfli) for Wikidata JSON dump.

Given the length of time Wikidata weekly dumps take to run, do we still want to do this? What sort of cpu/memory requirements will it have, compared to gzip?

Tue, Jan 16, 4:00 AM · Datasets-General-or-Unknown, Wikidata

Thu, Jan 11

hoo added a comment to T117534: DCAT-AP: XML produces invalid output with HHVM.

meh. I meant, if we moved the task to run on PHP 7 machine, that would mean we could still get correct XML, right? I.e. it does not require PHP 5?

Thu, Jan 11, 7:04 PM · User-thiemowmde, Upstream, HHVM, Patch-For-Review, Wikidata, Datasets-General-or-Unknown
hoo edited P6114 mwgrep: Wikidata modules that iterate over all entity claims if a Statement is searched for by property label.
Thu, Jan 11, 2:47 AM
hoo closed T166056: Add mw.wikibase.entity:getStatements for convenience as Resolved.
Thu, Jan 11, 2:43 AM · MW-1.31-release-notes (WMF-deploy-2018-01-16 (1.31.0-wmf.17)), Wikidata-Sprint-2018-01-03, Easy, MediaWiki-extensions-WikibaseClient, Wikidata
hoo closed T166056: Add mw.wikibase.entity:getStatements for convenience, a subtask of T112073: Lua in Wikibase (tracking), as Resolved.
Thu, Jan 11, 2:43 AM · MediaWiki-extensions-WikibaseClient, Tracking, Wikidata
hoo closed T166056: Add mw.wikibase.entity:getStatements for convenience, a subtask of T182147: more convenience functions for Lua, as Resolved.
Thu, Jan 11, 2:43 AM · MediaWiki-extensions-WikibaseClient, Wikidata
hoo added a comment to T181936: Give misc dump crons their own host.

How do you see your capacity needs increasing over the next few years? Do you have plans to add new cron jobs to the mix?

Thu, Jan 11, 2:30 AM · Operations, hardware-requests, Datasets-General-or-Unknown, Dumps-Generation

Sat, Jan 6

hoo added a comment to T184322: Enable fine grained lua tracking gradually in client wikis.

The description is a bit unclear: Shall it be enabled for hewiki only initially (and if so for how long) or hewiki + cawiki, trwiki, …?

Sat, Jan 6, 1:02 PM · Patch-For-Review, Wikidata-Sprint-2018-01-03, User-Daniel, Wikidata, MediaWiki-extensions-WikibaseClient, User-Ladsgroup

Fri, Jan 5

hoo moved T177257: ArticlePlaceholder hit counts from bnwiki seem bogus from To Do Next to Done on the ArticlePlaceholder board.
Fri, Jan 5, 1:07 AM · User-Addshore, WMDE-Analytics-Engineering, Wikidata, ArticlePlaceholder
hoo closed T177257: ArticlePlaceholder hit counts from bnwiki seem bogus as Resolved.

I think this is all fine actually :)

Fri, Jan 5, 1:06 AM · User-Addshore, WMDE-Analytics-Engineering, Wikidata, ArticlePlaceholder

Thu, Jan 4

hoo closed T184129: Large number of broken 'pagelinks' on eswiki as Resolved.

Ran again, see P6522#36749.

Thu, Jan 4, 11:02 PM · Wikimedia-maintenance-script-run, Wikimedia-Site-requests
hoo added a comment to P6522 (An Untitled Masterwork).
hoo@terbium:~$ mwscript namespaceDupes.php --wiki eswiki --fix
0 pages to fix, 0 were resolvable.
Thu, Jan 4, 11:01 PM

Mon, Jan 1

hoo renamed T183874: Request creation of wikibase-nearest-neighbors from Request creation of <PROJECT-NAME> VPS project to Request creation of wikibase-nearest-neighbors.
Mon, Jan 1, 8:56 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
hoo created T183874: Request creation of wikibase-nearest-neighbors.
Mon, Jan 1, 8:56 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)

Wed, Dec 27

hoo added a comment to T121105: Mails from MediaWiki seem to get (partially) lost.

@hoo, @Lydia_Pintscher : Still an issue, 18 months later? Or should this task be closed?

Wed, Dec 27, 11:33 AM · MediaWiki-Watchlist, Mail, Operations

Dec 17 2017

hoo added a subtask for T88991: improve Wikidata dumps [tracking]: T179681: Add HDT dump of Wikidata.
Dec 17 2017, 11:46 PM · Wikidata, Tracking, Datasets-General-or-Unknown
hoo added a parent task for T179681: Add HDT dump of Wikidata: T88991: improve Wikidata dumps [tracking].
Dec 17 2017, 11:46 PM · Wikidata

Dec 8 2017

Liuxinyu970226 awarded T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 a Heartbreak token.
Dec 8 2017, 11:33 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo closed T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 as Resolved.

Dump crons are back (af1f7dabee931dbdb7366b7be1f93698f2c56108), so I expect dumps to arrive in time next week!

Dec 8 2017, 12:47 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo closed T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007, a subtask of T88991: improve Wikidata dumps [tracking], as Resolved.
Dec 8 2017, 12:47 AM · Wikidata, Tracking, Datasets-General-or-Unknown

Dec 6 2017

hoo closed T182243: Fatal in AffectedPagesFinder: Call to a member function getSiteLinkChanges() on a non-object (string) as Resolved.
Dec 6 2017, 9:53 PM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Wikimedia-log-errors, MediaWiki-extensions-WikibaseRepository, MediaWiki-extensions-WikibaseClient, Wikidata
hoo closed T182243: Fatal in AffectedPagesFinder: Call to a member function getSiteLinkChanges() on a non-object (string), a subtask of T180244: 1.31.0-wmf.11 deployment blockers, as Resolved.
Dec 6 2017, 9:53 PM · Release, Train Deployments
hoo added a comment to T182243: Fatal in AffectedPagesFinder: Call to a member function getSiteLinkChanges() on a non-object (string).

This was due to f77b47c5e30c297a43e4ce354c1dfb4a29a638fa being deployed to early, it should have not been in wmf11.

I'm reverting it on the branch now which should fix this.

Dec 6 2017, 9:24 PM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Wikimedia-log-errors, MediaWiki-extensions-WikibaseRepository, MediaWiki-extensions-WikibaseClient, Wikidata
hoo added a comment to T182243: Fatal in AffectedPagesFinder: Call to a member function getSiteLinkChanges() on a non-object (string).

This was due to f77b47c5e30c297a43e4ce354c1dfb4a29a638fa being deployed to early, it should have not been in wmf11.

Dec 6 2017, 9:12 PM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Wikimedia-log-errors, MediaWiki-extensions-WikibaseRepository, MediaWiki-extensions-WikibaseClient, Wikidata
hoo triaged T182243: Fatal in AffectedPagesFinder: Call to a member function getSiteLinkChanges() on a non-object (string) as Unbreak Now! priority.
Dec 6 2017, 8:52 PM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Wikimedia-log-errors, MediaWiki-extensions-WikibaseRepository, MediaWiki-extensions-WikibaseClient, Wikidata
hoo removed a project from T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently: Patch-For-Review.
Dec 6 2017, 7:29 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
Addshore awarded T179008: Wikidata change propagation: introduce --batch-grace parameter to replace --dispatch-interval a Goat token.
Dec 6 2017, 10:36 AM · User-Addshore, MediaWiki-extensions-WikibaseRepository, Wikidata
Envlh awarded T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 a Heartbreak token.
Dec 6 2017, 8:50 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown

Dec 4 2017

hoo lowered the priority of T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 from Unbreak Now! to High.

Downgrading this, as this seems to be working again.

Dec 4 2017, 6:48 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

So I merged the patch that makes sure we only actually collect this data if not in CLI mode.

Dec 4 2017, 6:01 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown

Dec 2 2017

hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

How long do these run? The sample rate in config is set to be extremely low. So perhaps:

  • The buffering class buffers things that won't even be saved

That's definitely the case here, given that the sample rate is only applied later on.

Dec 2 2017, 9:41 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo edited projects for T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007, added: Performance-Team; removed MediaWiki-extensions-WikibaseRepository.
Dec 2 2017, 4:24 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo assigned T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 to aaron.

I don't see a clear way to fix, thus reassigning to Aaron.

Dec 2 2017, 3:37 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

So I finally managed to track this down:

Dec 2 2017, 3:10 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown

Dec 1 2017

hoo renamed T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 from Wikidata truthy nt dumpers stuck with 100% CPU on snapshot1007 to Wikidata entity dumpers stuck with 100% CPU on snapshot1007.
Dec 1 2017, 2:47 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T179317: Varnish and Apache debug tools and logs for hoo.

Regarding the appsevers, the canary ones are indeed (mostly) enough. But for the Varnishes, having access to the actual ones would be very nice… I don't see that much value in the canary caches for me here.

Dec 1 2017, 2:44 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

When changing CacheRetrievingEntityRevisionLookup to always use it's underlying EntityRevisionLookup ($this->lookup) on mwdebug1001, both the call with --no-cache and without it show very similar memory usage behavior.

Dec 1 2017, 1:31 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

Reverting 795350da2e5c49efa66c1950bd034f46aeb3768a also doesn't seem to make any difference.

Dec 1 2017, 12:52 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

When changing CacheRetrievingEntityRevisionLookup to always use it's underlying EntityRevisionLookup ($this->lookup) on mwdebug1001, both the call with --no-cache and without it show very similar memory usage behavior.

Dec 1 2017, 12:48 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo raised the priority of T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 from High to Unbreak Now!.
Dec 1 2017, 12:05 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown

Nov 28 2017

hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

So I just ran one JSON dumper with --no-cache and one without on mwdebug1001 (with HHVM)… the results are baffling:

Nov 28 2017, 11:57 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown

Nov 27 2017

hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

I just tested this briefly locally by dumping my wiki several times (up to 100 times), but I couldn't see any kind of memory leak (but again, I dumped the same entities several times).

Nov 27 2017, 6:28 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.

Not a single entity managed to get dumped since I created this ticket more than 7h ago, thus I killed all related processes now.
Given that we will have a new dump this week anyway, I wont bother re-starting it manually.

Nov 27 2017, 6:10 PM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo created T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007.
Nov 27 2017, 11:37 AM · MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), Performance-Team, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T179317: Varnish and Apache debug tools and logs for hoo.

@hoo:
Which hosts are we looking at then, varnish servers and the app servers?

Yes.

Nov 27 2017, 11:24 AM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests

Nov 21 2017

hoo updated the task description for T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.
Nov 21 2017, 4:22 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown

Nov 20 2017

hoo closed T180934: Wikidata json dumps filling /var/log as Resolved.

Both scripts look fine again and the dumpers are running… sorry for the mess :/

Nov 20 2017, 10:24 AM · User-ArielGlenn, Dumps-Generation, Wikidata

Nov 16 2017

hoo updated subscribers of T180727: WikibaseClient resources are broken in WMF.

This at least breaks the client link item widget, also probably user scripts.

Nov 16 2017, 7:19 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Performance-Team (Radar), User-Addshore, Patch-For-Review, MediaWiki-ResourceLoader, MediaWiki-extensions-WikibaseClient, Wikidata
hoo closed T117534: DCAT-AP: XML produces invalid output with HHVM, a subtask of T94277: Convert snapshot hosts to use HHVM and trusty, as Resolved.
Nov 16 2017, 12:33 PM · Dumps-Generation, Operations, Patch-For-Review, HHVM
hoo closed T117534: DCAT-AP: XML produces invalid output with HHVM as Resolved.

The new version works on hhvm as well as on Zend. We can switch this back to hhvm now…thanks for tackling this.

Nov 16 2017, 12:33 PM · User-thiemowmde, Upstream, HHVM, Patch-For-Review, Wikidata, Datasets-General-or-Unknown
hoo renamed T117534: DCAT-AP: XML produces invalid output with HHVM from DCAT-AP: XML output no longer valid to DCAT-AP: XML produces invalid output with HHVM.
Nov 16 2017, 12:28 PM · User-thiemowmde, Upstream, HHVM, Patch-For-Review, Wikidata, Datasets-General-or-Unknown

Nov 14 2017

hoo added a comment to T180177: Move wikidata dump generation 3 days ahead (or 4 days after).

Personally I would like to avoid running them over the weekend… but that's not a show stopper here, I just want to keep the number of moving parts down during the weekend.

Nov 14 2017, 11:05 PM · Dumps-Generation, Wikidata
hoo closed T180048: Wikidata dump significantly slower than before as Resolved.

From next week on this should be ok again.

Nov 14 2017, 11:04 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Patch-For-Review, Dumps-Generation, Wikidata

Nov 13 2017

hoo added a comment to T117534: DCAT-AP: XML produces invalid output with HHVM.

@ArielGlenn @hoo As this is an effect of hhvm would converting the script to python be a viable solution or is PHP a must for it to fit with other dump related mechanisms? (Haven't checked if it is viable codewise but asking to see what options there are).

Nov 13 2017, 7:29 PM · User-thiemowmde, Upstream, HHVM, Patch-For-Review, Wikidata, Datasets-General-or-Unknown
hoo claimed T180048: Wikidata dump significantly slower than before.

I finally found time to look into this: This is due to T178247 and the problems I anticipated in T177486#3718743.

Nov 13 2017, 5:35 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Patch-For-Review, Dumps-Generation, Wikidata
hoo added a comment to T179312: robots.txt prevents indexing of Special:EntityData.

If the search result in the screenshot is the only reason this ticket was created, I strongly suggest to close it, because stripping duplicates from search indexes is actually intended.

Nov 13 2017, 5:11 PM · MediaWiki-extensions-WikibaseRepository, Wikidata

Nov 10 2017

hoo added a comment to T180048: Wikidata dump significantly slower than before.

Duplicate of T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently? I'm aware of these huge fluctuations in time, but wasn't able to look into this in detail, yet.

Nov 10 2017, 1:13 AM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Patch-For-Review, Dumps-Generation, Wikidata

Nov 5 2017

hoo added a comment to T179317: Varnish and Apache debug tools and logs for hoo.

Can you elaborate what you need in specific to debug wikidata performance problems? We can arrange access to all the logs you need, but perf-roots grants full root access to nearly half the servers in production.

Nov 5 2017, 9:10 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
hoo added a comment to T114904: Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs.

Note: I just also found T179793: Consider dropping the "wb_items_per_site.wb_ips_site_page" index while looking at this… maybe this can be done at once?!

Nov 5 2017, 7:39 PM · DBA, Wikidata
hoo created T179793: Consider dropping the "wb_items_per_site.wb_ips_site_page" index.
Nov 5 2017, 7:37 PM · DBA, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo added a comment to T114904: Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs.

Giving the size of the table, changing this shouldn't be overly horrible. It's a fair bit of migration work… but I assume doing this for maintenance queries and consistency is worth it.

Nov 5 2017, 7:31 PM · DBA, Wikidata
hoo added a comment to T178459: Long running queries from pltools unlikely to finish.

The second query can be expressed as:

Nov 5 2017, 7:23 PM · Tools
hoo added a comment to T170779: Wikidata search suggestions do not display on screen if character whose decomposition contains nukta is present in search query.

@Snaterlicious, @hoo, @thiemowmde Do you know why the check is there and what it meant to be doing? @tstarling raised the following concern:

The search term is normalized by the server using $wgContLang->normalize(), which potentially includes transformations beyond NFC, especially if the content language is Arabic or Malayalam. So even if you do client-side NFC using the same version of Unicode as the server, there is at least a hypothetical possibility of a hang.
Nov 5 2017, 7:01 PM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), Wikidata-Former-Sprint-Board, User-Smalyshev, Discovery-Search (Current work), ValueView, MediaWiki-extensions-WikibaseRepository, Wikidata

Nov 2 2017

hoo added a comment to T179317: Varnish and Apache debug tools and logs for hoo.

@hoo: I've reviewed the L3 document, and I don't see your signature on it. This is likely due to the fact you've had server access since before phabricator. However, we like to have users sign this document when we make any changes to their access.

Can you please review and sign the L3 document before next Monday's ops meeting?

Nov 2 2017, 9:50 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn awarded T179317: Varnish and Apache debug tools and logs for hoo a Like token.
Nov 2 2017, 11:34 AM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests

Nov 1 2017

hoo updated the task description for T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.
Nov 1 2017, 8:22 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
hoo updated the task description for T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.
Nov 1 2017, 8:22 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.

This also indicates that our current changes had a huge impact… but also that ad332804b1fea069043d14d0195f6fe2ed5a6f4b and/ or 3164215d0d790f37cc1cf386ef22a188e81a10d0 might have a huge negative impact here :/

Nov 1 2017, 8:21 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.

Run time for the full TTL dump (the data diff is from the gzipped file):

Nov 1 2017, 8:14 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
Krinkle awarded T163551: Huge number of duplicate rows in wb_terms a Orange Medal token.
Nov 1 2017, 4:44 AM · Patch-For-Review, User-Ladsgroup, User-aude, Wikidata-Former-Sprint-Board, Wikidata, MediaWiki-extensions-WikibaseRepository

Oct 31 2017

hoo added a comment to T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.

(Probably) due to the DataModel updates the current JSON dump was created in just 25 hours, compared to ~34-35h last week. (This is data from one run only, so not overly reliable… but the difference is huge)

Oct 31 2017, 5:18 PM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown
hoo added a comment to T121274: Provide an RDF mapping for external identifiers.

In order to update that we need to run rebuildPropertyInfo script I guess.

Oct 31 2017, 4:43 PM · MW-1.31-release-notes (WMF-deploy-2017-10-10 (1.31.0-wmf.3)), User-Smalyshev, User-Daniel, Wikidata-Former-Sprint-Board, Wikidata-Sprint-2015-12-01, Patch-For-Review, MediaWiki-extensions-WikibaseRepository, Wikidata, Story
hoo added a comment to T179312: robots.txt prevents indexing of Special:EntityData.

Well, we could allow this, I guess… but we should at least set a canonical URL (or one per output?) as header (we can't put it in the html here, as there's none).

Oct 31 2017, 4:34 PM · MediaWiki-extensions-WikibaseRepository, Wikidata
hoo updated the task description for T179156: 503 spikes and resulting API slowness starting 18:45 October 26.
Oct 31 2017, 4:18 PM · Release-Engineering-Team (Watching / External), Patch-For-Review, Traffic, Wikimedia-Incident, Operations, ORES, Wikidata, Scoring-platform-team

Oct 30 2017

hoo created T179317: Varnish and Apache debug tools and logs for hoo.
Oct 30 2017, 5:41 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
hoo updated the task description for T178652: Wikidata dispatchers should use a LockManager with a short TTL.
Oct 30 2017, 4:57 PM · User-Addshore, Patch-For-Review, Wikimedia-Site-requests, Wikidata
hoo added a comment to T177257: ArticlePlaceholder hit counts from bnwiki seem bogus.

I looked at this again earlier today and there are actually a lot of requests to APs on bnwiki… all coming from Facebook IP ranges (ipv6), but with different UAs and organically looking patterns.

Oct 30 2017, 2:22 PM · User-Addshore, WMDE-Analytics-Engineering, Wikidata, ArticlePlaceholder
hoo added a comment to T146838: Allow seeking in UsageLookup::getPagesUsing results.

This is currently running into timeouts from time to time:

Oct 30 2017, 12:08 PM · MediaWiki-extensions-WikibaseClient, Wikidata
hoo updated the task description for T179156: 503 spikes and resulting API slowness starting 18:45 October 26.
Oct 30 2017, 11:51 AM · Release-Engineering-Team (Watching / External), Patch-For-Review, Traffic, Wikimedia-Incident, Operations, ORES, Wikidata, Scoring-platform-team
hoo closed T178180: Enable RDF mapping for external identifiers for Wikidata.org as Resolved.

This has been enabled (again) and this time it's here to stay! \o/

Oct 30 2017, 11:49 AM · User-Smalyshev, User-Daniel, Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata, User-Ladsgroup
hoo closed T178180: Enable RDF mapping for external identifiers for Wikidata.org, a subtask of T121274: Provide an RDF mapping for external identifiers, as Resolved.
Oct 30 2017, 11:49 AM · MW-1.31-release-notes (WMF-deploy-2017-10-10 (1.31.0-wmf.3)), User-Smalyshev, User-Daniel, Wikidata-Former-Sprint-Board, Wikidata-Sprint-2015-12-01, Patch-For-Review, MediaWiki-extensions-WikibaseRepository, Wikidata, Story
hoo closed T179060: Dispatchers occasionally seem to "freeze" for certain wikis as Resolved.
Oct 30 2017, 10:41 AM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo closed T179060: Dispatchers occasionally seem to "freeze" for certain wikis, a subtask of T108944: [Epic] Improve change dispatching, as Resolved.
Oct 30 2017, 10:40 AM · Epic, Wikidata
hoo added a comment to T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently.

While T178247: Use a retrieve only CachingEntityRevisionLookup for dumps will certainly make the dumps much faster, it will only do so (noticeably) on HHVM. This is because we split the cache between HHVM and Zend (see below), thus the (currently) Zend dumpers wont profit from the cache which is probably mostly populated in the HHVM version of the cache (as all app server run HHVM).
There are some other maintenance scripts using Zend which might also write into this cache… so maybe this will still help something, though.

Oct 30 2017, 9:36 AM · Wikibase-DataModel, Wikidata, Datasets-General-or-Unknown

Oct 29 2017

hoo moved T179060: Dispatchers occasionally seem to "freeze" for certain wikis from Proposed to Review on the Wikidata-Former-Sprint-Board board.
Oct 29 2017, 9:41 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo added a project to T179060: Dispatchers occasionally seem to "freeze" for certain wikis: Wikidata-Former-Sprint-Board.
Oct 29 2017, 9:41 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo added a comment to T179060: Dispatchers occasionally seem to "freeze" for certain wikis.

The above change will allow us to (almost) increase our maximum throughput by a factor of 8, thus should fix these issues for now at least.

Oct 29 2017, 9:41 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo added a comment to T179060: Dispatchers occasionally seem to "freeze" for certain wikis.

There are actually a lot of changes related to zhwiki happening:

Oct 29 2017, 8:58 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo added a comment to T179060: Dispatchers occasionally seem to "freeze" for certain wikis.

This just happened again, zhwiki got backlogged by 8h+.

Oct 29 2017, 8:43 PM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), Wikidata-Former-Sprint-Board, MediaWiki-extensions-WikibaseRepository, Wikidata
hoo moved T179235: "Create article" with translation causes "this.config is undefined mw.cx.SiteMapper.prototype.getPageUrl" from To Do Next to Done on the ArticlePlaceholder board.
Oct 29 2017, 10:50 AM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), ArticlePlaceholder, Wikidata
hoo closed T179235: "Create article" with translation causes "this.config is undefined mw.cx.SiteMapper.prototype.getPageUrl" as Resolved.
Oct 29 2017, 10:50 AM · MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), ArticlePlaceholder, Wikidata
hoo moved T132223: Track pageviews of specific pages that are rendered with ArticlePlaceholders from To Do Next to Backlog on the ArticlePlaceholder board.
Oct 29 2017, 10:50 AM · ArticlePlaceholder, Wikidata
hoo moved T141770: Newly created article should be connected to the Wikidata item from To Do Next to Backlog on the ArticlePlaceholder board.
Oct 29 2017, 10:50 AM · Story, Wikidata, ArticlePlaceholder
hoo moved T150786: [Task] Investigate options for automatically connecting article created through ArticlePlaceholder with corresponding Wikidata item from To Do Next to Backlog on the ArticlePlaceholder board.
Oct 29 2017, 10:50 AM · Wikidata, ArticlePlaceholder
hoo added a comment to T177257: ArticlePlaceholder hit counts from bnwiki seem bogus.

Apparently the refine query changed, I'm currently running the new query to see whether this explains the differences.

Oct 29 2017, 10:49 AM · User-Addshore, WMDE-Analytics-Engineering, Wikidata, ArticlePlaceholder