Page MenuHomePhabricator

So9q (Dennis Priskorn)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Sep 16 2019, 11:47 AM (231 w, 1 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
So9q [ Global Accounts ]

Recent Activity

Sat, Feb 17

So9q added a comment to T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster.

To add to this, the two-index approach has another rather beautiful property:

  1. It is important to understand that real-time updates have an inherent price. An engine that supports real-time updates can never be as fast as a read-only engine. But with the approach outlined in https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update we kind of get the best of both worlds:
  1. Combining the two indexes gives you full SPARQL 1.1 Update capability. There is an unavoidable penalty in runtime, but if the amount of updates is small relative to the size of the data already there (for Wikidata, we are talking millions of updates in a day vs. billions of triples already in the database), the penalty is relatively small.
  1. But you can also choose to only ask the large index. Than you get results on a snapshot of the data from a certain (known) date that lies up to 24 hours in the past. But you get the result with maximum speed.
  1. Since the approach naturally enables this choice for each individual query, every user can decide on the trade-off for themselves for each query.
Sat, Feb 17, 8:37 PM · Wikidata, Wikidata-Query-Service
So9q added a comment to T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster.

Most of SPARQL 1.1 is now supported with few exceptions. QLever is looking more promising by the day. Very nice work!
See https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard for details

Sat, Feb 17, 8:24 PM · Wikidata, Wikidata-Query-Service
So9q added a comment to T356525: Very long response headers cause a 502 response from the Toolforge front proxy.

I tested on my VPS in OVH and got:

image.png (42×1 px, 9 KB)

with a default proxy configuration like so:

Sat, Feb 17, 2:58 PM · Toolforge

Sat, Feb 10

So9q awarded T329368: tiny subsets from Wikidata a Love token.
Sat, Feb 10, 9:54 AM · Wikimania-Hackathon-2023, Wikidata

Fri, Feb 2

So9q added a comment to T356195: Request creation of topic-curator VPS project.

To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=video+game&terms=computer+game&terms=videogames
Select all
Click submit
->

image.png (568×787 px, 78 KB)

Fri, Feb 2, 12:50 PM · Cloud-VPS (Project-requests)
So9q added a comment to T356195: Request creation of topic-curator VPS project.
Fri, Feb 2, 12:07 PM · Cloud-VPS (Project-requests)
So9q closed T356195: Request creation of topic-curator VPS project as Resolved.

Big thanks for taking a deeper look into this and for the suggestion to adjust the encoding. I'll close this as resolved for now, while I try diving deeper into the issue and try to make it work.
Locally on my machine everything works fine. I can send batches as large as 5000 items to QS with no problems which translates to 10k qs-lines to edit.

Fri, Feb 2, 11:57 AM · Cloud-VPS (Project-requests)

Tue, Jan 30

So9q updated the task description for T356195: Request creation of topic-curator VPS project.
Tue, Jan 30, 4:53 PM · Cloud-VPS (Project-requests)
So9q updated the task description for T356195: Request creation of topic-curator VPS project.
Tue, Jan 30, 4:52 PM · Cloud-VPS (Project-requests)
So9q created T356195: Request creation of topic-curator VPS project.
Tue, Jan 30, 4:49 PM · Cloud-VPS (Project-requests)

Dec 14 2023

So9q created T353433: Support adding demonstrates form also.
Dec 14 2023, 10:54 AM · Luthor

Aug 6 2023

So9q added a comment to T223108: Build simple tool for checking Wikidata items against ShEx schema.

@VIGNERON the entityshape api is a very limited implementation of shex if I understood correctly.

Aug 6 2023, 7:15 AM · Wikimedia-Hackathon-2019, EntitySchema, Wikidata

Jul 19 2023

So9q created T342228: Support non-<ref> references.
Jul 19 2023, 11:58 AM · ProveIt-Gadget
So9q created T342227: Add support for extracting from {{webarchive}} template when no other template is found.
Jul 19 2023, 11:53 AM · Internet-Archive, ProveIt-Gadget
So9q created T342226: Add support for automatically extracting the Wayback Machine archive-date from a WM URL.
Jul 19 2023, 11:46 AM · ProveIt-Gadget
So9q created T342225: Add support for automatic lookup based on an archive link and extracting the original URL automatically.
Jul 19 2023, 11:44 AM · ProveIt-Gadget
So9q created T342221: Add support for Wayback Machine date format.
Jul 19 2023, 11:15 AM · Internet-Archive, ProveIt-Gadget
So9q created T342220: Add link from DOI to Scholia .
Jul 19 2023, 11:13 AM · ProveIt-Gadget
So9q renamed T342218: Add support for parsing the CiteQ template also from ProveIt should support parsing the CiteQ template also to Add support for parsing the CiteQ template also.
Jul 19 2023, 11:12 AM · ProveIt-Gadget
So9q created T342218: Add support for parsing the CiteQ template also.
Jul 19 2023, 11:11 AM · ProveIt-Gadget

Jun 26 2023

So9q added a comment to T229775: Validate EntitySchema and prevent invalid ones from being saved.

I also would like this. I recently saved a broken schema and had issues during validation.

Jun 26 2023, 8:20 PM · EntitySchema, Wikidata

Mar 13 2023

So9q updated the task description for T331852: Contribute to Wikipedia Article Reference Explorer.
Mar 13 2023, 10:16 AM · Internet-Archive, Wikimedia-Hackathon-2023
So9q created T331852: Contribute to Wikipedia Article Reference Explorer.
Mar 13 2023, 10:14 AM · Internet-Archive, Wikimedia-Hackathon-2023

Feb 21 2023

So9q created T330141: PAWS does not warn about all files (even hidden) might be accessible from the internet.
Feb 21 2023, 11:18 AM · Security, PAWS

Feb 20 2023

So9q added a comment to T330028: Enable show hidden files in PAWS JupyterLab.

I don’t need this task anymore as I found the real terminal and successfully edited the hidden file as I wanted.

Feb 20 2023, 4:00 PM · Security, PAWS
So9q updated the task description for T330028: Enable show hidden files in PAWS JupyterLab.
Feb 20 2023, 3:59 PM · Security, PAWS
So9q added a comment to T330028: Enable show hidden files in PAWS JupyterLab.
In T330028#8628160, @EpicPupper wrote:

PAWS files are viewable by anyone anyways.

Feb 20 2023, 3:58 PM · Security, PAWS

Feb 19 2023

So9q added a project to T330028: Enable show hidden files in PAWS JupyterLab: Security.
Feb 19 2023, 6:53 PM · Security, PAWS
So9q created T330028: Enable show hidden files in PAWS JupyterLab.
Feb 19 2023, 6:52 PM · Security, PAWS

Feb 14 2023

So9q added a comment to T329535: Cloud Ceph outage 2023-02-13.

Thanks for working hard on this to get it sorted out and sharing the root cause analysis. :)

Feb 14 2023, 8:14 AM · User-notice-archive, cloud-services-team (FY2023/2024-Q1-Q2), Goal, Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-User, User-dcaro, Wikimedia-Incident, Cloud-VPS

Jan 21 2023

So9q added a comment to T278162: gloss text entry box is too short and hard to edit.

I investigated this as I have trouble in Firefox on iOS (iphone 7) editing long lines.
I added a few fixes in my css see https://www.wikidata.org/wiki/User:So9q/vector-2022.css but I suggest switching to textarea and so user can set a height and avoid scroll issues when mobile editing lexemes.

Jan 21 2023, 11:40 AM · Wikidata Lexicographical data, Wikidata

Dec 2 2022

So9q updated the task description for T324350: Document max request size of wikibase.cloud instances.
Dec 2 2022, 5:02 PM · Wikibase Cloud
So9q updated the task description for T324350: Document max request size of wikibase.cloud instances.
Dec 2 2022, 5:02 PM · Wikibase Cloud
So9q updated the task description for T324350: Document max request size of wikibase.cloud instances.
Dec 2 2022, 5:01 PM · Wikibase Cloud
So9q created T324350: Document max request size of wikibase.cloud instances.
Dec 2 2022, 4:59 PM · Wikibase Cloud

Nov 21 2022

So9q added a comment to T288666: Create a user script for showing statistics on Wikipedia articles about the gender of those linked in the article.

I just made some changes, and it should now fix the issues you mentioned (also, I just removed the margin-left: 2em as it wasn't valid to apply anyway and had no effect)

Thanks -- looks great! I also was thinking that including all the non-people links in the stats didn't make a whole lot of sense so I added another parameter to the API call all that needs to be there if the non-gendered links are including in the API response. Let me know if you don't like that and I can either switch the API back or you can just add the all to the API response as below:

__________ All data: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)&all
Just gender subset: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)

I also loaded the script into my global javascript to make sure it works on other languages: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/global.js
In general it does! Pretty interesting to see how different languages end up with different distributions for the same article, especially when it comes to what links make it into the lede paragraph. Obviously would benefit from greater localization -- e.g., not always using English labels -- and the table gets switched around a little bit in Arabic or other right-to-left languages, but that feels likely beyond the scope of this initial working prototype :)

To me, this feels pretty complete. Let me know if you had additional thoughts / improvements you wanted to make though. I think just remaining then would be whether you want to present it at the showcase on Tuesday: https://wikimania.wikimedia.org/wiki/2021:Hackathon/Showcase

Nov 21 2022, 9:09 AM · research-ideas, Wikimania-Hackathon-2021

Nov 12 2022

So9q added a comment to T43686: [Story] Ability to filter recent changes and watchlist on Wikidata.

See discussion here https://github.com/openstreetmap/operations/issues/764#issuecomment-1312407828

Nov 12 2022, 9:23 AM · Growth-Team-Filtering, Growth-Team, MediaWiki-Watchlist, Story, Wikidata, patch-welcome, MediaWiki-extensions-WikibaseRepository

Nov 11 2022

So9q added a comment to T272088: Logging (pywiki module) always verbose if enabled.

7.3 seems to work thanks god ...

Nov 11 2022, 6:37 PM · Pywikibot
So9q added a comment to T322755: shut down cloud-vps 'maps' project.

I never heard about this project. Would it not be possible to move to the kubernetes cluster?

Nov 11 2022, 1:38 PM · User-Sandra_Fauconnier_WMSE, cloud-services-team (Kanban), Cloud-VPS (Debian Stretch Deprecation)

Oct 21 2022

So9q awarded T321390: Provide low-barrier way to create/edit items based on a schema a Like token.
Oct 21 2022, 1:21 PM · EntitySchema, Wikidata
So9q added a comment to T321390: Provide low-barrier way to create/edit items based on a schema .

Thanks for taking the time to submit this ticket. I support it.
Cradle is a good start, but it lacks many important features.
For example it lacks the check for duplicates that is present e.g in the Wikidata lexeme forms tool or through reconciliation in openrefine.

Oct 21 2022, 1:20 PM · EntitySchema, Wikidata

Jun 25 2022

So9q added a comment to T35470: Create API for mass deleting pages (aka Special:Nuke).

Added Wikibase tags which was requested by Lydia.

Jun 25 2022, 11:09 AM · Product-Feature, Wikibase Suite Team, Wikibase and Wikidata Architecture Overview, Wikibase (3rd party installations), MediaWiki-Action-API, MediaWiki-extensions-Nuke
So9q added projects to T35470: Create API for mass deleting pages (aka Special:Nuke): Wikibase (3rd party installations), Wikibase and Wikidata Architecture Overview.
Jun 25 2022, 11:08 AM · Product-Feature, Wikibase Suite Team, Wikibase and Wikidata Architecture Overview, Wikibase (3rd party installations), MediaWiki-Action-API, MediaWiki-extensions-Nuke
So9q added a comment to T35470: Create API for mass deleting pages (aka Special:Nuke).

I am loading a large number of elements in a local Wikibase installation and I often need to reprocess the dataset.
A batch delete function to mass delete elements inserted by a bot user would be useful.

Jun 25 2022, 8:35 AM · Product-Feature, Wikibase Suite Team, Wikibase and Wikidata Architecture Overview, Wikibase (3rd party installations), MediaWiki-Action-API, MediaWiki-extensions-Nuke

Jun 22 2022

So9q added a comment to T287164: Improve bulk import via API.

Any news on this? Is something hindering it from being triaged?

Jun 22 2022, 9:02 PM · Product-Feature, Wikibase Suite Team, Wikidata, [DEPRECATED] wdwb-tech, Wikibase (3rd party installations)
So9q awarded T287164: Improve bulk import via API a Like token.
Jun 22 2022, 9:01 PM · Product-Feature, Wikibase Suite Team, Wikidata, [DEPRECATED] wdwb-tech, Wikibase (3rd party installations)

May 4 2022

So9q added a comment to T272088: Logging (pywiki module) always verbose if enabled.

I'm leaning towards forking pywikibot and removing the offending lines in bot.py causing the verbose log of files.

May 4 2022, 10:16 PM · Pywikibot
So9q added a comment to T272088: Logging (pywiki module) always verbose if enabled.

I see. 'pywiki' logger will be initialized to level 11 and there is no easy way to change that. As a work-around you can modify the logger after it was initialized e.g.

import logging
import pywikibot
pywikibot.output('This will initialize the logger')
logger = logging.getLogger('pywiki')
logger.setLevel(logging.WARNING)
pywikibot.output('This message will not be logged anymore')
pywikibot.log('Also verbose lgs are hidden')
pywikibot.warning('Warnings are still logged')
pywikibot.error('Errors are logged too.)

This workaround did not work for me :/

I commented out the whole content of bot.py: writelogheader() and that did the trick!

@So9q, starting to work on this item now; by "did not work" do you mean you still see the ~300 lines of verbose log output? What platform are you running on?

May 4 2022, 10:10 PM · Pywikibot

Apr 11 2022

aborrero awarded T299039: All started jobs failed on Kubernetes during 24h with no visible error or output a Like token.
Apr 11 2022, 11:08 AM · Toolforge, Kubernetes
So9q closed T299039: All started jobs failed on Kubernetes during 24h with no visible error or output, a subtask of T285944: Toolforge: beta phase for the new jobs framework, as Resolved.
Apr 11 2022, 10:04 AM · cloud-services-team, Toolforge Jobs framework
So9q closed T299039: All started jobs failed on Kubernetes during 24h with no visible error or output as Resolved.

It now happens again.

Apr 11 2022, 10:04 AM · Toolforge, Kubernetes

Apr 8 2022

So9q added a comment to T304943: Make Wikibase error message "Malformed input" more meaningful .

I would love to see this improved. I often get this generic error when an empty string is passed with a string property. I would also really like to know which property was related to the input error so I don't have to scan through tens of properties to find the anomaly manually.

Apr 8 2022, 5:26 AM · MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), Wikidata Dev Team (Sprint-∞), patch-welcome, Wikidata, Wikidata-Campsite, good first task

Mar 31 2022

So9q created T305117: UI usability bug when swiping right.
Mar 31 2022, 4:54 AM · Wikipedia-iOS-App-Backlog, iOS-app-Bugs

Mar 30 2022

So9q added a comment to T289561: Evaluate Apache Rya as alternative to Blazegraph.

Based on the discussion above I suggest closing this task.

Mar 30 2022, 4:44 AM · Wikidata, Wikidata-Query-Service

Mar 18 2022

So9q renamed T301635: Update the infrastructure diagrams detailing the interactions of the WMF wikibase and blazegraph stack from Create/publish PlanUML diagrams detailing the interactions of the WMF tech stack to Update the infrastructure diagrams detailing the interactions of the WMF wikibase and blazegraph stack.
Mar 18 2022, 10:06 AM · WMF-Architecture-Team
So9q added a comment to T301635: Update the infrastructure diagrams detailing the interactions of the WMF wikibase and blazegraph stack.

They look good. Thanks. I’m missing Wikibase related infrastructure there though.
The blazegraph issue is the fact that WMDE and the search platform tream are unsure if the backend can handle any more triples without catastrophic failure. But it might not be possible to detect anyway using graphs like these.

Mar 18 2022, 10:03 AM · WMF-Architecture-Team

Mar 10 2022

So9q awarded T303488: [Gadget support] Add a Javascript function to create a new portlet section similar to mw.util.addPortletLink a Like token.
Mar 10 2022, 7:17 AM · MW-1.41-notes (1.41.0-wmf.20; 2023-08-01), Patch-For-Review, Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2)

Mar 9 2022

So9q created T303431: English Wikipedia Rest documentation does not show up.
Mar 9 2022, 5:52 PM · RESTBase-API

Feb 26 2022

So9q added a comment to T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster.

@Hannah_Bast informed in the last WDQS scaling meeting that QLever could have 2 indexes to provide near-realtime queries. See https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update

Feb 26 2022, 12:21 PM · Wikidata, Wikidata-Query-Service

Feb 18 2022

So9q added a comment to T195469: Warning to avoid creating duplicates of Lexemes.

Hangor and Ordia/Lexeme forms has this already. I use those to create lexemes because it is "safer" until this ticket is fixed. Unfortunately neither Hangor nor Lexeme forms support creating phrases or idioms yet.

Feb 18 2022, 6:12 AM · Wikidata Lexicographical data, Wikidata

Feb 13 2022

So9q created T301635: Update the infrastructure diagrams detailing the interactions of the WMF wikibase and blazegraph stack.
Feb 13 2022, 2:06 PM · WMF-Architecture-Team
So9q added a comment to T301227: Create RDF dataset for testing alternatives to Blazegraph.

related to https://phabricator.wikimedia.org/T260687 maybe a duplicate?

Feb 13 2022, 2:03 PM · Wikidata

Feb 10 2022

So9q added a comment to T199197: [2.11] Integrate Citoid in Wikidata.

Is there a reason this issue has stalled?

Feb 10 2022, 8:28 PM · Citoid, WMF-Legal, Wikidata, Epic

Feb 8 2022

So9q renamed T301227: Create RDF dataset for testing alternatives to Blazegraph from Create test RDF dataset for evaluating alternatives to Blazegraph to Create RDF dataset for testing alternatives to Blazegraph.
Feb 8 2022, 2:49 PM · Wikidata
So9q added a comment to T301243: Wikibase Bug: Unclear error message "save has failed".

For wbstack i tracked this for https://github.com/wbstack/private/issues/4
Quoting from there...

Wiki: https://kbtestwikibase.wiki.opencura.com/wiki/Main_Page Reported by: Olaf

Adam I set up a Wikibase via wbstack.com. First I created a property P1 : Same as, of Type URL. Then I created P2 Instance of. Then I want to add a P1 statement to P2 (same as P31 on Wikidata), and give it the value 'https://www.wikidata.org/entity/P31, and try to save that URL, it gives me an error message Failed to Save. Any ideas what might be wrong?

Trying to add a value to a P1 statement on https://kbtestwikibase.wiki.opencura.com/wiki/Property:P2 of https://www.wikidata.org/entity/P31

API Response:

{
  "error": {
    "code": "failed-save",
    "info": "The save has failed.",
    "messages": [
      {
        "name": "wikibase-api-failed-save",
        "parameters": [],
        "html": {
          "*": "The save has failed."
        }
      }
    ],
    "*": "See https://kbtestwikibase.wiki.opencura.com/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes."
  }
}

image.png (859×1 px, 318 KB)

Looking at the code paths it is probably coming from

https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/Api/EntitySavingHelper.php#L419

then hitting

https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/Api/EntitySavingHelper.php#L445

which ends up with the user.

So we are hitting EditEntity::ANY_ERROR and EditEntity is what is erroring.

Looking at EditEntity, that means it could be any one of these types of error (as these are the only time an error is set)

image.png (301×457 px, 23 KB)

Generally in the status returned the value of errorFlags seems to be set to something that would help us determine what is happening.

This status does make it into ApiErrorReporter::dieStatus, but it doesnt look like the value itself is returned? Could file an upstream issue for this...

And finally with some debugging...

After the above debugging I can get a log of:

[info] [WBSTACK] Wikibase\Repo\Api\EntitySavingHelper::handleStatus: {"errorFlags":32}

This relates to:

	/**
	 * Indicates that the content triggered an edit filter that uses
	 * the EditFilterMergedContent hook to supervise edits.
	 */
	/* public */ const FILTERED = 32;

FILTERED will be returned as a result of editFilterHookRunner hooks failing

Looking at kbtestwikibase I see that both I when testing and the user editing that reported the issue are not admins.
When I was testing on a second site I was using the default created admin user, and it was working.
On https://addshore-alpha.wiki.opencura.com/wiki/Item:Q1 I created a new user and then encountered the issue.

To investigate:

  • Which extension filter actually caused this?
  • What is the desired default behaviour here for site owners / users?

And then probably do one of:
1 - Upstream error should be better, so that the user knows what is going on
2 - Do not stop new users from adding links?

Feb 8 2022, 2:46 PM · wbstack, Wikibase (3rd party installations)
So9q updated the task description for T301244: Wbstack: URL property does not work.
Feb 8 2022, 1:31 PM · wbstack
So9q created T301244: Wbstack: URL property does not work.
Feb 8 2022, 1:30 PM · wbstack
So9q added a project to T301243: Wikibase Bug: Unclear error message "save has failed": wbstack.
Feb 8 2022, 1:22 PM · wbstack, Wikibase (3rd party installations)
So9q added a comment to T301243: Wikibase Bug: Unclear error message "save has failed".

Terrible error messages like these push the user away. The system is unreliable, saving other statements work, sometimes. A system that cannot explain why it does not work as intended leads to bad UX.

Feb 8 2022, 1:18 PM · wbstack, Wikibase (3rd party installations)
So9q created T301243: Wikibase Bug: Unclear error message "save has failed".
Feb 8 2022, 1:17 PM · wbstack, Wikibase (3rd party installations)
So9q added a comment to T27909: Add a drop-down list for the tags in Special:Newpages, Special:Log and Special:Contributions.

Okay, so here's yet another attempt at this. I tried to re-use as much code as possible from the fancy dropdown in RCFilters, which will hopefully make this easier to review and approve than the past attempts.

Nice, definitely an improvement! This could be one of the earliest completed wishes of this year's survey.

Feb 8 2022, 12:31 PM · User-notice-archive, Community-Wishlist-Survey-2022, MW-1.38-notes (1.38.0-wmf.23; 2022-02-21), Growth-Team-Filtering, Platform Team Workboards (External Code Reviews), Growth-Team, MediaWiki-Change-tagging
So9q added a comment to T290240: Evaluate whether RDF Delta is a good idea to have in the backend.
Feb 8 2022, 10:01 AM · Wikidata, Wikidata-Query-Service
So9q added a comment to T290240: Evaluate whether RDF Delta is a good idea to have in the backend.

We should probably start with a problem we're be trying to solve. What would this be for this one?

Good idea.

So as a data consumer I want to know which triples have been changed between 2 dumps from Wikidata.

As an enterprise company I want to replicate Wikidatas triple store inhouse and therefore consume the RDF Delta to do queries on own infrastructure.

Feb 8 2022, 9:54 AM · Wikidata, Wikidata-Query-Service
So9q added a subtask for T206560: [Epic] Evaluate alternatives to Blazegraph: T301227: Create RDF dataset for testing alternatives to Blazegraph.
Feb 8 2022, 9:50 AM · Wikidata, Epic, Wikidata-Query-Service
So9q added a parent task for T301227: Create RDF dataset for testing alternatives to Blazegraph: T206560: [Epic] Evaluate alternatives to Blazegraph.
Feb 8 2022, 9:50 AM · Wikidata
So9q created T301227: Create RDF dataset for testing alternatives to Blazegraph.
Feb 8 2022, 9:50 AM · Wikidata
So9q updated subscribers of T299460: Evaluate the Apache Jena Framework.

@So9q : How would you like to serve everything from one place? It is normal to have replica of data. One of the big bottlenecks is IO. Or do I understand something wrong?

Feb 8 2022, 9:41 AM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q added a comment to T299460: Evaluate the Apache Jena Framework.

FYI: I added https://www.wikidata.org/wiki/Q110853896 RDF Delta and Andy to Wikidata.

Feb 8 2022, 9:31 AM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q renamed T299460: Evaluate the Apache Jena Framework from Evaluate Apache Jena to Evaluate the Apache Jena Framework.
Feb 8 2022, 9:29 AM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q updated the task description for T301220: Create new lexeme missing on mobile.
Feb 8 2022, 8:09 AM · Wikidata, Wikidata Mobile
So9q created T301220: Create new lexeme missing on mobile.
Feb 8 2022, 8:08 AM · Wikidata, Wikidata Mobile

Jan 29 2022

So9q created T300445: Remove the custom logging levels.
Jan 29 2022, 9:41 PM · Pywikibot
So9q updated subscribers of T222608: Should Wikidata Integrator and Pywikibot merge?.

I recommend using WikibaseIntegrator v0.12 instead(RC1 was recently released). It already supports most if not all of Wikibase and has nice APIs ;-)
See the notebooks here for a demonstration: https://github.com/LeMyst/WikibaseIntegrator/tree/rewrite-wbi/notebooks

Jan 29 2022, 9:38 PM · Pywikibot-Wikidata, Pywikibot, Wikimedia-Hackathon-2019
So9q added a comment to T272088: Logging (pywiki module) always verbose if enabled.

I see. 'pywiki' logger will be initialized to level 11 and there is no easy way to change that. As a work-around you can modify the logger after it was initialized e.g.

import logging
import pywikibot
pywikibot.output('This will initialize the logger')
logger = logging.getLogger('pywiki')
logger.setLevel(logging.WARNING)
pywikibot.output('This message will not be logged anymore')
pywikibot.log('Also verbose lgs are hidden')
pywikibot.warning('Warnings are still logged')
pywikibot.error('Errors are logged too.)
Jan 29 2022, 8:43 PM · Pywikibot
So9q awarded T272088: Logging (pywiki module) always verbose if enabled a Like token.
Jan 29 2022, 8:30 PM · Pywikibot
So9q added a comment to T300432: WDQS does not return all the descriptions.

The problem was in the query, stuffing everything in one optional clause.

Jan 29 2022, 1:38 PM · Wikidata, Wikidata-Query-Service
So9q closed T300432: WDQS does not return all the descriptions as Invalid.
Jan 29 2022, 1:38 PM · Wikidata, Wikidata-Query-Service
So9q updated the task description for T300432: WDQS does not return all the descriptions.
Jan 29 2022, 1:31 PM · Wikidata, Wikidata-Query-Service
So9q renamed T300432: WDQS does not return all the descriptions from WDQS label service does not return all descriptions to WDQS does not return all the descriptions.
Jan 29 2022, 1:28 PM · Wikidata, Wikidata-Query-Service
So9q created T300432: WDQS does not return all the descriptions.
Jan 29 2022, 1:26 PM · Wikidata, Wikidata-Query-Service

Jan 28 2022

So9q added a comment to T289621: Evaluate Halyard as alternative to Blazegraph.

That's a bit disappointing b/c it does look like it can scale and has been run through some paces. https://www.linkedin.com/pulse/halyard-tipstricks-trillion-statements-challenge-adam-sotona/

Jan 28 2022, 7:02 PM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q added a comment to T289621: Evaluate Halyard as alternative to Blazegraph.

That's a bit disappointing b/c it does look like it can scale and has been run through some paces. https://www.linkedin.com/pulse/halyard-tipstricks-trillion-statements-challenge-adam-sotona/

I'm actually trying to get this to compile with the latest versions but a few things have changed since them so it's a bit of a sludge.

Jan 28 2022, 6:54 PM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q updated subscribers of T289621: Evaluate Halyard as alternative to Blazegraph.

Here is their sparql evaluation strategy:

Actual Halyard Evaluation Strategy turns the previous model inside-out. I call it "PUSH Model". The SPARQL query is transformed into a chain (or tree) of pipes (Binding Set Pipe) and then it is asynchronously filled with data. An army of working threads periodically take requests with the highest priorities from the priority queue and perform them (usually by requesting the data from the underlying store and by processing them through the pipes). Each working thread can serve its own synchronous requests to the underlying storage system or process the data through the system almost independently of the others. There are two critical parts of the model implementation to make it really working. One hard part is synchronisation of the joints, where bad synchronisation leads to data corruption. And the second (with the same importance) is perfect balancing of the thread workers jobs. It was critical to design the system to do not let thread workers block each other. When most of the thread workers are blocked, it leads to the performance similar to the previous model. Halyard Strategy handles the worker threads jobs in a priority queue, where the priority is determined from the position in the parsed SPARQL query tree. Pipe iterations and active pumps are another methods to connect Halyard Strategy model with the original RDF4J API (or in some unfinished cases also with Iterations implemented in the original model).

For example your have a SPARQL query containing inner join. The request for data from left part of the join is enqueued with priority N. A worker thread that asynchronously delivers that data to the left pipe of the join also enqueues a request to receive relevant data from the right part of the join (with priority N+1). The higher priority of the right part here is very important to reflect the fact that once you get the right data, you can finish the join procedure and “reduce” the cached load and proceed down the pipes. However (based on the priority queue) the other worker threads can simultaneously prefetch more data for the left part of the join. In ideal situation you can see a continuous CPU load of all thread workers in a connected Java Profiler.

I should mention some numbers here. According to the experiments the Halyard Strategy has been approximately 250 times faster with 50 working threads and a SPARQL query containing 26 various joins. The effectivity of the Halyard Strategy is higher with more joins and unions. However feel free to compare my experimental measurements with your own. Both strategies can be individually selected for each Halyard repository. For an experiment you can set up two repositories (both pointing to the same data) with different SPARQL evaluation strategies.

source: https://www.linkedin.com/pulse/inside-halyard-2-when-one-working-thread-enough-push-versus-sotona

Jan 28 2022, 6:25 PM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q added a comment to T289621: Evaluate Halyard as alternative to Blazegraph.

I researched this solution a little:
https://merck.github.io/Halyard/img/architecture.png

architecture.png (1×2 px, 677 KB)

Jan 28 2022, 6:05 PM · MediaWiki-Stakeholders-Group, Wikidata, Epic, Wikidata-Query-Service
So9q closed T290082: Evaluate Apache HBase and RDF4J as alternative to Blazegraph, a subtask of T206560: [Epic] Evaluate alternatives to Blazegraph, as Declined.
Jan 28 2022, 5:30 PM · Wikidata, Epic, Wikidata-Query-Service
So9q closed T290082: Evaluate Apache HBase and RDF4J as alternative to Blazegraph as Declined.
Jan 28 2022, 5:30 PM · Wikidata, Wikidata-Query-Service
So9q added a comment to T290082: Evaluate Apache HBase and RDF4J as alternative to Blazegraph.

Yes, this issue can be closed unless WMF wanna implement an own solution based on the linked paper.

Jan 28 2022, 5:29 PM · Wikidata, Wikidata-Query-Service
So9q added a comment to T104762: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry but for SPARQL).

With the current SPARQL setup it's easy to share queries either by full url or by short url. I think we can close this one.

I disagree: one important part of this task, saving results, isn’t served at all by this. We want to be able to save query results and share them, and unlike on Quarry, it shouldn’t be possible to change those results later, even for the query author (who, on Quarry, can re-run the query, changing the results without assigning a new ID). Other than when privacy or legal concerns require the results to be deleted, the pages should be immutable.

Jan 28 2022, 10:16 AM · Epic, Quarry, patch-welcome, Wikidata-Query-Service, Discovery-ARCHIVED, Wikidata, VPS-Projects

Jan 25 2022

So9q created T300084: Help missing on filter form.
Jan 25 2022, 11:34 PM · Wikidata

Jan 24 2022

So9q added a comment to T288262: Estimate how many Wikidata items have low/no ORES score.

The analysis is done here (for Q-ids): Wikidata_Item_ORES_Score_Analysis

Jan 24 2022, 9:56 PM · ORES, Machine-Learning-Team, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service