I don’t need this task anymore as I found the real terminal and successfully edited the hidden file as I wanted.

In T330028#8628160, @EpicPupper wrote:

PAWS files are viewable by anyone anyways.

Thanks for working hard on this to get it sorted out and sharing the root cause analysis. :)

I investigated this as I have trouble in Firefox on iOS (iphone 7) editing long lines.
I added a few fixes in my css see https://www.wikidata.org/wiki/User:So9q/vector-2022.css but I suggest switching to textarea and so user can set a height and avoid scroll issues when mobile editing lexemes.

In T288666#7283290, @Isaac wrote:

I just made some changes, and it should now fix the issues you mentioned (also, I just removed the margin-left: 2em as it wasn't valid to apply anyway and had no effect)

Thanks -- looks great! I also was thinking that including all the non-people links in the stats didn't make a whole lot of sense so I added another parameter to the API call all that needs to be there if the non-gendered links are including in the API response. Let me know if you don't like that and I can either switch the API back or you can just add the all to the API response as below:

__________ All data: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)&all
Just gender subset: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)

I also loaded the script into my global javascript to make sure it works on other languages: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/global.js
In general it does! Pretty interesting to see how different languages end up with different distributions for the same article, especially when it comes to what links make it into the lede paragraph. Obviously would benefit from greater localization -- e.g., not always using English labels -- and the table gets switched around a little bit in Arabic or other right-to-left languages, but that feels likely beyond the scope of this initial working prototype :)

To me, this feels pretty complete. Let me know if you had additional thoughts / improvements you wanted to make though. I think just remaining then would be whether you want to present it at the showcase on Tuesday: https://wikimania.wikimedia.org/wiki/2021:Hackathon/Showcase

See discussion here https://github.com/openstreetmap/operations/issues/764#issuecomment-1312407828

In T272088#8389652, @WolfgangFahl wrote:

7.3 seems to work thanks god ...

I never heard about this project. Would it not be possible to move to the kubernetes cluster?

Thanks for taking the time to submit this ticket. I support it.
Cradle is a good start, but it lacks many important features.
For example it lacks the check for duplicates that is present e.g in the Wikidata lexeme forms tool or through reconciliation in openrefine.

Added Wikibase tags which was requested by Lydia.

In T35470#8024864, @Versant.2612 wrote:

I am loading a large number of elements in a local Wikibase installation and I often need to reprocess the dataset.
A batch delete function to mass delete elements inserted by a bot user would be useful.

Any news on this? Is something hindering it from being triaged?

I'm leaning towards forking pywikibot and removing the offending lines in bot.py causing the verbose log of files.

In T272088#7772946, @maynorc wrote:

In T272088#7661705, @So9q wrote:

In T272088#6751351, @Xqt wrote:

I see. 'pywiki' logger will be initialized to level 11 and there is no easy way to change that. As a work-around you can modify the logger after it was initialized e.g.

import logging
import pywikibot
pywikibot.output('This will initialize the logger')
logger = logging.getLogger('pywiki')
logger.setLevel(logging.WARNING)
pywikibot.output('This message will not be logged anymore')
pywikibot.log('Also verbose lgs are hidden')
pywikibot.warning('Warnings are still logged')
pywikibot.error('Errors are logged too.)

This workaround did not work for me :/

I commented out the whole content of bot.py: writelogheader() and that did the trick!

@So9q, starting to work on this item now; by "did not work" do you mean you still see the ~300 lines of verbose log output? What platform are you running on?

It now happens again.

I would love to see this improved. I often get this generic error when an empty string is passed with a string property. I would also really like to know which property was related to the input error so I don't have to scan through tens of properties to find the anomaly manually.

Based on the discussion above I suggest closing this task.

They look good. Thanks. I’m missing Wikibase related infrastructure there though.
The blazegraph issue is the fact that WMDE and the search platform tream are unsure if the backend can handle any more triples without catastrophic failure. But it might not be possible to detect anyway using graphs like these.

@Hannah_Bast informed in the last WDQS scaling meeting that QLever could have 2 indexes to provide near-realtime queries. See https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update

Hangor and Ordia/Lexeme forms has this already. I use those to create lexemes because it is "safer" until this ticket is fixed. Unfortunately neither Hangor nor Lexeme forms support creating phrases or idioms yet.

related to https://phabricator.wikimedia.org/T260687 maybe a duplicate?

Is there a reason this issue has stalled?

In T301243#7693242, @Addshore wrote:

For wbstack i tracked this for https://github.com/wbstack/private/issues/4
Quoting from there...

Wiki: https://kbtestwikibase.wiki.opencura.com/wiki/Main_Page Reported by: Olaf

Adam I set up a Wikibase via wbstack.com. First I created a property P1 : Same as, of Type URL. Then I created P2 Instance of. Then I want to add a P1 statement to P2 (same as P31 on Wikidata), and give it the value 'https://www.wikidata.org/entity/P31, and try to save that URL, it gives me an error message Failed to Save. Any ideas what might be wrong?

Trying to add a value to a P1 statement on https://kbtestwikibase.wiki.opencura.com/wiki/Property:P2 of https://www.wikidata.org/entity/P31

API Response:

{
  "error": {
    "code": "failed-save",
    "info": "The save has failed.",
    "messages": [
      {
        "name": "wikibase-api-failed-save",
        "parameters": [],
        "html": {
          "*": "The save has failed."
        }
      }
    ],
    "*": "See https://kbtestwikibase.wiki.opencura.com/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."
  }
}

Looking at the code paths it is probably coming from

https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/Api/EntitySavingHelper.php#L419

then hitting

https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/Api/EntitySavingHelper.php#L445

which ends up with the user.

So we are hitting EditEntity::ANY_ERROR and EditEntity is what is erroring.

Looking at EditEntity, that means it could be any one of these types of error (as these are the only time an error is set)

Generally in the status returned the value of errorFlags seems to be set to something that would help us determine what is happening.

This status does make it into ApiErrorReporter::dieStatus, but it doesnt look like the value itself is returned? Could file an upstream issue for this...

And finally with some debugging...

After the above debugging I can get a log of:

[info] [WBSTACK] Wikibase\Repo\Api\EntitySavingHelper::handleStatus: {"errorFlags":32}

This relates to:

	/**
	 * Indicates that the content triggered an edit filter that uses
	 * the EditFilterMergedContent hook to supervise edits.
	 */
	/* public */ const FILTERED = 32;

FILTERED will be returned as a result of editFilterHookRunner hooks failing

Looking at kbtestwikibase I see that both I when testing and the user editing that reported the issue are not admins.
When I was testing on a second site I was using the default created admin user, and it was working.
On https://addshore-alpha.wiki.opencura.com/wiki/Item:Q1 I created a new user and then encountered the issue.

To investigate:

• Which extension filter actually caused this?
• What is the desired default behaviour here for site owners / users?

And then probably do one of:
1 - Upstream error should be better, so that the user knows what is going on
2 - Do not stop new users from adding links?

Terrible error messages like these push the user away. The system is unreliable, saving other statements work, sometimes. A system that cannot explain why it does not work as intended leads to bad UX.

In T27909#7692677, @matej_suchanek wrote:

In T27909#7660574, @matmarex wrote:

Okay, so here's yet another attempt at this. I tried to re-use as much code as possible from the fancy dropdown in RCFilters, which will hopefully make this easier to review and approve than the past attempts.

Nice, definitely an improvement! This could be one of the earliest completed wishes of this year's survey.

In T290240#7328112, @So9q wrote:

In T290240#7328110, @Lydia_Pintscher wrote:

We should probably start with a problem we're be trying to solve. What would this be for this one?

Good idea.

So as a data consumer I want to know which triples have been changed between 2 dumps from Wikidata.

As an enterprise company I want to replicate Wikidatas triple store inhouse and therefore consume the RDF Delta to do queries on own infrastructure.

In T299460#7644901, @DD063520 wrote:

@So9q : How would you like to serve everything from one place? It is normal to have replica of data. One of the big bottlenecks is IO. Or do I understand something wrong?

FYI: I added https://www.wikidata.org/wiki/Q110853896 RDF Delta and Andy to Wikidata.

I recommend using WikibaseIntegrator v0.12 instead(RC1 was recently released). It already supports most if not all of Wikibase and has nice APIs ;-)
See the notebooks here for a demonstration: https://github.com/LeMyst/WikibaseIntegrator/tree/rewrite-wbi/notebooks

In T272088#6751351, @Xqt wrote:

I see. 'pywiki' logger will be initialized to level 11 and there is no easy way to change that. As a work-around you can modify the logger after it was initialized e.g.

import logging
import pywikibot
pywikibot.output('This will initialize the logger')
logger = logging.getLogger('pywiki')
logger.setLevel(logging.WARNING)
pywikibot.output('This message will not be logged anymore')
pywikibot.log('Also verbose lgs are hidden')
pywikibot.warning('Warnings are still logged')
pywikibot.error('Errors are logged too.)

The problem was in the query, stuffing everything in one optional clause.

In T289621#7660272, @nguyenm9 wrote:

That's a bit disappointing b/c it does look like it can scale and has been run through some paces. https://www.linkedin.com/pulse/halyard-tipstricks-trillion-statements-challenge-adam-sotona/

In T289621#7660272, @nguyenm9 wrote:

That's a bit disappointing b/c it does look like it can scale and has been run through some paces. https://www.linkedin.com/pulse/halyard-tipstricks-trillion-statements-challenge-adam-sotona/

I'm actually trying to get this to compile with the latest versions but a few things have changed since them so it's a bit of a sludge.

Here is their sparql evaluation strategy:

Actual Halyard Evaluation Strategy turns the previous model inside-out. I call it "PUSH Model". The SPARQL query is transformed into a chain (or tree) of pipes (Binding Set Pipe) and then it is asynchronously filled with data. An army of working threads periodically take requests with the highest priorities from the priority queue and perform them (usually by requesting the data from the underlying store and by processing them through the pipes). Each working thread can serve its own synchronous requests to the underlying storage system or process the data through the system almost independently of the others. There are two critical parts of the model implementation to make it really working. One hard part is synchronisation of the joints, where bad synchronisation leads to data corruption. And the second (with the same importance) is perfect balancing of the thread workers jobs. It was critical to design the system to do not let thread workers block each other. When most of the thread workers are blocked, it leads to the performance similar to the previous model. Halyard Strategy handles the worker threads jobs in a priority queue, where the priority is determined from the position in the parsed SPARQL query tree. Pipe iterations and active pumps are another methods to connect Halyard Strategy model with the original RDF4J API (or in some unfinished cases also with Iterations implemented in the original model).

For example your have a SPARQL query containing inner join. The request for data from left part of the join is enqueued with priority N. A worker thread that asynchronously delivers that data to the left pipe of the join also enqueues a request to receive relevant data from the right part of the join (with priority N+1). The higher priority of the right part here is very important to reflect the fact that once you get the right data, you can finish the join procedure and “reduce” the cached load and proceed down the pipes. However (based on the priority queue) the other worker threads can simultaneously prefetch more data for the left part of the join. In ideal situation you can see a continuous CPU load of all thread workers in a connected Java Profiler.

I should mention some numbers here. According to the experiments the Halyard Strategy has been approximately 250 times faster with 50 working threads and a SPARQL query containing 26 various joins. The effectivity of the Halyard Strategy is higher with more joins and unions. However feel free to compare my experimental measurements with your own. Both strategies can be individually selected for each Halyard repository. For an experiment you can set up two repositories (both pointing to the same data) with different SPARQL evaluation strategies.

source: https://www.linkedin.com/pulse/inside-halyard-2-when-one-working-thread-enough-push-versus-sotona

I researched this solution a little:
https://merck.github.io/Halyard/img/architecture.png

• Here is an overview of the mapping and explanation of the choice of hashing of triples and quads into columns and not storing anything in the value column. https://www.linkedin.com/pulse/inside-halyard-1-triples-keys-columns-values-upside-adam-sotona

Yes, this issue can be closed unless WMF wanna implement an own solution based on the linked paper.

In T104762#3805254, @Lucas_Werkmeister_WMDE wrote:

In T104762#2635939, @Multichill wrote:

With the current SPARQL setup it's easy to share queries either by full url or by short url. I think we can close this one.

I disagree: one important part of this task, saving results, isn’t served at all by this. We want to be able to save query results and share them, and unlike on Quarry, it shouldn’t be possible to change those results later, even for the query author (who, on Quarry, can re-run the query, changing the results without assigning a new ID). Other than when privacy or legal concerns require the results to be deleted, the pages should be immutable.

In T288262#7636689, @AKhatun_WMF wrote:

The analysis is done here (for Q-ids): Wikidata_Item_ORES_Score_Analysis

In T288262#7629267, @Lydia_Pintscher wrote:

Yeah I think the underlying question we came to with this was if it would make sense to consider kicking out the low-quality Items from the Query Service for the disaster planning. The more I think about the less I think we should, because the query service is such an important piece of infrastructure for the workflows to get exactly these low-quality Items improved.

@Lydia_Pintscher can you remember why this was declined back in 2015? What was the state of Jena back then compared to BlazeGraph?

I read the whole thread and just want to point out that Jena supports SPARQL Update also.

It did not. It still got killed and I more or less gave up on running this on k8s. I just tried running it on the bastion instead and got this:

lowered to saving every 1000 lines now. I hope that will solve the issue

still gets killed. hm

still getting killed with 10000 as limit. hm.

A job just got killed again.
This time I was extracting using this exact code: https://github.com/dpriskorn/WikidataMLSuggester/commit/8f411459cbf685852aea9a238719988e5ba0611e

Note this is low priority for me because I found a workaround and simply output a pickle for every x lines. Afterwards I can join them all easily to one big dataframe.

In T299121#7619402, @Majavah wrote:

Duplicate of T299039: All started jobs failed on Kubernetes during 24h with no visible error or output?

During writing of the output the process was killed.
Evidence is here of partial write of the output file:

In T299039#7616070, @aborrero wrote:

could you please paste here the concrete toolforge-jobs command line you are using to create the job?