I like where this is going :)

Closed as invalid aka won't fix. the defaults should suffice for everyone.

I rewrote my tool using React and it now works in Toolforge so I suggest closing this bug.

I looked at the log locally and found this

In T380737#10357599, @rook wrote:

Thank you @Spinster.

@So9q are you able to reproduce?

In T377678#10244982, @Bugreporter wrote:

I will however support introducing RocksDB-like NoSQL database as a secondary storage of current version of Wikidata entities, in order to reduce the load (i.e. number of queries) of main Wikidata database (see T375352#10184517), but such database should be secondary in nature so it does not replace the MediaWiki revision system.

I wonder how "text user" is defined.
Is it terms editing only for WD?
What does it mean for Wikipedia?

this can be closed now. not reproducible after the implementation of wikibase types

related to T376779 and T358431

related to T358431

There are exceptions. See e.g. https://en.wikipedia.org/w/index.php?title=Macaca_majori&diff=prev&oldid=1250246922 where s2cid does not give access to the pdf but instead links to pubmed.

In T375470#10170498, @Ladsgroup wrote:

Turning MW's database to a distributed one is not really technically feasible. It's basically something along the lines of "rewrite mw to golang" in terms of amount of resources needed.

Interestingly enough, this particular database has very recently moved to a proprietary license model, so it is not a viable solution to scaling wikidatawiki.

Thanks for the reply and reaction :)

Evaluation by chatgpt4o:
Prompt:
Compare mariadb master with multiple replica setup and manual failover setup with CockroachDB
Response:
When comparing MariaDB master with multiple replica setup and manual failover to CockroachDB, several factors come into play, including scalability, fault tolerance, failover mechanisms, ease of management, and consistency. Here's a comparison:

Could anyone update this?
Given https://www.wikidata.org/wiki/User:ASarabadani_(WMF)/Growth_of_databases_of_Wikidata has been scaring the community I suggest we make it really clear how urgent or not this really is.

I agree a check should be implemented to avoid duplicates.

Response from MW is here:
{

"error": {
  "code": "wikilambda-zerror",
  "info": "Error of type Z548",
  "title": "Invalid JSON",
  "message": "Invalid JSON",
  "zerror": {
    "Z1K1": "Z5",
    "Z5K1": "Z548",
    "Z5K2": {
      "Z1K1": {
        "Z1K1": "Z7",
        "Z7K1": "Z885",
        "Z885K1": "Z548"
      },
      "K1": "Syntax error",
      "K2": {
        "Z1K1": "Z99",
        "Z99K1": "{\"Z1K1\":\"Z2\",\"Z2K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Z18346\"},\"Z2K2\":{\"Z1K1\":\"Z14\",\"Z14K1\":\"Z18336\",\"Z14K3\":{\"Z1K1\":\"Z16\",\"Z16K1\":\"Z610\",\"Z16K2\":\"def Z18336(Z18336K1):\\n    # strip prefix dash\\n    string = str(Z18336K1)[1:].strip()\\n    if '-' in string and string.startswith('L'):\\n        # cheching form or sense id\\n        pattern = r'(L\\d+-[FS]\\d+)'\\n        import re\\n        if re.search(pattern, string):\\n            return True\\n        return False\\n    else:\\n        if not string.startswith(('L', 'P', 'Q')) or not string[1:].isdigit():\\n            return False\\n        else:\\n            return True\\n\"}},\"Z2K3\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\",{\"Z1K1\":\"Z11\",\"Z11K1\":\"Z1002\",\"Z11K2\":\"is valid Wikidata entity identifier? python\"}]},\"Z2K4\":{\"Z1K1\":\"Z32\",\"Z32K1\":[\"Z31\"]},\"Z2K5\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\"]}}"
      }
    }
  },
  "labelled": {
    "type": "Error",
    "error type": "Invalid JSON",
    "error value": {
      "type": {
        "type": "Function call",
        "function": "Errortype to type",
        "errortype": "Invalid JSON"
      },
      "error message": "Syntax error",
      "input": {
        "type": "Quote",
        "Quote": "{\"Z1K1\":\"Z2\",\"Z2K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Z18346\"},\"Z2K2\":{\"Z1K1\":\"Z14\",\"Z14K1\":\"Z18336\",\"Z14K3\":{\"Z1K1\":\"Z16\",\"Z16K1\":\"Z610\",\"Z16K2\":\"def Z18336(Z18336K1):\\n    # strip prefix dash\\n    string = str(Z18336K1)[1:].strip()\\n    if '-' in string and string.startswith('L'):\\n        # cheching form or sense id\\n        pattern = r'(L\\d+-[FS]\\d+)'\\n        import re\\n        if re.search(pattern, string):\\n            return True\\n        return False\\n    else:\\n        if not string.startswith(('L', 'P', 'Q')) or not string[1:].isdigit():\\n            return False\\n        else:\\n            return True\\n\"}},\"Z2K3\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\",{\"Z1K1\":\"Z11\",\"Z11K1\":\"Z1002\",\"Z11K2\":\"is valid Wikidata entity identifier? python\"}]},\"Z2K4\":{\"Z1K1\":\"Z32\",\"Z32K1\":[\"Z31\"]},\"Z2K5\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\"]}}"
      }
    }
  },
  "*": "See https://www.wikifunctions.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."
},
"servedby": "mw-wikifunctions.eqiad.main-78b5ccbb46-lfcz7"

}

Thanks to @Mahir256 for linking to https://t.me/Wikifunctions/9592 "Right now your shouldn't be able to make any outside requests. If you are, please let us know immediately." from @DVrandecic

This might be worked around by T282926 being done for this particular test case and implementation.

Here is a chat with chatgpt where we try to diagnose and work around the issue:
https://chatgpt.com/share/89369c64-2531-485f-bc39-5cae75160cd8
Chatgpt suggested the following:

P1 -> 1P also doesn't work, see https://www.wikifunctions.org/view/en/Z18328
P1 -> 1P also doesn't work, see https://www.wikifunctions.org/view/en/Z18329

In T291903#7739739, @Hannah_Bast wrote:

To add to this, the two-index approach has another rather beautiful property:

1. It is important to understand that real-time updates have an inherent price. An engine that supports real-time updates can never be as fast as a read-only engine. But with the approach outlined in https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update we kind of get the best of both worlds:

2. Combining the two indexes gives you full SPARQL 1.1 Update capability. There is an unavoidable penalty in runtime, but if the amount of updates is small relative to the size of the data already there (for Wikidata, we are talking millions of updates in a day vs. billions of triples already in the database), the penalty is relatively small.

3. But you can also choose to only ask the large index. Than you get results on a snapshot of the data from a certain (known) date that lies up to 24 hours in the past. But you get the result with maximum speed.

4. Since the approach naturally enables this choice for each individual query, every user can decide on the trade-off for themselves for each query.

Most of SPARQL 1.1 is now supported with few exceptions. QLever is looking more promising by the day. Very nice work!
See https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard for details

I tested on my VPS in OVH and got:

To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=video+game&terms=computer+game&terms=videogames
Select all
Click submit
->

In T356195#9500686, @bd808 wrote:

With the changes from https://github.com/dpriskorn/WikidataTopicCurator/commit/d532547c74c4ca156d712299f580bc72e50f645a now in place on your Toolforge tool I can't use something like https://topic-curator.toolforge.org/articles?qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome to try and help debug the problem as the live code aborts with a deliberate failure.

Big thanks for taking a deeper look into this and for the suggestion to adjust the encoding. I'll close this as resolved for now, while I try diving deeper into the issue and try to make it work.
Locally on my machine everything works fine. I can send batches as large as 5000 items to QS with no problems which translates to 10k qs-lines to edit.

@VIGNERON the entityshape api is a very limited implementation of shex if I understood correctly.