In T375470#10170498, @Ladsgroup wrote:

Turning MW's database to a distributed one is not really technically feasible. It's basically something along the lines of "rewrite mw to golang" in terms of amount of resources needed.

Interestingly enough, this particular database has very recently moved to a proprietary license model, so it is not a viable solution to scaling wikidatawiki.

Thanks for the reply and reaction :)

Evaluation by chatgpt4o:
Prompt:
Response:
When comparing MariaDB master with multiple replica setup and manual failover to CockroachDB, several factors come into play, including scalability, fault tolerance, failover mechanisms, ease of management, and consistency. Here's a comparison:

Could anyone update this?
Given https://www.wikidata.org/wiki/User:ASarabadani_(WMF)/Growth_of_databases_of_Wikidata has been scaring the community I suggest we make it really clear how urgent or not this really is.

I agree a check should be implemented to avoid duplicates.

Response from MW is here:
{

"error": {
  "code": "wikilambda-zerror",
  "info": "Error of type Z548",
  "title": "Invalid JSON",
  "message": "Invalid JSON",
  "zerror": {
    "Z1K1": "Z5",
    "Z5K1": "Z548",
    "Z5K2": {
      "Z1K1": {
        "Z1K1": "Z7",
        "Z7K1": "Z885",
        "Z885K1": "Z548"
      },
      "K1": "Syntax error",
      "K2": {
        "Z1K1": "Z99",
        "Z99K1": "{\"Z1K1\":\"Z2\",\"Z2K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Z18346\"},\"Z2K2\":{\"Z1K1\":\"Z14\",\"Z14K1\":\"Z18336\",\"Z14K3\":{\"Z1K1\":\"Z16\",\"Z16K1\":\"Z610\",\"Z16K2\":\"def Z18336(Z18336K1):\\n    # strip prefix dash\\n    string = str(Z18336K1)[1:].strip()\\n    if '-' in string and string.startswith('L'):\\n        # cheching form or sense id\\n        pattern = r'(L\\d+-[FS]\\d+)'\\n        import re\\n        if re.search(pattern, string):\\n            return True\\n        return False\\n    else:\\n        if not string.startswith(('L', 'P', 'Q')) or not string[1:].isdigit():\\n            return False\\n        else:\\n            return True\\n\"}},\"Z2K3\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\",{\"Z1K1\":\"Z11\",\"Z11K1\":\"Z1002\",\"Z11K2\":\"is valid Wikidata entity identifier? python\"}]},\"Z2K4\":{\"Z1K1\":\"Z32\",\"Z32K1\":[\"Z31\"]},\"Z2K5\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\"]}}"
      }
    }
  },
  "labelled": {
    "type": "Error",
    "error type": "Invalid JSON",
    "error value": {
      "type": {
        "type": "Function call",
        "function": "Errortype to type",
        "errortype": "Invalid JSON"
      },
      "error message": "Syntax error",
      "input": {
        "type": "Quote",
        "Quote": "{\"Z1K1\":\"Z2\",\"Z2K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Z18346\"},\"Z2K2\":{\"Z1K1\":\"Z14\",\"Z14K1\":\"Z18336\",\"Z14K3\":{\"Z1K1\":\"Z16\",\"Z16K1\":\"Z610\",\"Z16K2\":\"def Z18336(Z18336K1):\\n    # strip prefix dash\\n    string = str(Z18336K1)[1:].strip()\\n    if '-' in string and string.startswith('L'):\\n        # cheching form or sense id\\n        pattern = r'(L\\d+-[FS]\\d+)'\\n        import re\\n        if re.search(pattern, string):\\n            return True\\n        return False\\n    else:\\n        if not string.startswith(('L', 'P', 'Q')) or not string[1:].isdigit():\\n            return False\\n        else:\\n            return True\\n\"}},\"Z2K3\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\",{\"Z1K1\":\"Z11\",\"Z11K1\":\"Z1002\",\"Z11K2\":\"is valid Wikidata entity identifier? python\"}]},\"Z2K4\":{\"Z1K1\":\"Z32\",\"Z32K1\":[\"Z31\"]},\"Z2K5\":{\"Z1K1\":\"Z12\",\"Z12K1\":[\"Z11\"]}}"
      }
    }
  },
  "*": "See https://www.wikifunctions.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."
},
"servedby": "mw-wikifunctions.eqiad.main-78b5ccbb46-lfcz7"

}

Thanks to @Mahir256 for linking to https://t.me/Wikifunctions/9592 "Right now your shouldn't be able to make any outside requests. If you are, please let us know immediately." from @DVrandecic

This might be worked around by T282926 being done for this particular test case and implementation.

Here is a chat with chatgpt where we try to diagnose and work around the issue:
https://chatgpt.com/share/89369c64-2531-485f-bc39-5cae75160cd8
Chatgpt suggested the following:

P1 -> 1P also doesn't work, see https://www.wikifunctions.org/view/en/Z18328
P1 -> 1P also doesn't work, see https://www.wikifunctions.org/view/en/Z18329

In T291903#7739739, @Hannah_Bast wrote:

To add to this, the two-index approach has another rather beautiful property:

1. It is important to understand that real-time updates have an inherent price. An engine that supports real-time updates can never be as fast as a read-only engine. But with the approach outlined in https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update we kind of get the best of both worlds:

2. Combining the two indexes gives you full SPARQL 1.1 Update capability. There is an unavoidable penalty in runtime, but if the amount of updates is small relative to the size of the data already there (for Wikidata, we are talking millions of updates in a day vs. billions of triples already in the database), the penalty is relatively small.

3. But you can also choose to only ask the large index. Than you get results on a snapshot of the data from a certain (known) date that lies up to 24 hours in the past. But you get the result with maximum speed.

4. Since the approach naturally enables this choice for each individual query, every user can decide on the trade-off for themselves for each query.

Most of SPARQL 1.1 is now supported with few exceptions. QLever is looking more promising by the day. Very nice work!
See https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard for details

I tested on my VPS in OVH and got:

To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=video+game&terms=computer+game&terms=videogames
Select all
Click submit
->

In T356195#9500686, @bd808 wrote:

With the changes from https://github.com/dpriskorn/WikidataTopicCurator/commit/d532547c74c4ca156d712299f580bc72e50f645a now in place on your Toolforge tool I can't use something like https://topic-curator.toolforge.org/articles?qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome to try and help debug the problem as the live code aborts with a deliberate failure.

Big thanks for taking a deeper look into this and for the suggestion to adjust the encoding. I'll close this as resolved for now, while I try diving deeper into the issue and try to make it work.
Locally on my machine everything works fine. I can send batches as large as 5000 items to QS with no problems which translates to 10k qs-lines to edit.

@VIGNERON the entityshape api is a very limited implementation of shex if I understood correctly.

I also would like this. I recently saved a broken schema and had issues during validation.

I don’t need this task anymore as I found the real terminal and successfully edited the hidden file as I wanted.

In T330028#8628160, @EpicPupper wrote:

PAWS files are viewable by anyone anyways.

Thanks for working hard on this to get it sorted out and sharing the root cause analysis. :)

I investigated this as I have trouble in Firefox on iOS (iphone 7) editing long lines.
I added a few fixes in my css see https://www.wikidata.org/wiki/User:So9q/vector-2022.css but I suggest switching to textarea and so user can set a height and avoid scroll issues when mobile editing lexemes.

In T288666#7283290, @Isaac wrote:

I just made some changes, and it should now fix the issues you mentioned (also, I just removed the margin-left: 2em as it wasn't valid to apply anyway and had no effect)

Thanks -- looks great! I also was thinking that including all the non-people links in the stats didn't make a whole lot of sense so I added another parameter to the API call all that needs to be there if the non-gendered links are including in the API response. Let me know if you don't like that and I can either switch the API back or you can just add the all to the API response as below:

__________ All data: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)&all
Just gender subset: https://article-gender-data.wmcloud.org/api/v1/details?lang=en&title=Pose_(TV_series)

I also loaded the script into my global javascript to make sure it works on other languages: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/global.js
In general it does! Pretty interesting to see how different languages end up with different distributions for the same article, especially when it comes to what links make it into the lede paragraph. Obviously would benefit from greater localization -- e.g., not always using English labels -- and the table gets switched around a little bit in Arabic or other right-to-left languages, but that feels likely beyond the scope of this initial working prototype :)

To me, this feels pretty complete. Let me know if you had additional thoughts / improvements you wanted to make though. I think just remaining then would be whether you want to present it at the showcase on Tuesday: https://wikimania.wikimedia.org/wiki/2021:Hackathon/Showcase

See discussion here https://github.com/openstreetmap/operations/issues/764#issuecomment-1312407828

In T272088#8389652, @WolfgangFahl wrote:

7.3 seems to work thanks god ...

I never heard about this project. Would it not be possible to move to the kubernetes cluster?

Thanks for taking the time to submit this ticket. I support it.
Cradle is a good start, but it lacks many important features.
For example it lacks the check for duplicates that is present e.g in the Wikidata lexeme forms tool or through reconciliation in openrefine.

Added Wikibase tags which was requested by Lydia.

In T35470#8024864, @Versant.2612 wrote:

I am loading a large number of elements in a local Wikibase installation and I often need to reprocess the dataset.
A batch delete function to mass delete elements inserted by a bot user would be useful.

Any news on this? Is something hindering it from being triaged?

I'm leaning towards forking pywikibot and removing the offending lines in bot.py causing the verbose log of files.

In T272088#7772946, @maynorc wrote:

In T272088#7661705, @So9q wrote:

In T272088#6751351, @Xqt wrote:

I see. 'pywiki' logger will be initialized to level 11 and there is no easy way to change that. As a work-around you can modify the logger after it was initialized e.g.

import logging
import pywikibot
pywikibot.output('This will initialize the logger')
logger = logging.getLogger('pywiki')
logger.setLevel(logging.WARNING)
pywikibot.output('This message will not be logged anymore')
pywikibot.log('Also verbose lgs are hidden')
pywikibot.warning('Warnings are still logged')
pywikibot.error('Errors are logged too.)

This workaround did not work for me :/

I commented out the whole content of bot.py: writelogheader() and that did the trick!

@So9q, starting to work on this item now; by "did not work" do you mean you still see the ~300 lines of verbose log output? What platform are you running on?

It now happens again.