Increase length limit for external identifier, string and URL datatype
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

As an editor I want to be able to enter longer strings in order to be able to enter more data.

Task:

Make sure that the string-limits config var covers all string types that we want to increase the length limit for.
Increase the configured length limit of strings on wikidata* to 1500 (T154660#4735134) using the string-limits repo option (recently introduced).

Acceptance criteria:

string lengths (for monoling text, URL, string, external id (NOT TERMS :))) should be configured to be 1500 on wikidata*
wikidatawiki, testwikidatawiki and beta wikidata all have the same length limits for the values changed

Original task description

We have had several requests that the length limit of the external identifier, string and URL datatype is too low to cover some usecases. https://www.wikidata.org/wiki/Property:P234 is one example. We should increase the limit.

Open question: What should the limit be? What's the technical limit we run into?

Strawman proposal by Lydia: set all to 4000

Some numbers to help with the decision for external identifier:
InChI string lengths on the 92 Mio PubChem compounds:

99% 99.9%  100%
311   676  4502

Details

Subject	Repo	Branch	Lines +/-
Add parameter type assertions to StringLengthValidator	mediawiki/extensions/Wikibase	master	+6 -3
Fix configuration of max string length	mediawiki/extensions/Wikibase	master	+6 -3
Don't set Wikibase StringLengths on Commons	operations/mediawiki-config	master	+4 -2
Read WikibaseStringLimit in Wikibase.php	operations/mediawiki-config	master	+2 -0
Set Wikibase string-limits for wikidata dblist	operations/mediawiki-config	master	+18 -0

Customize query in gerrit

Related Objects

Mentioned In: T259440: Not able to save a string longer than 400 characters
T212936: Wikibase UI should disable publish button in a consistent way based on string-limits for values.
T192136: Supporting PGN chess notation data
Mentioned Here: T198108: Modify character limit for "string" and "monolingual text" on wikibase instances (not Wikidata).
P234 List last update time of deploy branches

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2017, 12:38 PM

Pasleim subscribed.Jan 5 2017, 12:54 PM

Lydia_Pintscher moved this task from incoming to consider for next sprint on the Wikidata board.Jan 5 2017, 12:56 PM

Does this run into some of the same problems as https://meta.wikimedia.org/wiki/Community_Tech/Edit_summary_length_for_non-Latin_languages ?

Scott_WUaS subscribed.Jan 5 2017, 8:59 PM

Thanks Lydia!

For the InChI example above, 4000 should be fine, question is what other properties outside my expertise would need more than 4000 chars.

Best,

Sebastian

The InChI is not the only use case for chemistry, btw. SMILES also runs into the char limit right now for a number of compounds.

We already discussed this somewhere else. (But where?) On the web you will find a lot of Stackoverflow-type questions and answers. The most relevant are:

Some extremely old browsers had an URL limit of 256 characters.
Some old versions of Internet Explorer had a limit of 1024 characters. These versions are not relevant any more.
Modern browsers basically don't have a limit any more (multiple megabytes, if you want).
We still want some limit to block users from overflowing our database with stuff that does not belong there. Typically a "round" binary number like 2048 or 16384 is used.
A relevant number is 768 (this is 3 * 256). This is the maximum length of a field in a MySQL index. This was very relevant when we tried to build a query service based on MySQL indexing, and might still be relevant for the planned "external identifier lookup" as well as third party users. If we still want to give the guarantee that the lookup will work without Cirrus, we should stick to that maximum of 768.

The current limit for the "string" and "external-id" property types is 400, but "url" is 500. A comment quotes http://stackoverflow.com/a/417184 and an absolute upper limit of 2000, which sounds reasonable to me.

Can somebody calculate the percentage for 768, please?

I calculated these numbers above, they are solely valid for the chemical structure property InChI (P234), based on ~68 million InChI values in the largest public chemistry database PubChem (also valid for other chemical structure properties like canonical and isomeric SMILES). For any other data of Wikidata datatype string/text, I cannot provide numbers, as I lack the distribution of string lengths relevant to other data which should be represented as strings in Wikidata. And as you can see from the distribution above, increasing the limit would only influence representation of the top ~1% of total chemical structure data.

That said, for me, increasing from 400 to 768 is not wort the effort. The reason is that larger biomolecules will not fit anyway (these are currently increasing in relevance), no matter if the limit is 400 or 768, and if we cannot cover these in a comprehensive fashion, trying to tackle it at all does not seem worthwile.

-sebastian

I don't get that. o_O Increasing the limit to 768 will cover almost 1% more in addition to the 99% we have now. How are these almost 100% "not worth the effort"?

@thiemowmde based on your numbers, we are talking about a 1% improvement. If we could get to the full 100%, we would no longer have to account for the failure case. But if we still have to account for that case anyway, going from 99% to 99.9% doesn't really gain us much.

Again, how are almost 100% "not much", compared to the 99% we have now? This is a huge difference. It will decrease the error rate by a factor of 10.

I don't believe that increasing the limit to 2000 would make any difference. The next day somebody will come and ask for 2500. And the week after somebody will ask for 4000. And a month later somebody will still hit this limit.

There is no 100%. Every limit is arbitrary. Every limit will block certain use cases – as it should.

All I want is us to be crystal clear about:

Why we have such limits in the first place. I believe we should document the positive effects such limits have. Otherwise it's not worth much and could as well be dropped.
That an increase will make certain use cases much harder. We should document which known use cases are affected by the decision we make here, because we are basically dropping the support for them.

That's what I think we as a team should do. Personally, I still believe that strings longer than 768 characters are not identifiers.

I am not sure how much we should worry about the exact percentages for PubChem; to me, more important is are the percentages of the chemistry we have in Wikidata. These are likely correlated, and since PubChem is a lot bigger puts things in perspective. InChIs are identifiers, but not as we are common too, and I understand the point about indexing and ID length.

Quick question... is the semantic meaning of an 'external identifier' that it must be indexed, all of them?

@thiemowmde agreed, it would bring down the error rate for this specific identifier/string. But currently, that's the only one we have a size distribution for. I think, Lydia intended to solve this issue here for every property of data type string. So, if for technical reasons (MySQL index field length) it should be limited to 768 for now, this would also be fine for chemistry for now, but how about other properties?

The general question is, if it would be useful/doable to alllow e.g. a medium blob for a text field in the first place, or create a new data type which stores text in blobs. Currently, medium blobs are used to store the whole json, if I am correct. Certainly, this creates indexing, access and potential misuse problems. It also prohibits data access via WDQS.

Another point to keep in mind is that SPARQL queries will get problematic if the field length gets too long. I think I hit a limit of ~4K chars regarding max query length on query.wikidata.org. Is there a maximum length which the currently Wikidata design and SPARQL endpoint agree on which, in order to lift it, would require a major redesign of Wikidata technical infrastructure?

Liuxinyu970226 subscribed.Jan 27 2017, 9:51 AM

Lydia_Pintscher triaged this task as Low priority.Jun 11 2017, 4:37 PM

The length limit does also affect the migration of constraints from property talk pages to statements on properties. ISBN-13 has a format expression which is 481 characters long and therefore can't be added to the property page.

eranroz mentioned this in T192136: Supporting PGN chess notation data.Apr 20 2018, 5:55 AM

Cirdan subscribed.May 14 2018, 6:44 AM

• Arbnos subscribed.Jun 7 2018, 11:46 AM

Daniel_Mietchen subscribed.Aug 18 2018, 8:06 PM

The 400 character limit can also be a problem for the titles of old books, maps, etc -- the original titles of these, as given eg in library catalogues, can get pretty long.

Addshore added a project: Wikidata-Campsite.Oct 9 2018, 7:13 AM

Addshore moved this task from Incoming to Needs Work on the Wikidata-Campsite board.

Addshore subscribed.

Wostr subscribed.Oct 9 2018, 5:08 PM

Marsupium subscribed.Oct 10 2018, 1:47 PM

We recently made these options more easily configurable in T198108.
So if we come up with some new values for wikidata.org we can make this change pretty easily.
I'll leave that to @Lydia_Pintscher though.

Ok then let's put it to 1500 as a next step and then reevaluate.

Addshore updated the task description. (Show Details)Nov 13 2018, 11:39 AM

Addshore updated the task description. (Show Details)Nov 13 2018, 2:09 PM

Addshore set the point value for this task to 3.

Addshore moved this task from Ready to estimate to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.

Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

@Tarrow @Lucas_Werkmeister_WMDE this could be a good candidate to try out your deployment access? :)

Addshore renamed this task from increase length limit for external identifier, string and URL datatype to Increase length limit for external identifier, string and URL datatype.Nov 13 2018, 3:08 PM

Addshore moved this task from consider for next sprint to in progress on the Wikidata board.Nov 13 2018, 3:34 PM

Tarrow moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 14 2018, 10:54 AM

Tarrow claimed this task.Nov 14 2018, 10:56 AM

Change 473694 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[operations/mediawiki-config@master] Set Wikibase string-limits

https://gerrit.wikimedia.org/r/473694

gerritbot added a project: Patch-For-Review.Nov 15 2018, 8:43 AM

Addshore updated the task description. (Show Details)Nov 15 2018, 11:15 AM

Change 473716 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[operations/mediawiki-config@master] Read WikibaseStringLimit in Wikibase.php

https://gerrit.wikimedia.org/r/473716

Change 473694 merged by jenkins-bot:
[operations/mediawiki-config@master] Set Wikibase string-limits for wikidata dblist

https://gerrit.wikimedia.org/r/473694

daniel unsubscribed.Nov 15 2018, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-15T12:21:29Z] <tarrow@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:473694]] Set Wikibase string-limits for wikidata dblist T154660 (duration: 00m 54s)

Change 473716 merged by jenkins-bot:
[operations/mediawiki-config@master] Read WikibaseStringLimit in Wikibase.php

https://gerrit.wikimedia.org/r/473716

Mentioned in SAL (#wikimedia-operations) [2018-11-15T12:30:45Z] <tarrow@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:473716]] Read WikibaseStringLimit in Wikibase.php T154660 (duration: 00m 53s)

Addshore moved this task from Doing to Needs Announcement on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 15 2018, 1:03 PM

Change 473815 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[operations/mediawiki-config@master] Don't set Wikibase StringLengths on Commons

https://gerrit.wikimedia.org/r/473815

Change 473815 abandoned by Tarrow:
Don't set Wikibase StringLengths on Commons

Reason:
addshore beat me

https://gerrit.wikimedia.org/r/473815

beta commons was fixed with https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473814/ and https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473816/

@Tarrow Could you give me an update on this? Is it already deployed, what's the final max length? Thanks :)

I believe it’s 1500 characters (example edit), for monolingual text, strings, and URLs.

Um, actually, way longer strings are apparently still permitted…?

We are seemingly not checking the length limit properly. We have to fix this.
https://www.wikidata.org/w/index.php?title=Q4115189&diff=next&oldid=804887286

Lydia_Pintscher raised the priority of this task from Low to High.Dec 10 2018, 11:40 AM

Lydia_Pintscher moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Apparently, the string length validator now looks like this in production:

Wikibase\Repo\Validators\StringLengthValidator::__set_state(array(
  'minLength' => 1,
  'maxLength' => 
  array (
    'length' => 1500,
  ),
  'measure' => 'mb_strlen',
  'errorCodePrefix' => '',
)),

Because maxLength is an array with a length member instead of a plain integer, the comparison $length > $this->maxLength is now never true, which is why strings of any length are accepted. I’ll try to find out where exactly this is happening.

(Aside: Do we have any place where we collect bugs like this one, which would have been prevented if we had full PHP7 support, including primitive type hints? Just to further motivate the PHP7 transition?)

Change 478678 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Fix configuration of max string length

https://gerrit.wikimedia.org/r/478678

Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 10 2018, 2:31 PM

Change 478678 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix configuration of max string length

https://gerrit.wikimedia.org/r/478678

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.8; 2018-12-11).Dec 11 2018, 10:00 AM

Michael moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 11 2018, 11:12 AM

Change 478928 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Add parameter type assertions to StringLengthValidator

https://gerrit.wikimedia.org/r/478928

Change 478928 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add parameter type assertions to StringLengthValidator

https://gerrit.wikimedia.org/r/478928

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.9; 2018-12-18); removed MW-1.33-notes (1.33.0-wmf.8; 2018-12-11).Dec 12 2018, 10:00 AM

Tested on the sandbox on wikidata.org. The expected behavior (error message) happens correctly for properties of datatype string. However, for external IDs and monolingual string, no error message shows up, the publish link is disabled.

Apparently, for external identifiers the “publish” link becomes disabled at a length of 4592 characters, and for monolingual text, at 4566 characters. (Tested with strings of x characters, generated via printf '%*s' 4566 | tr ' ' 'x' | xclip, and using the en language code for monolingual text.) With values between 1501 and ~4500 characters, you still get the server-side error on submission.

I notice that the difference between 4566 and 4592 characters is exactly the same (26 characters) as the length of {"language":"en","value":}, the overhead of monolingual text values compared to plain string values. But that doesn’t explain why this apparent limit on data value length doesn’t seem to affect string-type properties.

Addshore claimed this task.Jan 4 2019, 1:41 PM

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 4 2019, 1:41 PM

I just looked at this and as the limits work and the acceptance criteria have been met I am going to close the ticket.
I'll open another one about the consistency in the UI for these different cases.

Addshore mentioned this in T212936: Wikibase UI should disable publish button in a consistent way based on string-limits for values..Jan 4 2019, 2:17 PM

Liuxinyu970226 unsubscribed.Jan 16 2019, 1:32 AM

Peter_James mentioned this in T259440: Not able to save a string longer than 400 characters.Aug 2 2020, 8:42 PM

Increase length limit for external identifier, string and URL datatypeClosed, ResolvedPublic3 Estimated Story PointsActions

Description

Details

Related Objects

Event Timeline

Increase length limit for external identifier, string and URL datatype
Closed, ResolvedPublic3 Estimated Story Points
Actions