Page MenuHomePhabricator

Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties
Closed, ResolvedPublicBUG REPORT

Description

As a
Wikidata user who is editing a newly created Property,

I want to
be able to add multiple statements (e.g., Wikidata property examples) to the new Property without encountering 500 errors or timeouts.

In order to
improve the experience of creating and editing Properties in Wikidata.

What problem are you trying to get solved?
Multiple editors have experienced frequent 500 errors and timeouts when attempting to add Statements to new Properties. Specifically, after successfully adding the first "Wikidata property example" statement, attempts to add a second statement result in the Property page becoming inaccessible for several hours.

Describe the impact that solving this problem will have on users:
Fixing this issue will prevent delays caused by inaccessible pages.

Describe the implications/cost of not fixing this bug:
If this issue is not resolved, it will slow users' contributions to Wikidata and lead to frustration.

List of steps to reproduce:

  1. Create a new Property.
  2. Add the first Statement, e.g. Wikidata property example (P1855) and save it successfully.
  3. Attempt to add a second Statement.
  4. The property page becomes inaccessible, resulting in a 500 error and timeout. It takes several hours before the page becomes accessible again.

Error Details:

If you report this error to the Wikimedia System Administrators, please include the details below.
Request from <ip redacted> via cp3072 cp3072, Varnish XID 344343058
Upstream caches: cp3072 int
Error: 500, Internal Server Error at Fri, 23 Aug 2024 13:46:36 GMT

Acceptance Criteria:

  • Users should be able to add multiple statements to new Properties without encountering 500 errors or timeouts.
  • The Property pages should remain accessible for further editing after each statement is added.

Previous discussion:
See discussion on Wikidata:Report a technical problem#Issues with adding statements to a new property.

Open Questions:

  • Could this issue be related to recent system outages, and if so, how can we prevent this from happening in the future?
  • Since this issue appears to be reported only for Item-type Properties, could there be specific differences between Item-type Properties and other types, such as External Edentifier Properties or String Properties, that might be contributing to the problem?

Event Timeline

wbgetentity/wbeditentity API also times out.

A workaround is install https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb / https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/ and load (edit) this property using one of debug hosts, where the property page will load normally.

A workaround is install https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb / https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/ and load (edit) this property using one of debug hosts, where the property page will load normally.

The workaround does indeed seem to work to view the page, though edits still seem to fail.

image.png (926×352 px, 34 KB)

Interesting, coming here from the merged in ticket, there are some logs that might be relevant, relating to property lookups or caching.
The logs there also seemed to look like OOMs,
Also seems reproducible on the k8s debug host, so might be a k8s mw issue?

I believe this must have been an infrastructure issue which hasn't occured any more in the last three months.

From logstash, I can only see 11 responses with http status 500 for wikidata Properties ("uri_path": "/wiki/Property:") in the last 90 days, none of which was for a newly created Property.

Additionally, I checked the revision history of the last 10 newly created Wikidata Properties and found that none of them were unavailable for hours, immediately after their creation:

Property IDcreated onedited until# of edits
P1341205.04.2025 09:20:1205.04.2025 09:30:2517
P1341307.04.2025 02:00:0607.04.2025 02:00:4940
P1341408.04.2025 04:01:2908.04.2025 04:14:3221
P1341508.04.2025 04:23:2508.04.2025 04:33:1419
P1341608.04.2025 04:34:3408.04.2025 04:42:1016
P1341708.04.2025 04:45:1108.04.2025 04:49:3512
P1341809.04.2025 09:22:0209.04.2025 09:37:5121
P1341909.04.2025 10:23:0609.04.2025 10:30:4116
P1342009.04.2025 10:36:0209.04.2025 10:43:3316
P1342110.04.2025 04:49:3610.04.2025 05:02:1828

last 10 newly created Wikidata Properties

Note the issue are only reported in item or lexeme typed properties. None of these recent 10 property is item or lexeme type.

The most recent report is March 17: https://www.wikidata.org/wiki/Wikidata:Report_a_technical_problem/Archive/2025/03#c-Trade-20250317020600-items_classified_(P13359)

Bugreporter renamed this task from Frequent 500 Errors and Timeouts When Adding Statements to New Properties to Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties.Apr 11 2025, 2:20 PM

The problem seems to occur specifically, when a newly created Property is used in a stamement on itself, as is usually the case when creating "Wikidata property example" (P1855) statements. The reason for this is probably a loop that was introduced with “data type aware” value deserialization (T359421), which requires a data type lookup for deserializing values.

Change #1138090 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[mediawiki/extensions/Wikibase@master] Avoid self-referencing loop for new properties

https://gerrit.wikimedia.org/r/1138090

Problem is still there - I just created P13478 (first item-valued property after creating a large number of external-id properties) and it froze.

https://www.wikidata.org/wiki/Property:P13478

I'm patiently waiting for this problem to be resolved, but I'll be honest and say that it's getting really annoying as I not only can't finalize the property creation (I have to wait almost a whole day, see just a recent case https://www.wikidata.org/wiki/Property:P13504), but I can't even just edit anymore. My edits are now taking hours to complete too and it's a huge inconvenience to me. Do you know when this problem might be solved? I ask because the report was made since last year, and the problem has not disappeared at all. Perhaps we should temporarily stop creating properties with the item data type, since there is no solution yet? Also, why is there no prior warning anywhere (at least before creating a property) that you will encounter technical issues when creating a property with the item data type? Is it possible to do something like that?

Change #1138090 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Avoid self-referencing loop for new Properties

https://gerrit.wikimedia.org/r/1138090

@Kirilloparma We have created and merged a patch that will hopefully fix the issue. It is due for deployment on Wednesday. Please do leave a comment here if it is solved or not, once a new Item type Property is created.

Until then: as noted above, the problem is not actually caused simply by creating a new (Item or Lexeme type) Property - it only occurs when using that same Property in a statement on itself (e.g. via P1855) shortly after its creation. So if people should be warned, it would have to be about creating statements that use newly created Properties on themselves.

@Silvan_WMDE Thanks for working on this! I would note that the problem does not arise for external-id (or presumably String)-valued properties, only for item-valued (or possibly lexeme- or other entity-valued?) properties. Does the fix you have implemented align with that pattern?

Does the fix you have implemented align with that pattern?

It does, indeed. Only for entity-typed Properties an additional database lookup is needed, which potentially causes an infinite loop. We now have a mechanism to prevent that loop.

@Kirilloparma We have created and merged a patch that will hopefully fix the issue. It is due for deployment on Wednesday. Please do leave a comment here if it is solved or not, once a new Item type Property is created.

@Silvan_WMDE I just created two new properties with the item data type and I intentionally added Wikidata property example (P1855) at the very end of the creation (the same property that contributes to freeze everything for several hours), and it seems that the problem unfortunately persists (see P13518, P13519).

@Silvan_WMDE I just created two new properties with the item data type and I intentionally added Wikidata property example (P1855) at the very end of the creation (the same property that contributes to freeze everything for several hours), and it seems that the problem unfortunately persists (see P13518, P13519).

As @Silvan_WMDE wrote in T374230#10771849, the fix will be applied on Wednesday (tomorrow), between 8:00 - 10:00 UTC with the regular train deployment if all goes well. No changes are to be expected before that.

Hi - has this been done yet? I'm ready to test it on live Wikidata...

Since it's well after 10:00 UTC I gave it a try - problem is still there!!! See https://www.wikidata.org/wiki/Property:P13549

Hmm, it seems to have resolved now. Maybe I'll try another one...

Nope - new property frozen also as soon as I added an example. But the first one did resolve after less than an hour which seems faster than previously.

So https://www.wikidata.org/wiki/Property:P13551 has now been frozen for 8 hours! Is the problem currently resolved by a process that only runs once or twice a day? Anyway the fix that was promised for today does not seem to have been installed, or if it was it didn't fix it!

Ollie.Shotton_WMDE subscribed.

Is the problem currently resolved by a process that only runs once or twice a day?

No, we attempted to solve this problem by detecting the infinite loop that was causing the issue.

I'll move this task to our current sprint board for the team to have another look at, but please be aware that it is a bank holiday in Germany today and most of the team are off until Monday.

I know this is a frustrating situation, but please be patient with us. It is a tricky bug to reproduce and track down! 🐛

It might be that the fix hasn't actually been deployed yet — according to https://versions.toolforge.org/, wikidatawiki is still running wmf.25 (at a guess, probably due to the current Gerrit issue (T393034) that's blocked the deployment train)

True, the deployment had not actually happened when P13551 was created and blocked. For P13552 we do not see any 500 errors in the logs, but it is not quite clear if that's because the fix was working by that time already.

Yes, P13552 did not result in delay, although I did wait some time before adding the examples, which were not done with Quickstatements. If this fix had been deployed when that was done (May 1 later in the day) then maybe this issue is fixed? Unfortunately we don't have any pending property proposals for item-valued or other entity-valued properties, so it may be some time before we can fully confirm it is fixed. But ok with me if this is closed as done.

Closing the ticket as all the work from our end is done. If someone comes across this issue again, feel free to re-open it.

Confirming this works for me now - https://www.wikidata.org/wiki/Property:P13574 was just created with no delays.