Page MenuHomePhabricator

Content too big! Entity: Q27972199
Closed, ResolvedPublic

Description

Error

Request ID: AWjAypKOnlBds_JjF_kR

message
Content too big! Entity: Q27972199
trace
[{exception_id}] {exception_url}   MWContentSerializationException from line 155 of /srv/mediawiki/php-1.33.0-wmf.16/extensions/Wikibase/lib/includes/Store/EntityContentDataCodec.php: Content too big! Entity: Q27972199

#0 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/Content/EntityContent.php(377): Wikibase\Lib\Store\EntityContentDataCodec->encodeEntity(Wikibase\DataModel\Entity\Item, string)
#1 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/RepoHooks.php(653): Wikibase\EntityContent->getTextForFilters()
#2 /srv/mediawiki/php-1.33.0-wmf.14/includes/Hooks.php(174): Wikibase\RepoHooks::onAbuseFilterContentToString(Wikibase\ItemContent, NULL)
#3 /srv/mediawiki/php-1.33.0-wmf.14/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#4 /srv/mediawiki/php-1.33.0-wmf.14/extensions/AbuseFilter/includes/AbuseFilter.php(3162): Hooks::run(string, array)
#5 /srv/mediawiki/php-1.33.0-wmf.14/extensions/AbuseFilter/includes/AbuseFilterHooks.php(53): AbuseFilter::contentToString(Wikibase\ItemContent)
#6 /srv/mediawiki/php-1.33.0-wmf.14/includes/Hooks.php(174): AbuseFilterHooks::onEditFilterMergedContent(DerivativeContext, Wikibase\ItemContent, Status, string, User, boolean, string)
#7 /srv/mediawiki/php-1.33.0-wmf.14/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#8 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/EditEntity/MediawikiEditFilterHookRunner.php(116): Hooks::run(string, array)
#9 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/EditEntity/StatsdTimeRecordingEditFilterHookRunner.php(44): Wikibase\Repo\EditEntity\MediawikiEditFilterHookRunner->run(Wikibase\DataModel\Entity\Item, User, string)
#10 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/EditEntity/MediawikiEditEntity.php(692): Wikibase\Repo\EditEntity\StatsdTimeRecordingEditFilterHookRunner->run(Wikibase\DataModel\Entity\Item, User, string)
#11 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/EditEntity/StatsdSaveTimeRecordingEditEntity.php(73): Wikibase\Repo\EditEntity\MediawikiEditEntity->attemptSave(Wikibase\DataModel\Entity\Item, string, integer, string, boolean)
#12 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/Api/EntitySavingHelper.php(351): Wikibase\Repo\EditEntity\StatsdSaveTimeRecordingEditEntity->attemptSave(Wikibase\DataModel\Entity\Item, string, integer, string)
#13 /srv/mediawiki/php-1.33.0-wmf.14/extensions/Wikibase/repo/includes/Api/CreateClaim.php(102): Wikibase\Repo\Api\EntitySavingHelper->attemptSaveEntity(Wikibase\DataModel\Entity\Item, string)
#14 /srv/mediawiki/php-1.33.0-wmf.14/includes/api/ApiMain.php(1596): Wikibase\Repo\Api\CreateClaim->execute()
#15 /srv/mediawiki/php-1.33.0-wmf.14/includes/api/ApiMain.php(531): ApiMain->executeAction()
#16 /srv/mediawiki/php-1.33.0-wmf.14/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#17 /srv/mediawiki/php-1.33.0-wmf.14/api.php(87): ApiMain->execute()
#18 /srv/mediawiki/w/api.php(3): include(string)
#19 {main}

Impact

unknown

Notes

Event Timeline

mmodell created this task.Feb 6 2019, 4:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 6 2019, 4:16 AM
Addshore added a subscriber: Lydia_Pintscher.EditedFeb 6 2019, 8:51 AM

@Lydia_Pintscher the only way to fix this is to increase the entity size limit. :)
Wich we can and I think we should do!

As reported by mediawiki the entity is 3,618,510 bytes

Addshore triaged this task as Normal priority.Feb 6 2019, 8:52 AM
Addshore moved this task from Incoming to Needs Work on the Wikidata-Campsite board.
Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.
Daimona added a subscriber: Daimona.EditedFeb 6 2019, 11:01 AM

(For the record, it happens every time the entity is edited, which is roughly 900 times in the last 12 hours.)
I must also say that such a big page is pretty slow to load, so while increasing the size limit would fix this specific problem, it could lead to performance issues.

mmodell added a comment.EditedFeb 6 2019, 8:21 PM

This seems like a error that should be handled gracefully in the UI instead of bubbling up to the fatal monitor. I'd be satisfied if we simply reported this as a DEBUG or INFO level log entry. (or even as WARNING)

Addshore updated the task description. (Show Details)Feb 8 2019, 9:27 AM

Looks like we haven't seen this in fatalmonitor since 2019-02-09T08:45, but it doesn't look like the code has been updated at all. Did something change recently?

This seems like a error that should be handled gracefully in the UI instead of bubbling up to the fatal monitor. I'd be satisfied if we simply reported this as a DEBUG or INFO level log entry. (or even as WARNING)

+1

Having this show up as in ERROR means we see it on the dashboards in logstash that we use to determine if something went wrong when we deploy. This problem doesn't indicate something is wrong with code/data/environment/config, but just an unhandled (but known and understood) data state that could cause spikes in error logs that reduce our confidence in a deployment.

mmodell added a comment.EditedFeb 11 2019, 9:29 PM

+1
Having this show up as in ERROR means we see it on the dashboards in logstash that we use to determine if something went wrong when we deploy. This problem doesn't indicate something is wrong with code/data/environment/config, but just an unhandled (but known and understood) data state that could cause spikes in error logs that reduce our confidence in a deployment.

I'm not really sure what transient condition caused it to spike and then subside, however, I believe it's an indicator of a somewhat problematic trend: I've noticed quite a lot of instances similar to this where api-level or library level code is written to strictly and thoroughly validate data, complete with many specific exceptions that log as ERROR. Meanwhile callers of said code are much less thorough with their error handling code. This results in un-handled errors regularly bubbling up to fatalmonitor. I don't think we can expect a revolutionary improvement in the discipline of developers calling these APIs but I do think that it's fair to ask the developers implementing thorough validation regimens to exercise some discipline in exception level logging.

What might be the best venue to advocate for better error level classification when raising exceptions? It seems fairly obvious to me that this (and similar) validation errors should be DEBUG or INFO level as they are only useful to the consumer of an API and not relevant to the overall health and operation of the software itself.

Thinking about this a bit deeper, what it seems like we need is a way to differentiate between user-input-validation-exceptions and filter them out of deployment dashboards, etc. These are only bugs in the sense that front-end code should have caught the exception. In the case of an API call coming from 3rd party callers there is absolutely nothing we can do to act on the error and it's not at all useful to surface it in places where we regularly monitor for operational or deployment-related issues.

zeljkofilipin raised the priority of this task from Normal to High.Feb 28 2019, 12:25 PM
zeljkofilipin added a subscriber: zeljkofilipin.

This is the most common error right now in both fatal-monitor and mediawiki-errors 😱 even more than T204871

From a quick look at https://www.wikidata.org/wiki/Q27972199 its property Main suject (P921) has a lot of duplicates. An example is conserved Plasmodium protein, unknown function which shows up more than a thousand time.

The page history shows recent statements have been done by User:QuickStatementsBot. Maybe it is broken and inserted a lot of duplicates?

The bot is operated by @Magnus . Can you please check claims maid on Q27972199? Seems like it is full of duplicates ;)

So here is what happens: I create(d) lots of gene/protein items (example) for various species. For many statements, I can create references, as I get them from the upstream source. That paper is one of the often-cited ones, about a determination method.

What I also do is add the gene/protein in question as the "main subject" of the publication item. In most cases that makes sense, but not in that particular case.
I have now limited P921 for all publications to 100 entries, that should prevent this from happening again.

The P921 in the item in question can be removed (no, they are not duplicates, the code checks for those).

Update: Will remove them with QuickStatements now

Removal is running.

This seems to be the only item affected with so many P921.

Magnus closed this task as Resolved.Mar 5 2019, 9:59 AM
Magnus claimed this task.
hashar reopened this task as Open.Mar 5 2019, 10:42 AM

In the server log, I see some error from browsing https://www.wikidata.org/wiki/Q27972199

https://www.wikidata.org/w/api.php?action=wbcheckconstraints&format=json&formatversion=2&uselang=en&id=Q27972199&status=violation%7Cwarning%7Cbad-parameters eventually reaches a timeout: set-time-limit.php: the execution time limit of 60 seconds was exceeded

#0 /srv/mediawiki/php-1.33.0-wmf.19/vendor/wikibase/data-model/src/Statement/Statement.php(248): {closure}(integer)
#1 /srv/mediawiki/php-1.33.0-wmf.19/vendor/wikibase/data-model/src/Statement/StatementList.php(157): Wikibase\DataModel\Statement\Statement->getPropertyId()
#2 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/TypeCheckerHelper.php(247): Wikibase\DataModel\Statement\StatementList->getByPropertyId(Wikibase\DataModel\Entity\PropertyId)
#3 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/TypeCheckerHelper.php(195): WikibaseQuality\ConstraintReport\ConstraintCheck\Helper\TypeCheckerHelper->getStatementsByPropertyIds(Wikibase\DataModel\Statement\StatementList, array)
#4 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/TypeChecker.php(118): WikibaseQuality\ConstraintReport\ConstraintCheck\Helper\TypeCheckerHelper->hasClassInRelation(Wikibase\DataModel\Statement\StatementList, array, array)
#5 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(566): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\TypeChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#6 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(460): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#7 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(374): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForMainSnak(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, NULL, array)
#8 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(346): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, NULL, array)
#9 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(155): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, NULL, array)
#10 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, NULL, array, array)
#11 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(247): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, NULL, array)
#12 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(179): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, NULL, array)
#13 /srv/mediawiki/php-1.33.0-wmf.19/extensions/WikibaseQualityConstraints/src/Api/CheckConstraints.php(179): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, NULL, array)
#14 /srv/mediawiki/php-1.33.0-wmf.19/includes/api/ApiMain.php(1596): WikibaseQuality\ConstraintReport\Api\CheckConstraints->execute()
#15 /srv/mediawiki/php-1.33.0-wmf.19/includes/api/ApiMain.php(531): ApiMain->executeAction()
#16 /srv/mediawiki/php-1.33.0-wmf.19/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#17 /srv/mediawiki/php-1.33.0-wmf.19/api.php(87): ApiMain->execute()
#18 /srv/mediawiki/w/api.php(3): require(string)
#19 {main}

That is from today at 2019-03-05T10:00:30 UTC.

I also noticed one occurence of mediawiki.slow-parse, mentioning it took 35 seconds to parse the page.

We believe to have fixed to change the behaviour in T217741 so that the error is handled in the UI (i.e. exception is caught).
Is the error still occurring in logs?

Looks like this has now not happened since March 1st

addshore@mwlog1001:~$ zgrep -cF "Content too big! Entity: Q" /srv/mw-log/archive/exception.log-20190* | grep -v ":0"
/srv/mw-log/archive/exception.log-20190214.gz:10
/srv/mw-log/archive/exception.log-20190226.gz:36
/srv/mw-log/archive/exception.log-20190228.gz:6590
/srv/mw-log/archive/exception.log-20190301.gz:3772
hashar closed this task as Resolved.May 14 2019, 8:53 PM
hashar assigned this task to noarave.
hashar added a subscriber: noarave.
Restricted Application added a project: User-Noarave. · View Herald TranscriptMay 14 2019, 8:53 PM