After I did the rollback to 1.33.0-wmf.18 we had the same exception for wmf.18 albeit only 315 occurences versus the 2400+ we had.

Wikimedia\Assert\PostconditionException from line 140 of /srv/mediawiki/php-1.33.0-wmf.18/vendor/wikimedia/assert/src/Assert.php: Postcondition failed: Namespace for entity type property must be defined!

Gotta check with WMDE / Wikidata people during European day time.

Change 493319 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/mediawiki-config@master] group1 (except commonswiki) to 1.33.0-wmf.19

https://gerrit.wikimedia.org/r/493319

Change 493319 abandoned by Hashar:
group1 (except commonswiki) to 1.33.0-wmf.19

Reason:
Per internal discussion, it is better to just block the whole train instead of leaving commonswiki behind.

https://gerrit.wikimedia.org/r/493319

Mentioned in SAL (#wikimedia-operations) [2019-02-27T20:50:23Z] <hashar> 1.33.0-wmf.19 not rolled to group1. Pending T217285 (Wikibase raising exception on commonswiki). To be figured out during European day time.

I suspect the issue is in extensions/WikibaseMediaInfo - looking into it.

Change 493327 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikibaseMediaInfo@master] Move up checks to test if we should construct depicts widgets

https://gerrit.wikimedia.org/r/493327

There is some JS in WikibaseMediaInfo that gets executed on every file page, even though it shouldn't yet.
It performs 2 action=wbformatvalue calls (with invalid arguments, because they haven't properly been configured yet - likely why the Wikibase API throws an exception)
I believe above patch should fix that issue.

Change 493327 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Move up checks to test if we should construct depicts widgets

https://gerrit.wikimedia.org/r/493327

It's not just wbformatevalue. it's basically every wikibase API module:
https://commons.wikimedia.org/w/api.php?action=wbgetentities&ids=P17
https://commons.wikimedia.org/w/api.php?action=wbgetclaims&entity=Q42&property=P31

Basically federation is completely broken now.

I suspect it's a config issue. Federation works in beta cluster:
https://commons.wikimedia.beta.wmflabs.org/w/api.php?action=wbgetentities&ids=P17
https://commons.wikimedia.beta.wmflabs.org/w/api.php?action=wbformatvalue&format=json&datavalue={%22type%22%3A%22wikibase-entityid%22%2C%22value%22%3A{%22id%22%3A%22P18%22}}&options={%22lang%22%3A%22en%22}&generate=text%2Fhtml

Change 493339 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add federation configs for commonswiki

https://gerrit.wikimedia.org/r/493339

This is result of the entity namespace lookup for commons in beta:

$namespaceLookup = \Wikibase\Repo\WikibaseRepo::getDefaultInstance()->getEntityNamespaceLookup();

> var_dump( $namespaceLookup );
object(Wikibase\Lib\Store\EntityNamespaceLookup)#580 (2) {
  ["entityNamespaces":"Wikibase\Lib\Store\EntityNamespaceLookup":private]=>
  array(3) {
    ["mediainfo"]=>
    int(6)
    ["item"]=>
    int(0)
    ["property"]=>
    int(120)
  }
  ["entitySlots":"Wikibase\Lib\Store\EntityNamespaceLookup":private]=>
  array(3) {
    ["mediainfo"]=>
    string(9) "mediainfo"
    ["item"]=>
    string(4) "main"
    ["property"]=>
    string(4) "main"
  }
}

This is the same for prod:

> $namespaceLookup = \Wikibase\Repo\WikibaseRepo::getDefaultInstance()->getEntityNamespaceLookup();

> var_dump( $namespaceLookup );
object(Wikibase\Lib\Store\EntityNamespaceLookup)#592 (2) {
  ["entityNamespaces":"Wikibase\Lib\Store\EntityNamespaceLookup":private]=>
  array(1) {
    ["mediainfo"]=>
    int(6)
  }
  ["entitySlots":"Wikibase\Lib\Store\EntityNamespaceLookup":private]=>
  array(1) {
    ["mediainfo"]=>
    string(9) "mediainfo"
  }
}

So my patch should fix the issue at hand.

Federation is not and should not currently be turned on on commonswiki.
It looks like the thing that caused the logspam was indeed media info UI code calling the API, when it should not have been calling it.
However the underlying issue is in Wikibase (and only shows itself on commons now as this is the first time we have not had all entities enabled)
When calls are made to the API for entity types that are not enabled it should fail gracefully, not expose an exception to the user.

In T217285#4990679, @Addshore wrote:

Federation is not and should not currently be turned on on commonswiki.
It looks like the thing that caused the logspam was indeed media info UI code calling the API, when it should not have been calling it.
However the underlying issue is in Wikibase (and only shows itself on commons now as this is the first time we have not had all entities enabled)
When calls are made to the API for entity types that are not enabled it should fail gracefully, not expose an exception to the user.

Thank you for telling me. I wasn't aware of that. Sorry for the extra work.
That brings me to two questions:

• All of Wikibase APIs are exposed here. So It's not just a frontend issue. Basically we need to either return error gracefully or don't expose these APIs at all.
• In SoS notes I read multimedia team is about to enable "depicts" statements in commons. Is it possible to do this without federation? It got me confused :(

Change 493339 abandoned by Ladsgroup:
Add federation configs for commonswiki

Reason:
https://phabricator.wikimedia.org/T217285#4990679

https://gerrit.wikimedia.org/r/493339

Change 493402 had a related patch set uploaded (by Hashar; owner: Matthias Mullie):
[mediawiki/extensions/WikibaseMediaInfo@wmf/1.33.0-wmf.19] Move up checks to test if we should construct depicts widgets

https://gerrit.wikimedia.org/r/493402

In T217285#4990679, @Addshore wrote:

Federation is not and should not currently be turned on on commonswiki.
It looks like the thing that caused the logspam was indeed media info UI code calling the API, when it should not have been calling it.
However the underlying issue is in Wikibase (and only shows itself on commons now as this is the first time we have not had all entities enabled)
When calls are made to the API for entity types that are not enabled it should fail gracefully, not expose an exception to the user.

If I understand it properly, the issue pre date 1.33.0-wmf.19 but surfaced due to media info starting to hit the entry point?

Can we possibly get a patch that fails gracefully and avoid the exception / log spam?

@matthiasmullie as I got your WikibaseMediaInfo patch would stop generating the API requests. I cherry picked it at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseMediaInfo/+/493402/

Assuming WikibaseMediaInfo was the main source of issues, I am willing to promote group1 to 1.33.0-wmf.19 during the European time window (14:00UTC / 45 minutes from now). But I guess we also need a patch to get rid of the exception and convert it to a "nice" api error instead.

Change 493402 merged by Hashar:
[mediawiki/extensions/WikibaseMediaInfo@wmf/1.33.0-wmf.19] Move up checks to test if we should construct depicts widgets

https://gerrit.wikimedia.org/r/493402

Mentioned in SAL (#wikimedia-operations) [2019-02-28T14:28:22Z] <hashar@deploy1001> Synchronized php-1.33.0-wmf.19/extensions/WikibaseMediaInfo: Move up checks to test if we should construct depicts widgets - T217285 (duration: 00m 58s)

I have promoted group1 to 1.33.0-wmf.19 and it is apparently no more occuring. Thank you for the quick fixes.

Maybe we need another task to follow up on not throwing an exception?

This indeed sounds like a good idea @hashar. Filed T217335.

Should the priority of thid be lowered given the error seem to be gone?

Now Resolved?

In T217285#5001519, @Jdforrester-WMF wrote:

Now Resolved?

Not yet

No longer seen in production as of April 18.

@alaa_wmde Unsure what was deployed that week, don't see anything tagged on this task. Perhaps the error message was changed. Can you check if the underlying issue is also resolved?