Are you still seeing this error? If this was from a log have you got the timestamp? Just trying to see if it was related to the outage last night (https://wikitech.wikimedia.org/wiki/Incident_documentation/20200407-Wikidata%27s_wb_items_per_site_table_dropped)?

Please feel free to not migrate over the indices either :)

Thanks so much for reaching out!

Didn't hear more back about this but the jobs were stopped in 2018

@Volans Thanks for the awesomely quick response! I've added more details of the sort of tasks I'm expecting to undertake

@WMDE-leszek Any chance you can take a look and preemptively give the WMDE 👍 ?

Just investigating this now

Thanks for those thoughts!

I've just merged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/572259 which I think will stem most (maybe all?) of these errors. From my side I'd be happy for this to not block the train in either case but we could also backport this band-aid to reduce the logspam

Yes, I don't think this should be a blocker; clearly I should have done a better job of fixing this symptom earlier rather than going for the cause since it's making so many people worry

It appears that the only two places that this CachingFallbackLabelDescriptionLookup is used are here in the Lua Code and in \Wikibase\Lib\Formatters\ItemPropertyIdHtmlLinkFormatter

As discussed with the Campsite and with @Jakob_WMDE I'm just now investigating the point of this class to try and understand if the caching functionality could be replaced by a CachingPrefetchingTermLookup instead

I think this can be merged into T245295.

I'll be looking at this today

Hope it's ok but I'm taking this over in the attempt to finish what we started rather than picking up new work

We were going to backport this in SWAT but we decided not due to the risk of of some problem. This is due to the fact that during sync each file happens separately and by changing the signature of \Wikibase\DataAccess\SingleEntitySourceServices we'd be sure to create some errors.

Checked on beta and doesn't throw any more, just returns nil nil

Thanks for reporting! I've merged this into a previous report

So, I was a little mislead and thought this was mostly about turning on entity federation.

Stack trace locally looks like:

WikiPageEntityMetaDataLookup.php:206, Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->assertCanHandleEntityId()
WikiPageEntityMetaDataLookup.php:200, Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->assertCanHandleEntityIds()
WikiPageEntityMetaDataLookup.php:86, Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
TypeDispatchingWikiPageEntityMetaDataAccessor.php:88, Wikibase\Lib\Store\Sql\TypeDispatchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
PrefetchingWikiPageEntityMetaDataAccessor.php:267, Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch()
PrefetchingWikiPageEntityMetaDataAccessor.php:189, Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
WikiPageEntityRevisionLookup.php:178, Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getLatestRevisionId()
ByTypeDispatchingEntityRevisionLookup.php:64, Wikibase\DataAccess\ByTypeDispatchingEntityRevisionLookup->getLatestRevisionId()
CacheRetrievingEntityRevisionLookup.php:158, Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getLatestRevisionId()
CachingEntityRevisionLookup.php:126, Wikibase\Lib\Store\CachingEntityRevisionLookup->getLatestRevisionId()
CacheRetrievingEntityRevisionLookup.php:158, Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getLatestRevisionId()
CachingEntityRevisionLookup.php:126, Wikibase\Lib\Store\CachingEntityRevisionLookup->getLatestRevisionId()
RedirectResolvingLatestRevisionLookup.php:36, Wikibase\Lib\Store\RedirectResolvingLatestRevisionLookup->lookupLatestRevisionResolvingRedirect()
CachingFallbackLabelDescriptionLookup.php:113, Wikibase\Lib\Store\CachingFallbackLabelDescriptionLookup->getTerm()
CachingFallbackLabelDescriptionLookup.php:107, Wikibase\Lib\Store\CachingFallbackLabelDescriptionLookup->getLabel()
UsageTrackingLanguageFallbackLabelDescriptionLookup.php:71, Wikibase\Client\Usage\UsageTrackingLanguageFallbackLabelDescriptionLookup->getLabel()
WikibaseLanguageDependentLuaBindings.php:60, Wikibase\Client\DataAccess\Scribunto\WikibaseLanguageDependentLuaBindings->getLabel()
Scribunto_LuaWikibaseLibrary.php:605, Wikibase\Client\DataAccess\Scribunto\Scribunto_LuaWikibaseLibrary->getLabel()
LuaStandaloneInterpreter.php:387, Scribunto_LuaStandaloneInterpreter->callback()
LuaStandaloneInterpreter.php:358, Scribunto_LuaStandaloneInterpreter->handleCall()
LuaStandaloneInterpreter.php:424, Scribunto_LuaStandaloneInterpreter->dispatch()
LuaStandaloneInterpreter.php:265, Scribunto_LuaStandaloneInterpreter->callFunction()
LuaEngine.php:439, Scribunto_LuaStandaloneEngine->runConsole()
ApiScribuntoConsole.php:102, ApiScribuntoConsole->runConsole()
ApiScribuntoConsole.php:68, ApiScribuntoConsole->execute()
ApiMain.php:1590, ApiMain->executeAction()
ApiMain.php:522, ApiMain->executeActionWithErrorHandling()
ApiMain.php:493, ApiMain->execute()
api.php:84, {main}()

In T245396#5912675, @Jdforrester-WMF wrote:

New handful of errors in prod:

class@anonymous /srv/mediawiki/php-1.35.0-wmf.20/extensions/Wikibase/lib/includes/SimpleCacheWithBagOStuff.php0x7f9378c5e02f    wmf.20 e/W/l/i/SimpleCacheWithBagOStuff.php:271  Cache key contains characters that are not allowed: `Q177856_1116973465_⧼lang⧽_label`

Is this caused by running in uselang=qqx, perhaps?

https://gerrit.wikimedia.org/r/c/wikibase/javascript-api/+/573628 adds support for unknown errors that contain no more localisable information (e.g. the only thing we have is the error code). I quickly asked @Lydia_Pintscher about that in the daily and we think that's the best thing to show to our users.

I'm just doing the rebase of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/570996

Just to add to the documentation of this incident: did this only break beta in some subtle way? I saw there was a 5 hour-ish gap between the patch being +2 and us reverting.

It appears (to me) to have been resolved by: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/572921/2

Moving to camp as we closed the hike!

In T244722#5888828, @Addshore wrote:

The issue seems to be somewhere in the update process and query service itself rather than the wikidata edit rate (although of course the edit rate has some effect).
Over the past months even when the wikidata edit rate drops the query service lag can continue to increase.

With a successfully logged in users @Jakob_WMDE and I found that we were intermittently regularly getting CSRF token failures for just minted tokens that should have been valid. Sounds like it could be directly related to this (session storage) too.

So we (@Jakob_WMDE and @Rosalie_WMDE and me) looked at this in some detail. We came to the following conclusions (chip in if I misrepresented anything)

We (the wikidata campsite) will pick this up and have a look this morning.

Github PR is in.

Is this now doing what we want?

I'm also probably adding un thought through ideas to this but...
If TermStores can't read is having them in lib the right place? Wouldn't we be smarter to lock them up in repo?

Lerna looks interesting but as far as I can see we'd need to commit to it for all packages in our "monorepo" (and it's really not clear to me how that would work with submodules like termbox).

Release done; lets just check then npm magic works and we can {{done}} this

release PR is up

as far as I could see it was already on there under daily. The data actually arrives live and is then aggregated into daily chunks for display. I moved it under the 'live' tag and added 'Wikidata ' to the front of the dashboard name for consistency.

I saw this revert patch come up; it's a great idea to clean up if not needed.

I'm very happy to say I agree!

Not sure about if test would have had unmigrated item terms

Current theory is that WRITE_NEW is actually still writing both but writing to the new store first not the old one. This is "good" becuase hopefully we don't have any holes but a little confusing. we're just confirming that's the case