==== Error ====
* mwversion: 1.42.0-wmf.5
* reqId: 75783b23-579a-46f5-87e6-4716bb837de7
* [[ https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=(time:(from:'2023-11-26T08:24:26.696Z',to:'2023-11-28T08:24:26.696Z'))&_a=(query:(query_string:(query:'reqId:%2275783b23-579a-46f5-87e6-4716bb837de7%22'))) | Find reqId in Logstash ]]
```name=normalized_message,lines=10
[{reqId}] {exception_url} PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat.
```
```name=exception.trace,lines=10
from /srv/mediawiki/php-1.42.0-wmf.5/extensions/WikimediaEvents/includes/BlockUtils.php(134)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.42.0-wmf.5/extensions/WikimediaEvents/includes/BlockUtils.php(134): geoip_country_code_by_name(string)
#2 /srv/mediawiki/php-1.42.0-wmf.5/extensions/WikimediaEvents/includes/BlockUtils.php(101): WikimediaEvents\BlockUtils::getCountryCode()
#3 /srv/mediawiki/php-1.42.0-wmf.5/extensions/WikimediaEvents/includes/EditPage/EditPageHooks.php(45): WikimediaEvents\BlockUtils::logBlockedEditAttempt(MediaWiki\User\User, MediaWiki\Title\Title, string, string)
#4 /srv/mediawiki/php-1.42.0-wmf.5/includes/HookContainer/HookContainer.php(161): WikimediaEvents\EditPage\EditPageHooks->onEditPage__showReadOnlyForm_initial(MediaWiki\EditPage\EditPage, MediaWiki\Output\OutputPage)
#5 /srv/mediawiki/php-1.42.0-wmf.5/includes/HookContainer/HookRunner.php(1593): MediaWiki\HookContainer\HookContainer->run(string, array)
#6 /srv/mediawiki/php-1.42.0-wmf.5/includes/editpage/EditPage.php(1004): MediaWiki\HookContainer\HookRunner->onEditPage__showReadOnlyForm_initial(MediaWiki\EditPage\EditPage, MediaWiki\Output\OutputPage)
#7 /srv/mediawiki/php-1.42.0-wmf.5/includes/editpage/EditPage.php(993): MediaWiki\EditPage\EditPage->displayViewSourcePage(WikitextContent, string)
#8 /srv/mediawiki/php-1.42.0-wmf.5/includes/editpage/EditPage.php(664): MediaWiki\EditPage\EditPage->displayPermissionsError(array)
#9 /srv/mediawiki/php-1.42.0-wmf.5/includes/actions/EditAction.php(66): MediaWiki\EditPage\EditPage->edit()
#10 /srv/mediawiki/php-1.42.0-wmf.5/includes/MediaWiki.php(583): EditAction->show()
#11 /srv/mediawiki/php-1.42.0-wmf.5/includes/MediaWiki.php(363): MediaWiki->performAction(Article, MediaWiki\Title\Title)
#12 /srv/mediawiki/php-1.42.0-wmf.5/includes/MediaWiki.php(960): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.42.0-wmf.5/includes/MediaWiki.php(613): MediaWiki->main()
#14 /srv/mediawiki/php-1.42.0-wmf.5/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.42.0-wmf.5/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}
```
==== Impact ====
That is the first entry which started happening on Monday 11/27 at 8:24:26 UTC. It only affects MediaWiki on Kubernetes as far as I can tell, I guess the image we build is missing the proprietary GeoIP database.
Note how the trace are for `1.42.0-wmf.5` which is the MediaWiki code from two weeks ago. So it is not really related to this week deployment (1.42.0-wmf.7 T350083) but seems to be an issue with how we define the MediaWiki on Kubernetes image.
==== Notes ====
SAL entries around that time for Nov 27:
```
lang=irc
08:43 <taavi@deploy2002> Finished scap: Backport for [[gerrit:966598|Add virtual domain mapping for OATHAuth (T348484)]] (duration: 07m 53s) [production]
08:41 <godog> restart prometheus/k8s-staging in eqiad - T343529 [production]
08:37 <taavi@deploy2002> taavi: Continuing with sync [production]
08:36 <taavi@deploy2002> taavi: Backport for [[gerrit:966598|Add virtual domain mapping for OATHAuth (T348484)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:35 <taavi@deploy2002> Started scap: Backport for [[gerrit:966598|Add virtual domain mapping for OATHAuth (T348484)]] [production]
08:29 <taavi@deploy2002> Finished scap: Backport for [[gerrit:976804|GrowthExperiments: enable AddLink frontend for 16,17th rounds of wikis (T308142 T308143)]] (duration: 19m 54s) [production]
08:23 <taavi@deploy2002> taavi and sgimeno: Continuing with sync [production]
08:18 <taavi@deploy2002> taavi and sgimeno: Backport for [[gerrit:976804|GrowthExperiments: enable AddLink frontend for 16,17th rounds of wikis (T308142 T308143)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:14 <moritzm> installing dpkg bugfix updates on bullseye [production]
08:09 <taavi@deploy2002> Started scap: Backport for [[gerrit:976804|GrowthExperiments: enable AddLink frontend for 16,17th rounds of wikis (T308142 T308143)]]
```
Which is merely the first deployment of the week which caused a new image to be build. The actual root cause would be in the image definition.