Mentioned in SAL (#wikimedia-cloud) [2020-06-03T20:58:41Z] <wm-bot> <root> Hard restart (T254361)

Mentioned in SAL (#wikimedia-cloud) [2020-06-03T21:05:11Z] <wm-bot> <root> Hard restart with --canonical after clearing cached gridengine state in front proxy. (T254361)

The service is up and running now. There was a strange problem with the routing to this tool. No webservice was running on the job grid, but the front proxy thought there was. This misconfiguration in the Redis data used by the initial proxy layer was making it try to route all requests to a host and port on the grid engine job grid that were not there. It was also not obvious at all that this was what was happening. I finally figured it out by stopping the Kubernetes webservice and then noticing that the same 502 error was returned rather than the expected "No webservice" page from the fourohfour tool.

My fix from that point was:

$ webservice --backend=gridengine lighttpd start
:# This overwrote the bad data in the front proxy's Redis lookup
:# I was able to browse to the tool at https://tools.wmflabs.org/most-wanted and see a result
$ webservice stop
:# This stopped the job in a controlled manner which deleted the backend registration from the front proxy
$ webservice --backend=kubernetes --canonical php7.3 start
:# Start the tool back up as a Kubernetes pod using the PHP 7.3 runtime

I also made a $HOME/service.template file for this tool so that webservice start will "just work" and do the same thing as the longer webservice --backend=kubernetes --canonical php7.3 start command.

The index.php script is emitting some loud "Undefined offset:" warnings now. I think this is a PHP runtime version change issue. The complaining code is:

foreach ( $articles AS $s ) {
        $parts = array () ;
        preg_match ( '/^\s*(\d+)\s(.+)$/' , $s , $parts ) ;
        $count = $parts[1] ;
        $article = $parts[2] ;

The assignments of $count and $article are unchecked array dereferences. I made a quick live hack to the code to fix this using the null coalescing operator (??):

foreach ( $articles AS $s ) {
        $parts = array () ;
        preg_match ( '/^\s*(\d+)\s(.+)$/' , $s , $parts ) ;
        $count = $parts[1] ?? 0;
        $article = $parts[2] ?? '';

Thanks @bd808 !