Change Details

For HHVM, the hhvm-fatal-error.php entries to syslog were rewritten to appear as MediaWiki. But the PHP7 equivalent appears to not be doing that yet or not anymore. This means they can be included in Logstash dashboards that include a `type:mediawiki` filter (where developers will discover it naturally as part of routine monitoring, and also during train/SWAT deployments). Using a different `type` is not feasible even as workaround because Kibana doesn't support that unless you bypass the UI and use JSON queries directly. If I recall correctly, this rewrite would be in the puppetised logstash/filter file.

For HHVM, the hhvm-fatal-error.php entries to syslog were rewritten to appear as MediaWiki. But the PHP7 equivalent appears to not be doing that yet or not anymore. This means they can be included in Logstash dashboards that include a `type:mediawiki` filter (where developers will discover it naturally as part of routine monitoring, and also during train/SWAT deployments). Using a different `type` is not feasible even as workaround because Kibana doesn't support that unless you bypass the UI and use JSON queries directly. If I recall correctly, this rewrite would be in the puppetised logstash/filter file. ------- >>! In T233828#5534006, @Krinkle wrote: >>>! In T233828#5532983, @Joe wrote: >>[…] After looking further into this, I think we're actually counting these errors multiple times - see for instance https://logstash.wikimedia.org/goto/c92404ce11520d59feb9211471d5bf09 > > One is fpm stderr, which is not useful (no stack traces, no rich MW context fields). This is similar to the hhvm syslog we've had in the past, and our custom syslog rewriter for that. > > The one from fatal-error.php should appear under `type:mediawiki channel:fatal` instead of `type:syslog`. Right now these are not being found by developers and most monitoring queries. Classifying as high priority because this means every Scap deploy and train promotion may be acting on false confidence – given 100% PHP 7 traffic.

For HHVM, the hhvm-fatal-error.php entries to syslog were rewritten to appear as MediaWiki. But the PHP7 equivalent appears to not be doing that yet or not anymore. This means they can be included in Logstash dashboards that include a `type:mediawiki` filter (where developers will discover it naturally as part of routine monitoring, and also during train/SWAT deployments). Using a different `type` is not feasible even as workaround because Kibana doesn't support that unless you bypass the UI and use JSON queries directly. If I recall correctly, this rewrite would be in the puppetised logstash/filter file. ------- >>! In T233828#5534006, @Krinkle wrote: >>>! In T233828#5532983, @Joe wrote: >>[…] After looking further into this, I think we're actually counting these errors multiple times - see for instance https://logstash.wikimedia.org/goto/c92404ce11520d59feb9211471d5bf09 > > One is fpm stderr, which is not useful (no stack traces, no rich MW context fields). This is similar to the hhvm syslog we've had in the past, and our custom syslog rewriter for that. > > The one from fatal-error.php should appear under `type:mediawiki channel:fatal` instead of `type:syslog`. Right now these are not being found by developers and most monitoring queries. Classifying as high priority because this means every Scap deploy and train promotion may be acting on false confidence – given 100% PHP 7 traffic.