Ideas

1. Distributed tracing
   • Doesn't look like we have an established technology for that
     • Current (open) discussion
       • https://phabricator.wikimedia.org/T264667
     • Previous attempts don't look like they can provide some sort of established tracing tooling

2. Correlating logs
   • Publish start/stop events from echo/push service
     • On echo job submission
       • Logstash event
         • Event type
         • Notification ID
         • Start timestamp
     • On push queue flush
       • Logstash event
         • Event type
         • Notification ID
         • End timestamp
   • Correlate events
     • Logstash scripted field (?) for elapsed time
       • TODO: Need to check if they are allowed
     • Use a visualization directly
       • Group by notification ID
       • Visualize p95/p99/average of elapsed time
   • Too much computation time for something questionable if it's that much needed in such a detail
   • Too many log entries

3. Passing debug metadata
   • On echo job submission
     • Generate and pass an initial timestamp
   • On echo job execution
     • Calculate the current timediff and pass it to the API > request metadata
   • On push notification API request handling
     • Calculate the current timediff and pass it to the notification object
   • On queue flush
     • Expose it as a prometheus metric

It turns out it is getting too complicated to implement this and given that we have a lot of metric orchestration already it doesn't look like its a priority at this moment.