Page MenuHomePhabricator

Create an OpenSearch dashboard for Gerrit sshd logs
Open, MediumPublic

Description

The Gerrit ssh daemon logs to /var/log/gerrit/sshd_log which is ingested by our logging pipeline:

rsyslog::input::file { 'gerrit-json':
     path => '/var/log/gerrit/*_log.json',
}

Our Gerrit dashboard filters log.level: NOTICE by default and the ssh logs are thus hidden.

The Gerrit Events List panel lists the fields host.name, log.level, log.logger and message. The log.level seems to be set to NOTICE by our pipeline since the source has none and the log.logger is empty. We thus need a new dashboard tailored for sshd logs.

The log format is described at https://gerrit.wikimedia.org/r/Documentation/logs.html#_sshd_log , there is a common format that looks like:

{
  "timestamp": "2026-02-18T00:00:08.658Z",
  "session": "9fbadf8f",
  "thread": "sshd-SshDaemon[1a5247c8](port=22)-nio2-thread-11",
  "user": "jenkins-bot",
  "account_id": "a/75",
  "message": "LOGOUT"
}

Which is extended for git operations over ssh eg:

{
  "timestamp": "2026-02-18T00:00:08.362Z",
  "session": "e9f1e927",
  "thread": "SSH git-upload-pack /operations/dns (ncmonitor)",
  "user": "ncmonitor",
  "account_id": "a/11663",
  "message": "git-upload-pack./operations/dns",
  "wait_time": "2ms",
  "exec_time": "209ms",
  "total_cpu": "50ms",
  "user_cpu": "50ms",
  "memory": "12022736",
  "status": "0",
  "agent": "git/2.39.5",
  "time_negotiating": "0ms",
  "time_search_reuse": "1ms",
  "time_search_sizes": "0ms",
  "time_counting": "0ms",
  "time_compressing": "17ms",
  "time_writing": "10ms",
  "time_total": "28ms",
  "bitmap_index_misses": "-1",
  "deltas_total": "26",
  "objects_total": "88",
  "bytes_total": "95183"
}

Thus I guess we can use a panel for the generic messages and another one for the git operations.

Possibly our logging pipeline could slightly enhance the messages maybe by setting service.type: gerrit-sshd instead of gerrit or maybe set log.logger to something useful and vary log.level instead of always using NOTICE.

Event Timeline

self note: I get the disconection for a given user with Lucene search query:

message:gerrit.stream-events AND user.name:jenkins-bot AND process.thread.name:SshCommandDestroy*
LSobanski triaged this task as Medium priority.Feb 23 2026, 4:58 PM
LSobanski moved this task from Incoming to Awaiting Input on the collaboration-services board.