Page MenuHomePhabricator

Create a cli tool for reporting on various health and security metrics of a given Wikimedia repository
Closed, ResolvedPublic

Description

Similar to scorecard and similar efforts, this will be a simple cli tool that looks at a handful of metrics and attempts to produce a health score and/or risk rating for a given Wikimedia code repository. This is part of a WMF effort to better measure and understand potentially problematic, production-deployed code. Some initial design details:

  1. The cli tool will support MediaWiki extensions and skins as initial use-cases.
  2. The cli tool will be able to analyze either a local git repo (.git present) or remote hosted under gerrit.wikimedia.org, gitlab.wikimedia.org or github.com/wikimedia.
  3. The cli tool will support as much automation as possible from the discussed methodology/metrics section within the working doc.
  4. The cli tool will be designed with the idea of being able to be run within CI.
  5. The cli tool will be used to generate some initial reports (as test cases) for the Graph, StructuredDiscussions and Kartographer extensions.

Event Timeline

sbassett changed the task status from Open to In Progress.May 26 2023, 9:11 PM
sbassett added projects: Security, user-sbassett.
sbassett moved this task from Incoming to In Progress on the Security-Team board.
sbassett moved this task from Backlog to In Progress on the user-sbassett board.
This comment was removed by sbassett.
sbassett moved this task from In Progress to Our Part Is Done on the Security-Team board.

Going to call this initial effort done for now.

@sbassett is there a public URL where we can review the current output ? I'd love to scan the list and see if it fits my world view on code health.

@sbassett is there a public URL where we can review the current output ? I'd love to scan the list and see if it fits my world view on code health.

There's no public output as it's just a cli with no web frontend or anything like that. There are the initial requested results within the protected matrix sheet, but that's all there is for now. One could theoretically download and run the cli against any Wikimedia codebases they'd like and gather the tabular output that way. IME, the cli was taking around 2 to 3 minutes to run per repo.