Page MenuHomePhabricator
Create Task
Maniphest T379431

Test v0 metrics on example doc collections
Closed, ResolvedPublic
Actions

Description

Now that I have defined a draft set of v0 metrics definitions for MediaWiki technical documentation (T372102) and identified (and implemented) doc collections to use for testing the metrics, I am doing both manual/unstructured and structured analysis to explore the utility of the draft metrics.

So far, the process looks like this:

  1. Use existing tools to assess tech docs quality within doc collections,
  2. Identify pieces of data that I feel I'm missing when I try to understand the overall health of a collection, and where targeted documentation improvements may be needed,
  3. Manually gather data for the characteristics identified during the research & design phases of the metrics project,
  4. Exploring whether those additional data points would help me answer the questions that were unanswered in 2).

Target date to complete testing: Nov 30. (Actual completion date: Dec 20)
Target date to evaluate the process and publish conclusions: Dec 30. (Delayed by one month, now tracked in T382606)

Related Objects
Search...

Event Timeline

TBurmeister changed the task status from Open to In Progress.Nov 8 2024, 9:43 PM
TBurmeister triaged this task as Medium priority.
TBurmeister created this task.
TBurmeister moved this task from Backlog to In progress on the Tech-Docs-Team board.Nov 8 2024, 9:45 PM
Maryann-Onyinye subscribed.Nov 18 2024, 10:32 AM
TBurmeister added a comment.Nov 22 2024, 11:36 PM

This week I did a deep dive into how we may or may not be able to measure doc readability or succinctness by calculating page length. I used a variety of tools to gather and analyze data for a sample set of 20 docs from my test collections. Everything I did, and my findings, are documented at https://www.mediawiki.org/wiki/User:TBurmeister_(WMF)/Measuring_page_length

TBurmeister closed this task as Resolved.Dec 20 2024, 5:30 PM
TBurmeister mentioned this in T382606: Analyze and publish results of metrics testing.

So far we have:

  • Created a scoring system with weights for doc data elements. This will enable us to use the raw inputs from our data gathering to generate scores for doc characteristics in our v0 metrics categories.
  • Revised and adjusted data elements to enable relatively quick human assessment and boolean input wherever possible; made decisions about data elements to drop due to complexity or limited utility.
  • Created a standardized input form to collect data from human analysis, wiki page metadata, and Xtools.
  • Used that form to manually gather data and create a dataset to use for metrics calculation.

The resulting dataset (currently only available in Google Spreadsheet) includes data across 32 data elements, for 140 tech docs, spanning 5 collections of technical documentation.

Starting in 2025 (January), we will analyse the test data we generated in this task, and evaluate whether the doc characteristics we measured are useful as indicators of the metrics we care about:. That work is tracked in T382606.

TBurmeister updated the task description. (Show Details)Dec 20 2024, 5:52 PM
TBurmeister moved this task from In progress to Done on the Tech-Docs-Team board.Dec 20 2024, 6:05 PM
TBurmeister mentioned this in T388141: Implement prototype for metrics calculations.Mar 6 2025, 3:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits