Nemo_bis wrote on 2014-06-16 17:43:37 (UTC)

It's the same place as technical documentation.

I think this is the main concern. Currently, WMF teams have little to no incentive to write mediawiki.org documentation (especially Help pages, but even Extension pages). Usually, the community has to produce documentation on its own, often as spin off from pages describing features and so on.
As long as everything is on the wiki, it's easy to split, move, make translatable, refactor, discuss and whatever without any real barrier.

Moreover, volunteer devs are not likely to follow a structured process similar to WMF's and start describing features, ideas, plans and work done in the same way; so there would be two separate worlds for WMF development projects and volunteer development. Even within WMF, employees can and will have "pet projects" or volunteer projects for which they don't use any structure but easily can and are included in monthly reports when significant enough.

Additionally, what would happen when an extension or project is wrapped up? On the wiki it's just an historical page that can be marked as such with one edit but is as open as before to tweaks and discussion, and can be adopted at any time. With a phabricator project what would one do, reset permissions to make it "orphan" and leave all the rest as it is?

A quick note to say that I'm still thinking about this, and I'm now leaning towards moving to Phabricator.

Nemo_bis wrote on 2014-06-16 17:43:37 (UTC)

Currently WMF teams have little to no incentive to write mediawiki.org documentation (especially Help pages, but even Extension pages). Usually, the community has to produce documentation on its own, often as spin off from pages describing features and so on.
As long as everything is on the wiki, it's easy to split, move, make translatable, refactor, discuss and whatever without any real barrier.

I understand your concern. I think feature documentation is separate from project documentation. A lot of the planning and project management is already happening in Bugzilla, Trello, Mingle, and now Phabricator, so having reports there would lower the friction and maybe encourage people to report more often (especially if we can include some automagic reporting that includes a list of tickets / mock-ups etc. for a given period).

That wouldn't prevent us at all from improving how we handle feature documentation. On mediawiki.org, we've had a weird system where documentation was split between extension pages and project pages, so the distinction would be similar here. Also, I believe we're hiring a Technical writer who might be able to lead us to better documentation.

Additionally, what would happen when an extension or project is wrapped up? On the wiki it's just an historical page that can be marked as such with one edit but is as open as before to tweaks and discussion, and can be adopted at any time. With a phabricator project what would one do, reset permissions to make it "orphan" and leave all the rest as it is?

I'm not familiar enough with Phabricator yet to answer this.

Notes from a quick chat with @Qgil on IRC:

<qgil_> Alright, since I'm not the owner of this I can think something like this.
<qgil_> Forget any addition tool. Just Maniphest tasks and a Engineering-Report project plus a $MM-YYYY tag (I want to create these at some point) `
<qgil_> Create a task for the monthly/quarterly/whatever report.
<qgil_> Create subtasks for each section assigned to each project
<qgil_> These subtasks write their report in the description, and they can do things like adding other projects involved, blockers, etc.
<guillom> I was looking at the Blog prototype, and was thinking it might work for this, but your approach is interesting too.
<qgil_> the main task has a summary of the most relevant stuff, and links
<qgil_> these are real tasks, they appear in dashboards, might have owners, will have priorities
<qgil_> when they are done, they are resolved; the slackers (like me) are easy to spot. By everyone.
<qgil_> Once you're there, the easiest thing is to copy the content of the main description to a blog post.

<qgil_> because... look what do I have at the top of my dashboard: https://phabricator.wikimedia.org/maniphest/?statuses=open,stalled&assigned=PHID-USER-lluzkul4z7us4sxkayss#R
<qgil_> there is no way to avoid it (((a task for writing October report)))
<qgil_> so I thought: well, the next step is to write the report right there (or more precisely, in the corresponding subprojects
<qgil_> what's more, a reader could start in the blog post, and following links of links, they could end up in any of the sub-reports, or the tasks linking to it,
<qgil_> and there they would have a form to ask questions and post comments
<qgil_> they could subscribe, watch projects... all connected.

I hope you are brave and go for this option based in Phabricator tasks. The more I let it sink and consolidate in my mind, the more sense it makes.

For what is worth, a few days ago @RobLa-WMF also agreed in a casual conversation that this reporting should be based on an aggregation of tasks. To be more specific, he mentioned this idea, and I told them that I had already proposed it here. :)

@Qgil: I'm planning to meet next week with @Eloquence and @Tbayer to discuss how to handle tech reporting going forward. Should we (you and I, maybe @RobLa) meet beforehand to discuss what you have in mind? It seems clear in your head but it's still fuzzy in mine :)

@gpaumier I'm happy to pitch you the idea in a meeting. :)

I'm also curious about the overlap between Scrum of Scrums & the content of the monthly reports. Typically the weekly Scrum of Scrum updates have a mix of team dependencies and progress reports / communication to other teams. There is also a lot of information in there. The content is not as polished or summarized enough to replace the current reports, but if we looked for ultra-cheap weekly status updates then it might be worth considering reusing some of that information, or tweaking the status update portion of the Scrum of Scrums to facilitate its use for an external report.

I can only speak for myself. This kind of report needs anyway rephrasing/polishing work and this is not something I like to do nor automated. Being near or not of the "raw data" should not help a lot, in my case, most of the raw data are anyway not in Phabricator (I meanly use the Kiwix twitter feed, which already gather the most important news). Mediawiki is a publication platform I'm used to deal with. The current setup is for me OK, the only thing I would change, would be to move from a monthly report to a bi-monthly (only two months) report to reduce my reporting workload.

After meeting with @Tbayer and discussing this with a few more people, here's what I'm thinking:

With the shift to mandatory quarterly reporting, the teams are now devoting resources to summarizing and contextualizing their accomplishments on a regular basis. From this point of view, monthly reports seem like a duplication of effort for limited benefit.

The main difference is essentially the frequency of reporting, and Tech News partly addresses this concern by providing a channel where people can get technical updates on a weekly basis. However, the scope of Tech News is limited to changes that impact Wikimedia editors, so noteworthy work in other areas doesn't get reported there.

In order to optimize this decision under the constraints explained above, here's my proposal:

• We stop carrying the monthly engineering reports on mediawiki.org to avoid duplication of efforts. We focus of quarterly reports for accountability and transparency purposes, and Tech News for timely broadcasting.
• We ask teams to tag noteworthy changes using the #notice tag (and its sub-tags) as described in In T88468#1044279 to track noteworthy changes for different audiences in a timely manner.
• The notice tags also serve as a mechanism for people to get involved.
• We set up a lightweight mechanism allowing teams to post voluntary status updates in Phabricator, through the creation of a #summary tag (see T88470). The tag is added to tasks created solely to contain an update. Larger teams, in particular, may want to use this system to post updates similar to the ones previously posted on mediawiki.org, with links to specific tickets for more details.
• Those summaries are encouraged but voluntary, and the teams are welcome to reuse summaries created in other venues (notes from team meetings, SoS, notes from Stand-ups, weekly emails, etc.). The teams are free to choose the frequency at which they deem it appropriate to add updates.

Does this makes sense?

So would a team create a task with a name like "FY2014-15, Q3", add their team tag and the #summary tag and then just add comments periodically to the task as their project(s) progressed for the reporting period?

In T24#1047982, @bd808 wrote:

So would a team create a task with a name like "FY2014-15, Q3", add their team tag and the #summary tag and then just add comments periodically to the task as their project(s) progressed for the reporting period?

That's an interesting approach. In my mind, each status update would be a new task (and would have summary + the team's name as tags), but if more people prefer your approach, I'm fine with it too.

I love the general framework described by @gpaumier at T24#1045203. I'm sure there are details we still need to sort out (e.g. what the role of project pages should be going forward; how would one find relevant status from those pages), but the idea of rethinking the now largely redundant monthly report gets a +1 from me.

@gpaumier, how do we expect this proposal to work for teams that are not on Phab? Shouldn't we mark this as blocked by completing the team transition to Phab? cc @Jaredzimmerman-WMF @Maryana @JKatzWMF @KLans_WMF

In T24#1048280, @DarTar wrote:

completing the team transition to Phab

Related links: T825, T434

I like this plan! Notice and #summary might need more definition and discussion, but I think this proposal is good to go as it is, and we can fine tune the details as we tag and summarize.

In T24#1048280, @DarTar wrote:

@gpaumier, how do we expect this proposal to work for teams that are not on Phab?

I don't think the migration of all teams to Phabricator is a blocker of this task. If a team has been working in Trello and reporting in mediawiki.org, with this proposal, that team would be working in Trello and reporting in Phabricator. Where some teams would link to Phabricator tasks, that team could link to Trello cards.

In T24#1048012, @RobLa-WMF wrote:

I'm sure there are details we still need to sort out (e.g. what the role of project pages should be going forward; how would one find relevant status from those pages)

At first I was thinking that we'd need to continue to have project pages, but now I'm unsure we need them at all. Phabricator gives us a lot of things for free, and it'll be a lot easier to provide an up-to-date portal of current activities using a custom dashboard or similar. Updating wiki pages manually is unreliable and time-consuming.

This is particularly true if we manage to move the roadmap process to Phabricator as well, and if we revive efforts to maintain glossaries about WMF technical activities (see T90263), possibly as part of wider efforts to maintain a WMF/Community hub on meta (Cc:ing Community-Relations-Support s).

In T24#1049624, @Qgil wrote:

I don't think the migration of all teams to Phabricator is a blocker of this task. If a team has been working in Trello and reporting in mediawiki.org, with this proposal, that team would be working in Trello and reporting in Phabricator. Where some teams would link to Phabricator tasks, that team could link to Trello cards.

This ^

In T24#1048006, @gpaumier wrote:

In my mind, each status update would be a new task (and would have summary + the team's name as tags), but if more people prefer your approach, I'm fine with it too.

I think this part of the proposal is problematic. I think non-technical tasks and technical tasks are both well-suited to Phabricator. However, when someone creates a task, it should be something that genuinely has to get done, and is not yet done.

To choose a random example, "Met with WikiProject ABC maintainers to discuss how Flow could work for them" is a valid update. However, I don't think a task should be created for this after the fact solely to use #summary. I think this will lead to a large number of not-actually-tasks tasks just for the purpose of #summary, taking us too far afield from a task tracking system and bloating the task list.

In many cases, Notice would work well for non-technical (or rollup) tasks. E.g. T78640: Co-op: bot can create a Flow board for each new mentored editor (tracking) is an overall task, with both technical and social elements, which would be well-suited for Notice.

@Mattflaschen, indeed. We could use the task where the work has been done, and then add the Report tag and edit the description accordingly whenever we want to publicize it.

Silly question: how are we supposed to report for the month of February? Old or new system?

In T24#1067061, @Qgil wrote:

Silly question: how are we supposed to report for the month of February? Old or new system?

New system for January and later; I'll follow up as soon as the basics are in place.

There are a few things to figure out (e.g. disagreement between people who want to tag tasks a posteriori, and those who think that that shouldn't happen) but feel free to start tagging retroactively with #report and we can iterate.

See also T94180: Complete the transition to the new roadmap & reporting process.

Epilogue: Half a year later, I think we should record the fact that this approach has failed.

To recap, until December 2014, the following reporting mechanisms were in place for WMF Engineering work:

1. the WMF-wide reports, quarterly (formerly monthly), drawing material from 2.
2. more detailed team-level quarterly reviews
3. the monthly Wikimedia Engineering reports, compiled from status updates on mediawiki.org
4. Tech news (weekly - but as pointed out above, these are not really reports in an accountability sense)
5. (maybe also counts here, per Gabriel above:) Scrum of Scrums, weekly

As of August 2015, four of these are continuing as before (Tech News now facilitated by the User-notice tag).

But 3. was replaced by a Phabricator tag approach that never took off (T94180: Complete the transition to the new roadmap & reporting process culminated in an announcement that presented https://phabricator.wikimedia.org/tag/roadmap/ as a replacement for 3., but it seems never have to be used widely in that reporting sense, and it was abandoned in July).

One possible reason for this failure might be that Phabricator is perhaps not the ideal tool for all and everything (maybe the successful launch in late 2014 facilitated some exaggerated expectations in that respect). But my main takeaway for the future is that a reporting structure does not work without a set schedule and someone who - at least in the beginning and later occasionally - reminds and nudges teams about said schedule. Regardless of which technical platform is being used.

Just putting this here as a weekend thought and summary to inform possible future work in this area, without proposing any changes currently. And it might well be that the Wikimedia movement can live with this reduced level of reporting (at least I haven't seen many complaints). Still, personally I miss the ability to quickly find out what a team has doing in the last month or two via an accessible writeup, without having to pore over detailed Phabricator boards and decrypt lots of task descriptions.

Thanks Tilman for the writeup.

We have got a bunch of process changes in the last six month, and my memory might not serve me well, but... I thought that the lack of monthly / frequent reporting was due to the fact that now we have a strong process around quarterly goals and quarterly reviews for all WMF teams?

I know my team has stopped monthly reports, but it is not because of this task and the use of Phabricator. Nobody is asking us to provide reports other than the quarterly reviews, and for the smaller scale reporting we are just working transparently through our quarterly goals (defined in tasks like T101100: Engineering Community quarterly goals for July-September 2015) and our monthly sprints (i.e. ECT-August-2015).

I'm a pro-reporting, pro-transparency person, and I think that this is a good balance. What would you change?

The quarterly reports, as they get assembled in a WMF-wide quarterly report, provide scorecards with some details on what/how/why thing X went well or bad, which is useful only if someone already knows what the thing is.

There is no prose about what is going on. Also, no document specific to engineering, hence people uninterested in finances and stuff are presumably held back from digging the general thing.

In T24#1566422, @Qgil wrote:

We have got a bunch of process changes in the last six month, and my memory might not serve me well, but... I thought that the lack of monthly / frequent reporting was due to the fact that now we have a strong process around quarterly goals and quarterly reviews for all WMF teams?

This is about reporting, not goalsetting. As for the quarterly reviews, I am not aware of such a relation here, speaking as the current editor/publisher of the quarterly review documentation and the quarterly reports ;) The quarterly reviews existed long before this change; they began in 2012/13 and by 2014 most engineering teams were already doing them. While we have indeed made many process improvements for them this year, taking up additional reporting load from the monthly Engineering reports was definitely not a goal there, in fact they were streamlined rather than expanded in that respect.

I know my team has stopped monthly reports, but it is not because of this task and the use of Phabricator. Nobody is asking us to provide reports other than the quarterly reviews,

Yes, my point was precisely that nobody is asking teams for the kind of reporting that was envisaged here and in T94180, which is a main reason that the plan failed (another is the lack of a set schedule). Teams are busy.

and for the smaller scale reporting we are just working transparently through our quarterly goals (defined in tasks like T101100: Engineering Community quarterly goals for July-September 2015) and our monthly sprints (i.e. ECT-August-2015).

I'm glad WMF tech teams are working transparently on Phabricator, and perhaps your team (as the one that's also reponsible for Phabricator as a platform) is especially good at it. But you are not seriously suggesting that https://phabricator.wikimedia.org/tag/ect-august-2015/ is equally informative as, say
https://www.mediawiki.org/wiki/Wikimedia_Engineering/Report/2014/December#Engineering_Community_Team ? For starters, your link actually doesn't show any done tasks ;) - one has to be fluent enough in Phabricator technicalities to find the right filter settings that includes the done tasks, and savvy enough to know that you have to look for greyed out entries with a stricken number among all the non-greyed ones. Even at that point, clicking through a bunch of tickets that - for most teams - were written with an internal audience in mind, and are not updated with a summary after the task is done, is a far cry from reading an actual report.

In T24#1566491, @Nemo_bis wrote:

The quarterly reports, as they get assembled in a WMF-wide quarterly report, provide scorecards with some details on what/how/why thing X went well or bad, which is useful only if someone already knows what the thing is.

There is no prose about what is going on.

There's actually quite a bit of prose about what went on (in particular in the "successes"/"misses" slides). I agree that they could get a bit better at giving context on what each task is about; we already made some changes to the template last quarter to encourage the inclusion of explanatory links and I might present some ideas for further tweaks of the template for feedback soon.

However that's not the issue here - the quarterly reports serve their purpose fairly well IMHO, but come out up to four months after the fact.

Also, no document specific to engineering, hence people uninterested in finances and stuff are presumably held back from digging the general thing.

I fail to see your point here. If such a psychological problem existed (aversion to "finances and stuff"), it could remedied by clarifying more explicitly that the report covers the entire organization, or linking it more frequently from engineering-related pages. Besides, there is always the more detailed (if less polished) department-specific quarterly review documentation which is also linked from the general report.

Let me quote the decision and comment on it.

In T24#1045203, @gpaumier wrote:

With the shift to mandatory quarterly reporting, the teams are now devoting resources to summarizing and contextualizing their accomplishments on a regular basis. From this point of view, monthly reports seem like a duplication of effort for limited benefit.

This is when I understood that required reported focused on quarterly reviews only. If we are missing more frequent or more detailed reported. If deeper or more frequent reporting is missing, it is not because of Phabricator, but because there is no requirement to do more. Previously, monthly reporting didn't work because we used MediaWiki, but because Engineering management + persistent @gpaumier set a requirement and pursued it on a monthly basis.

The main difference is essentially the frequency of reporting, and Tech News partly addresses this concern by providing a channel where people can get technical updates on a weekly basis. However, the scope of Tech News is limited to changes that impact Wikimedia editors, so noteworthy work in other areas doesn't get reported there.

In order to optimize this decision under the constraints explained above, here's my proposal:

• We stop carrying the monthly engineering reports on mediawiki.org to avoid duplication of efforts. We focus of quarterly reports for accountability and transparency purposes, and Tech News for timely broadcasting.
• We ask teams to tag noteworthy changes using the #notice tag (and its sub-tags) as described in In T88468#1044279 to track noteworthy changes for different audiences in a timely manner.
• The notice tags also serve as a mechanism for people to get involved.
• We set up a lightweight mechanism allowing teams to post voluntary status updates in Phabricator, through the creation of a #summary tag (see T88470). The tag is added to tasks created solely to contain an update. Larger teams, in particular, may want to use this system to post updates similar to the ones previously posted on mediawiki.org, with links to specific tickets for more details.
• Those summaries are encouraged but voluntary, and the teams are welcome to reuse summaries created in other venues (notes from team meetings, SoS, notes from Stand-ups, weekly emails, etc.). The teams are free to choose the frequency at which they deem it appropriate to add updates.

As we have seen, most teams most of the times will not report "voluntarily", which is an endemic problem on reporting in general. If you want to improve the current situation, then I think the only way is to set more requirements.

However... I still don't see the current situation problematic. Team goals + personal goals publicly documented should provide an overview of what really matters, and we should make it easy for users to find these goals and their related reports. Then they can subscribe to the goals they care about. Users willing to go deeper can subscribe to projects and sprints. Making teams write more content in order to generate bigger reports more frequently doesn't guarantee that more users will be better informed about everything at the end.

BUT all this is my personal opinion. This sounds like a good topic to be owned by the Team-Practices group these days.

Some teams are doing monthly reports on their own, but it's hard to find them. I started categorising them at https://www.mediawiki.org/wiki/Category:Wikimedia_engineering_reports