Operations-related subprojects/tags reorganization
Closed, ResolvedPublic

Description

This is a follow-up from a session we held at the 2015 technical operations offsite. It's by no means complete but is a starting point for (re)organizing the tags used for #operations-related work. This follows efforts initially made during the Phabricator move (T1147) as well as a Labs-specific effort earlier this year (T89270).

The current status quo, mostly established due to legacy (imported Bugzilla projects, RT queues and initial Phabricator experimentation) is:

  • A couple of team/purple tags (operations, Labs)
  • A few project/blue tags (such as Traffic and NetOps)
  • Lots of random/inconsistent yellow tags, often technology-specific (e.g. Varnish)

The goal here is to:

  • Subdivide Operations work into multiple areas and assign project/blue tags to them, to make them more manageable and triage-able (especially with work boards)
  • Move away from technology-specific tags that are often misused or may become legacy (what does "Puppet" mean? what are we going to do with the "Varnish" tag if we switch to another software for our caching proxies?)

To that end, we've assessed the situation as follows:

Team tags:

Existing project tags (keep as is):

New project tags:

New umbrella tags:

  • #Incidents (for all Incidents, superset or possibly replacement for Incident-YYYYMMDD-Description tags)

(please help to complete the list)

faidon created this task.Dec 1 2015, 10:09 AM
faidon updated the task description. (Show Details)
faidon raised the priority of this task from to Needs Triage.
faidon added projects: Operations, Project-Admins.
faidon added a subscriber: faidon.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 1 2015, 10:09 AM
fgiunchedi triaged this task as Normal priority.Dec 1 2015, 4:24 PM
fgiunchedi added a subscriber: fgiunchedi.

@Aklapper et al., any comments? Shall we proceed?

Aklapper updated the task description. (Show Details)Jan 20 2016, 6:48 PM
Aklapper set Security to None.

media-storage (convert the old yellow media-storage tag)

Could you clarify the relationship with Wikimedia-Media-storage please?

Further current tags that might be Ops-related (I assume they stay-as-is as non-changing stuff has not been explicitly mentioned in the task desc):
Diamond, Graphite, Icinga, LDAP, Mail, monitoring (shared with Eng) but also Shinken, Puppet, Varnish

Just mentioning for completeness: Blocked-on-Operations.

media-storage (convert the old yellow media-storage tag)

Could you clarify the relationship with Wikimedia-Media-storage please?

I wasn't even aware of that project… From the description it does sound like the same thing, but looking at the tasks there-in there seem to be quite a few ones that do not meet the criteria for being storage related (e.g. "Wikimedia Commons should support searching by color"). I'm not sure how to properly categorize these issues, maybe @Tgr (owner of that project) or @MarkTraceur/@matmarex (multimedia team) have an opinion?

faidon added a subscriber: BBlack.Jan 20 2016, 7:15 PM

Further current tags that might be Ops-related (I assume they stay-as-is as non-changing stuff has not been explicitly mentioned in the task desc):
Diamond, Graphite, Icinga, LDAP, Mail, monitoring (shared with Eng) but also Shinken, Puppet, Varnish

Just mentioning for completeness: Blocked-on-Operations.

Mail and monitoring were explicitly covered in the original task description.

LDAP should be converted to a project-tag, as it's about our LDAP infrastructure in general.

Diamond, Graphite, Icinga, Shinken, refer to specific pieces of monitoring software. I personally doubt their usefulness and believe they should be dropped in favor of the monitoring project tag, but I wouldn't also mind if they are to be kept as yellow tags and not used as workboards etc.

I doubt that Varnish is of any usefulness and that the project tag Traffic is enough. Maybe @BBlack has a differing opinion?

Puppet has been traditionally ambiguous (is it about the puppet infrastructure or about puppet code in general?) and of limited usefulness (I don't think anyone ever triages tasks tagged as Puppet, ever). I'd vote for this to be dropped.

(thanks for being thorough on this, much appreciated!)

media-storage (convert the old yellow media-storage tag)

Could you clarify the relationship with Wikimedia-Media-storage please?

I wasn't even aware of that project… From the description it does sound like the same thing, but looking at the tasks there-in there seem to be quite a few ones that do not meet the criteria for being storage related (e.g. "Wikimedia Commons should support searching by color").

That particular example is my fault, have moved a couple back to General-or-Unknown. :/

Tgr added a comment.Jan 20 2016, 8:43 PM

I wasn't even aware of [Wikimedia-Media-storage]… From the description it does sound like the same thing, but looking at the tasks there-in there seem to be quite a few ones that do not meet the criteria for being storage related (e.g. "Wikimedia Commons should support searching by color"). I'm not sure how to properly categorize these issues, maybe @Tgr (owner of that project) or @MarkTraceur/@matmarex (multimedia team) have an opinion?

Not an owner, just watching it. The project is an old one (was imported from bugzilla); I think it tends to be used for any media bug that doesn't really make sense outside WMF (ie. not a MediaWiki code issue). No point in keeping separate from the Swift project.

The tag @Krenair is looking for is MediaWiki-File-management I think.

I doubt that Varnish is of any usefulness and that the project tag Traffic is enough. Maybe @BBlack has a differing opinion?

I'd say for a tag like Varnish, we could just make sure all things with that tag also have Traffic, then remove Varnish from the system.

Related, we also have HTTPS which is kind of a subset of Traffic. Regardless of color, maybe a herald rule that it always adds Traffic when it's added? (like we do for Traffic always adding Operations)?

A herald rule for Domains and DNS -> Traffic might be useful too.

media-storage (convert the old yellow media-storage tag)

Could you clarify the relationship with Wikimedia-Media-storage please?

I wasn't even aware of that project… From the description it does sound like the same thing, but looking at the tasks there-in there seem to be quite a few ones that do not meet the criteria for being storage related (e.g. "Wikimedia Commons should support searching by color"). I'm not sure how to properly categorize these issues, maybe @Tgr (owner of that project) or @MarkTraceur/@matmarex (multimedia team) have an opinion?

It does sound like the same thing to me too, and if any tasks don't match, then they're mis-tagged. I could totally clean this up and empty up Wikimedia-Media-storage, moving the reports in it to media-storage or elsewhere if appropriate, before you go on with the big reorganization here. @faidon @Aklapper Do you want me to?

(By the way, we're Editing / Multimedia, please don't make me debug file storage issues ;) )

empty up Wikimedia-Media-storage, moving the reports in it to media-storage or elsewhere if appropriate, before you go on with the big reorganization here.

Cleaning up tickets that are not in the right basket is always welcome. :)
"Emptying it": Well I'm wondering if moving the other way round might be easier, plus I'm also wondering if people it will be harder for folks to find the right project if we use "swift" more consistently (but there are alternative project name hashtags in Phab).

empty up Wikimedia-Media-storage, moving the reports in it to media-storage or elsewhere if appropriate, before you go on with the big reorganization here.

"Emptying it": Well I'm wondering if moving the other way round might be easier, plus I'm also wondering if people it will be harder for folks to find the right project if we use "swift" more consistently (but there are alternative project name hashtags in Phab).

I think it's better to move to media-storage (and then rename it to media-storage), for the benefit of people who watch it, bookmarked it ,used it in searches, or whatever. No one is using Wikimedia-Media-storage, allegedly.

Wikimedia-Media-storage has no more open tasks and it's been archived. media-storage has twenty or so new ones :), and MediaWiki-File-management has also gained a few previously mistagged ones.

matmarex removed a subscriber: matmarex.Jan 26 2016, 4:56 PM
faidon added a comment.Feb 5 2016, 3:34 PM

@Aklapper, what about the rest?

Proposal looks good to me so feel free to go ahead. (Not sure "what the rest" means here, if there are specific points please bring them up and I'm happy to comment on them.)

Proposal looks good to me so feel free to go ahead. (Not sure "what the rest" means here, if there are specific points please bring them up and I'm happy to comment on them.)

To whom are you referring here? I'm personally not in the Project-Admins group — and in any case, I would prefer someone more experienced with Phabricator project "management" (Herald etc.) to tackle this.

Thanks @faidon.
Let me summarize where we are:

DONE:

  • DC-Ops: Created new team project.
  • ops-security: Created as blue project, without restrictions (as this happens on a different level)
  • Packaging: Created as blue project
  • DNS: Renamed from DNS
  • monitoring: Converted from yellow to blue
  • Mail: Converted from yellow to blue
  • media-storage: Converted from yellow to blue
  • Pybal: Converted from yellow to blue
  • Domains: Converted from yellow to blue
  • In general, I have not yet created any Herald rules and no join/edit/visibility restrictions created or changed.

TO SORT OUT:

Thanks @faidon.
Let me summarize where we are:

Thank you so much! :)

TO SORT OUT:

Yes, ideally.

  • DC-Ops: Please make people join that project (and others created when applicable).

Done.

Not really, although I feel your pain regarding all these ambiguous security-related tags. Happy to consider less ambiguous alternatives.

  • Clarify relationship of DNS with Domains (I guess)

I think it's clear enough by now, at least to me :) The former is for the infrastructure, the latter for domain procurement etc. issues — any suggestions on what would you like to clarify further and/or how?

  • Mail needs a project description (plus people sometimes file tasks here about the "Email user" function in MediaWiki)

Done.

  • #incidents: I have not created this project yet for a simply reason: We have existing Incident* projects and we can mass-add this superset project, but I don't see how we can add such a tag automatically for any newly created Incident* projects.

Fair enough. I wonder if there is much point in creating and maintaining all those separate Incident-[...] projects — I almost never use the tags themselves, and we do link to individual tasks from our incident reports. I can see arguments for both ways, though. What would be your preference?

Yeah, I think so. It's consistent with all of our other renames/projects.

Reading that:

(no Varnish herald rules, as we agreed to kill Varnish entirely, which I don't believe has happened yet)

Done!

Aklapper added a project: DevRel-February-2016.

TO SORT OUT:

Yes, ideally.

TODO: Two last questions: Am I right that the Operations team tag will remain on these tasks? If yes: There are 3 open tasks who are not tagged as Operations. Should it be added?
As per query I assume that "#ops-$site" means ops-ulsfo, ops-esams, ops-eqord, ops-eqiad, ops-eqdfw, ops-codfw, and procurement?
FYI: As Herald does not tag retroactively, I'd also batch-editing currently 124 open tasks. (I would not edit 1351 open+closed tasks as any action like reopening would trigger the Herald rule anyway).

  • Clarify relationship of DNS with Domains (I guess)

The former is for the infrastructure, the latter for domain procurement etc. issues — any suggestions on what would you like to clarify further and/or how?

DONE: Your explanation is beautiful hence I've edited the project descriptions accordingly.

  • #incidents: I have not created this project yet for a simply reason: We have existing Incident* projects and we can mass-add this superset project, but I don't see how we can add such a tag automatically for any newly created Incident* projects.

Fair enough. I wonder if there is much point in creating and maintaining all those separate Incident-[...] projects — I almost never use the tags themselves, and we do link to individual tasks from our incident reports. I can see arguments for both ways, though. What would be your preference?

SHRUG: I don't have a preference here. Hence I'm personally lazy and keep as is (as Herald does not allow conditions like "if project name starts with XYZ"). Probably a question better suited for those folks actively using those Incident tags... :)

Yeah, I think so. It's consistent with all of our other renames/projects.

DONE: Renamed #Swift and clarified its description (in contrast to MediaWiki-File-management): https://phabricator.wikimedia.org/project/manage/76/

Reading that:

DONE:

  • Created H131:
    • When all of these conditions are met:
    • Take these actions every time this rule matches:
  • Batch-edited 53 open tasks. (Same as above, didn't edit all 356 open and resolved tasks as Herald would be triggered when reopening anyway).

@Aklapper - relatedly, since we're killing the Varnish tag, we should probably batch-edit them to include Traffic before killing it (so that they have some useful tag in its place).

So we add Traffic to the 58 Varnish tasks and then archive Varnish? I can do that, in addition to the one TODO item left in T119944#2056247 (once that's sorted out)

So we add Traffic to the 58 Varnish tasks and then archive Varnish?

Yup!

RobH added a subscriber: RobH.EditedFeb 29 2016, 5:01 PM

Please don't have any herald rules add any kind of ops site specific projects to procurement tasks. For now, each procurement task can have those added manually by me, since each task has to be triaged by me as the buyer. Since its a fairly new space, and we are still changing how we use it regularly, I rather not have any automatically created herald rules applied to it.

Also not every procurement task will tie to a specific site. Many of them are specification refresh or pricing, which is for all sites. Addtionally, it covers all ssl purchases (not site specific), support contracts (multi-site), and carrier contract implementation details (multi-site.)

Thanks!

Edit Addition: Having Operations tie to procurement is fine; just not a specific site.

Aklapper moved this task from Backlog to Doing on the DevRel-March-2016 board.

A #devops tag was created two days ago. No idea how that adds to the mix (and its description really does not help).

faidon added a comment.Mar 8 2016, 2:44 PM

A #devops tag was created two days ago. No idea how that adds to the mix (and its description really does not help).

Agreed — I don't know what that is either. Where is the task that requested it?

(I'm aware I owe you a couple of responses, I haven't forgotten, will do so soon)

Aklapper added a subscriber: ori.Mar 8 2016, 3:47 PM

A #devops tag was created two days ago. No idea how that adds to the mix (and its description really does not help).

Agreed — I don't know what that is either. Where is the task that requested it?

CC'ing @ori who created it. (Not sure if the guidelines were followed.)

ori added a comment.Mar 10 2016, 1:48 AM

A #devops tag was created two days ago. No idea how that adds to the mix (and its description really does not help).

Agreed — I don't know what that is either. Where is the task that requested it?

CC'ing @ori who created it. (Not sure if the guidelines were followed.)

No, they weren't. I wasn't aware of the requirements; sorry. Let's delete the tag for now and I'll pitch it again in the proper way.

TO SORT OUT:

Yes, ideally.

TODO: Two last questions: Am I right that the Operations team tag will remain on these tasks? If yes: There are 3 open tasks who are not tagged as Operations. Should it be added?

It does not matter much either way, so whatever is easier. If both are the same, I'd probably do the simplest one (which also follows what we did with Cloud-Services) and do not tag with Operations.

As per [[ https://phabricator.wikimedia.org/project/query/tuP36MZyXKL9/#R | query ]] I assume that "#ops-$site" means #ops-ulsfo, #ops-esams, #ops-eqord, #ops-eqiad, #ops-eqdfw, #ops-codfw, and #procurement?

Correct!

FYI: As Herald does not tag retroactively, I'd also batch-editing currently 124 open tasks. (I would not edit 1351 open+closed tasks as any action like reopening would trigger the Herald rule anyway).

Alright!

That should be all — I have only one last question on whether we should explore the subproject feature that was recently deployed in our Phabricator instance. I'm not very familiar with it, so I'd rely on your advice :)

Thank you so much for all of your work here!

! In T119944#2044189, @faidon wrote:

Yes, ideally.

I created H140: Add DC-Ops project to ops-$site projects:

As Herald does not tag retroactively, I'd also batch-editing currently 124 open tasks. (I would not edit 1351 open+closed tasks as any action like reopening would trigger the Herald rule anyway).

Alright!

I've mass-added DC-Ops to 135 open tasks in ops-ulsfo, ops-esams, ops-eqord, ops-eqiad, ops-eqdfw, ops-codfw, procurement.

Am I right that the Operations team tag will remain on these tasks? If yes: There are 3 open tasks who are not tagged as Operations. Should it be added?

It does not matter much either way, so whatever is easier. If both are the same, I'd probably do the simplest one (which also follows what we did with Cloud-Services) and do not tag with Operations.

I added Operations to those three tasks but I did not set up any Herald rule or such, as "it does not matter much".

only one last question on whether we should explore the subproject feature that was recently deployed in our Phabricator instance. I'm not very familiar with it, so I'd rely on your advice :)

I'd love to defer that question to Team Practices because I have not played enough with that new feature. There is initial documentation but it seems to not cover yet when and why to use it.

I guess we're done here? If so, feel free to resolve.

faidon closed this task as Resolved.Mar 17 2016, 12:24 PM

Thanks for everything :)

RobH added a comment.EditedMar 18 2016, 10:43 PM

S

! In T119944#2044189, @faidon wrote:

Yes, ideally.

I created H140: Add DC-Ops project to ops-$site projects:

As Herald does not tag retroactively, I'd also batch-editing currently 124 open tasks. (I would not edit 1351 open+closed tasks as any action like reopening would trigger the Herald rule anyway).

Alright!

I've mass-added DC-Ops to 135 open tasks in ops-ulsfo, ops-esams, ops-eqord, ops-eqiad, ops-eqdfw, ops-codfw, procurement.

Can procurement please not apply any DC-Ops project? Many of the #procurment tasks are for SSL orders, or other non-site-specific items. Additionally, I'd prefer if no herald rules applied DC-Ops to every procurement task.

The only site specific action to any procurement task is receiving in the order; which is only the very end of the process on most orders. Any order implementation/racking/installation is done via an independent on-site task, linked to the #procrement task, but independent of it. I think this further lessens the need for dc-ops association in procurement.

Edit addition: I changed my phrasing around to reflect that its all viewpoint, not fact.
Thanks!

I created H140: Add DC-Ops project to ops-$site projects:

Can procurement please not apply any DC-Ops project? Many of the procurement tasks are for SSL orders, or other non-site-specific items. Additionally, I'd prefer if no herald rules applied DC-Ops to every procurement task.

For the time being, I've removed procurement from H140. Hence no Herald rules add DC-Ops to procurement tasks anymore.
It's also possible to (mass-)remove DC-Ops from those 47 tasks that have both DC-Ops and procurement tags; it's just that someone needs to tell me what to (not) do. :)

Aklapper reopened this task as Open.Mar 22 2016, 11:06 AM
Aklapper lowered the priority of this task from Normal to Low.Apr 1 2016, 10:22 AM

It's also possible to (mass-)remove DC-Ops from those 47 tasks that have both DC-Ops and procurement tags; it's just that someone needs to tell me what to (not) do. :)

@RobH, @faidon: Any comments? If not I'll resolve this task as is (and leave it to you to potentially remove those tags again).

Aklapper closed this task as Resolved.Apr 3 2016, 12:10 PM

Resolving as per last comment.

BBlack reopened this task as Open.May 3 2016, 11:35 PM

@Aklapper - The Varnish tag is still causing issues. Apparently it's still possible for it to be used in new tasks, which then don't end up with Traffic . An example is T133866 . Can we re-batch such tickets to add Traffic and then kill the tag?

Aklapper closed this task as Resolved.EditedMay 4 2016, 9:14 AM

Thanks for catching that. I somehow ignored T119944#2066410. :(

There were 85 Varnish tasks, 51 of them which didn't have also Traffic associated. I batch-edited them and disabled the Varnish project (and updated its description pointing to Traffic).

For the records, the following projects were changed from yellow tags to blue components lately:
Diamond, Elasticsearch, Icinga, Shinken. (Graphite, LDAP, Puppet are still in yellow.)

For the records, the following projects were changed from yellow tags to blue components lately:
Diamond, Elasticsearch, Icinga, Shinken. (Graphite, LDAP, Puppet are still in yellow.)

Since most Wikimedia components no longer have a Wikimedia- prefix in the name, and there are many components for operations-related stuff, can some/all of them become subprojects of a suitable parent project? Discoverability would be greatly improved.

@Nemo_bis you are writing on an old, closed, first-step effort. Ops tickets reorganization keeps happening, but we were blocked on technical concerns on phabricator support. There is probably another ticket open about that, if not one will be created with the ongoing discussions, but be assured that they point more or less in the direction you are suggesting, it is just need agreement among so many people (ops and users of ops tickets) that it takes some time. Small steps.

can some/all of them become subprojects of a suitable parent project?

No, because only subprojects on different levels in different subproject hierarchies can share tasks. But that's probably a discussion for a Talk page and not for this task. :)

(For the records, I changed H131 from "Take these actions every time" to "Take these actions only the first time" to allow Traffic to get rid of the BadHerald column on their workboard (see this example). For general info on Herald options, see the docs.)