Page MenuHomePhabricator

Automate the creation of implementation task from rack/setup/install tasks for Serviceops
Open, LowPublic

Description

After the rack/setup/install tasks are completed, Serviceops needs a task to track the implementation.

This was done manually so far and is error prone, which risks that we don't use racks that are ready and waste resource.

We suggest adding an automation in the form of Herald rule on those tasks.

A production perspective of this is also to put an alert on cumin for insetup hosts: sudo cumin 'A:owner-serviceops and A:insetup' being run regularly would identify the stale hosts.

Event Timeline

Currently, this is the list of hosts:

deploy2003.codfw.wmnet,
mc-misc[2001-2002].codfw.wmnet,
mc-misc[1001-1002].eqiad.wmnet,
rdb[2011-2012].codfw.wmnet,
wikikube-ctrl[2004-2006].codfw.wmnet,
wikikube-worker2331.codfw.wmnet,
wikikube-worker[1328-1372].eqiad.wmnet

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

  • has rack/setup/install {hosts} in its task name
  • has the tags #DC-Ops and #serviceops
  • has its status becoming Resolved

Then:

  • create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

  • has rack/setup/install {hosts} in its task name
  • has the tags #DC-Ops and #serviceops
  • has its status becoming Resolved

Then:

  • create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

s/#serviceops/#serviceops-new/g

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

  • has rack/setup/install {hosts} in its task name
  • has the tags #DC-Ops and #serviceops
  • has its status becoming Resolved

Then:

  • create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

s/#serviceops/#serviceops-new/g

Please note I didn't know about #serviceops-new but I can adjust my workflow accordingly and when I create a racking task use serviceops-new rather than serviceops. This change overall would be awesome, as right now I manually create these implementation tasks when I create the racking task and they sit for serviceops to see the racking task resolve, this proposed workflow is better imo.

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

  • has rack/setup/install {hosts} in its task name
  • has the tags #DC-Ops and #serviceops
  • has its status becoming Resolved

Then:

  • create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

s/#serviceops/#serviceops-new/g

Please note I didn't know about #serviceops-new but I can adjust my workflow accordingly and when I create a racking task use serviceops-new rather than serviceops.

It's ok, any new task created with the old tag gets re-tagged automatically by Herald.

Phabricator (cc @Aklapper ) would you be able to create this Herald rule for us?

Those are recurring tasks every quarter, implementation task creation is manual and can be forgotten, resulting in waste of hardware resource.
This seems like an easy win to avoid missing one and streamline our operations between SRE teams.

As this rule affects task creation I don't have the rights to create it so will need your help.

Recent examples:

Please let me know if you need any other information

Herald rules currently can not automatically create new tasks

Would you advise any workarounds for this, should we use a periodic job to create those tasks?

@Aklapper would you have any idea on how to achieve this?

Hi, I unfortunately do not have a great idea how to "simply" achieve this. Currently this likely requires an external bot to create followup tasks via Conduit.
I wonder if there is anything similar around (e.g. maybe in procurement, or when creating the task dependency tree for releasing a new MediaWiki tarball release) but WMF is traditionally unorganized in documenting Phabricator-related scripts/bots and skillsets which may allow re-use of some existing code...

If this was ever supported by Herald rules then my first question would be who should be the "task author" to create such "automated" followup tasks - probably Herald itself? I'm pretty sure that upstream Phorge would reply that this is in the field of external code to use the API though...

If this was ever supported by Herald rules then my first question would be who should be the "task author" to create such "automated" followup tasks - probably Herald itself? I'm pretty sure that upstream Phorge would reply that this is in the field of external code to use the API though...

No strong opinion here, as long as the task is created with the right tags and lands in our Inbox.

@Jhancock.wm I see the procurement tasks always follow the same template, are they created by any automation or script?

@MLechvien-WMF we use a phabricator template that @RobH created. He'd be the one to ask about how it works.

Update: No progress on that yet due to competing priorities