Page MenuHomePhabricator

ServiceLVS without monitor breaks spicerack
Open, LowPublic

Description

spicerack.ServiceLVS based cookbooks are failing because of the wikireplicas services having their monitors commented out (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/924342 and T337446: Rebuild sanitarium hosts for context)

cgoubert@cumin1001:~$ sudo cookbook sre.discovery.datacenter status eqiad                                                   [978/978]
                                                                                                                                     
Exception raised while initializing the Cookbook sre.discovery.datacenter:                                                           
Traceback (most recent call last):                                                                                                   
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 197, in run                                                         
    runner = self.instance.get_runner(args)                                                                                          
  File "/srv/deployment/spicerack/cookbooks/sre/discovery/datacenter.py", line 214, in get_runner                                    
    return DiscoveryDcRouteRunner(args, self.spicerack)                                                                              
  File "/srv/deployment/spicerack/cookbooks/sre/discovery/datacenter.py", line 230, in __init__                                      
    self.discovery_records = self._get_all_services()                                                                                
  File "/srv/deployment/spicerack/cookbooks/sre/discovery/datacenter.py", line 472, in _get_all_services                             
    for service in self.catalog:                                                                                                     
  File "/usr/lib/python3/dist-packages/spicerack/service.py", line 552, in <genexpr>                                                 
    return (self.get(name) for name in self._catalog)                                                                                
  File "/usr/lib/python3/dist-packages/spicerack/service.py", line 607, in get                                                       
    params["lvs"] = ServiceLVS(**params["lvs"])                                                                                      
TypeError: __init__() missing 1 required positional argument: 'monitors'

Digging into it, spicerack's dataclass for ServiceLVS has monitors as a mandatory field, when puppet's modules/wmflib/types/service/lvs.pp describes it as optional.

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Clement_Goubert renamed this task from Service without monitor breaks spicerack to ServiceLVS without monitor breaks spicerack.Jun 15 2023, 2:47 PM

Change 930644 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/software/spicerack@master] service: Make lvs[monitors] optional in ServiceLVS

https://gerrit.wikimedia.org/r/930644

That was changed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/924342 without modifying spicerack although it's written on line 3:

# If modified, update also spicerack.service.ServiceLVS

I figured. I merged the change, tell me when you cut a release and we can resolve.

Change 930644 merged by jenkins-bot:

[operations/software/spicerack@master] service: Make lvs[monitors] optional in ServiceLVS

https://gerrit.wikimedia.org/r/930644

Change 931241 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Ensure service catalog schema matches spicerack release

https://gerrit.wikimedia.org/r/931241

Volans triaged this task as High priority.Jun 19 2023, 3:00 PM

The ideal solution would be to make the spicerack class accept happily any undefined parameter, the only problem with that is that are all dataclasses and would basically need to be converted back to normal classes and loose the benefit of the dataclass in order to do that.

Release has been cut, cookbook works ok now. @Volans do you want to keep the task open to keep looking for a way to enforce/workaround the issue of the tight coupling between the spicerack class and the puppet type?

I was about to come here and comment as I just deployed spicerack to production :)
I think we can keep it open for now to see if we reach any consensus, but we could also just resolve this and open a separate one for that.

joanna_borun lowered the priority of this task from High to Low.Nov 27 2023, 4:23 PM

We had only a couple of changes in the service.yaml schema in the last months and both were sent to Spicerack before hitting production on the Puppet side, so nothing broke in those cases.
What we were thinking is to instead of a refactor of the whole thing in spicerack maybe it would be simpler to have a CI check in puppet that checks that the fields are all there.

Anyway, leaving this open with lower priority for the long term fix.