@wiki_willy the related patch above should already help a lot, but as you know I'm off those days and I cannot give it the necessary testing for merging it, but if anyone else want to volunteer to merge+test it is welcome ;) Otherwise I'll take care of it as soon as I'm back.
As a workaround clearly is possible to add more permissions to dcops, it's a trivial change in puppet that anyone can do, but it's not to me to decide, that should be considered an access request to be decided by the owners of the group (SRE, usually discussed in the weekly meeting).

@crusnov in case you missed my IRC ping yesterday, please re-install the two public ones before proceeding with the installation as they had no firewall (see above hotfix patch).

In T229686#5455125, @CDanis wrote:

Here's a mini-design proposal for the dbctl feature itself (@Volans and @Joe please review):

• Add a flavor enum to the section schema. The default will be regular, which will cause the section to be output in the config in sectionLoads, as usual.
• The flavor externalstore will cause the section to be output in externalLoads.
  • Any groups set in the instances of a section with flavor=externalstore will be ignored.

Thanks for the update.

@wiki_willy The failed because of host unreachable, but is this still a commissioned host? I cannot see the records in the DNS repo, just the management ones are there. Also see its decom task: T224475

It was a maintenance, tracked with GIN-1-2116159603, that was not present to the calendar because sent to noc@ and not the maint announce ML. We need to update their settings.

It seems that the session is misconfigured on their side:

Aug 27 09:10:38  cr2-eqord rpd[13953]: bgp_process_open:4072: NOTIFICATION sent to 2001:418:0:5000::a34 (External AS 2914): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 2001:418:0:5000::a34 (External AS 2914) claims 65000, 2914 configured
Aug 27 09:10:42  cr2-eqord rpd[13953]: bgp_process_open:4072: NOTIFICATION sent to 128.241.2.53 (External AS 2914): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 128.241.2.53 (External AS 2914) claims 65000, 2914 configured

In T229657#5431230, @Marostegui wrote:

@CDanis @Volans can you confirm this command will set wikitech (db1073 is its master) on read-only?:

# set read-only
dbctl --scope eqiad section wikitech ro "Maintenance on wikitech T229657 " && dbctl config commit -m "Set wikitech as read-only for maintenance T229657"

Thanks!

That's because we pass memory={memory}g to the gnt-instance add command. We could instead accept a float in the cookbook, convert it to MB and use m in the command.
I'm fine either way. CCing @elukey that used it a lot and @crusnov

Test results of the module on cumin2001:

• fetch_host() doesn't raise if no host is found and returns None, should probably raise
• dry_run is not respected if using fetch_host() as the user can modify the object and call host.save(). I'm not sure what's the easy solution there.

Re-opening as there is still some work to do given that as it is right now is less redundant that it was before.
Namely I think we need:

• ensure images are replicated across both main datacenters
• ensure Content-Type is properly set so that image attachments are shown instead of downloaded
• decide a strategy to backup the attachments
• check if Netbox supports/is supposed to show thumbs of attached images
• check if on deletion the images are correctly removed from Swift too

I'm ok with the proposed UI but I'm wondering if we could reduce the duplication for the DBAs to set both a commit message and a note message when very often they will be the same.
The problem is that the note is set on the instance and the commit message only later when actually committing.
Also it would be hard to avoid to override pre-existing notes and to clean up when a host is back in normal shape.
So probably for now is just better to keep it simple at the cost of some duplication in some cases.

As the queue on grafana has gone back to zero too I'll resolve it for now. Thanks a lot for the fix @jbond

The solution that was agreed at the SRE summit for this is to add a dd to override the bootloader(s) so that the host cannot boot anymore and perform a shutdown of the host (most likely via IPMI, for VMs probably directly via ganeti).
At that point we can revoke certs in puppet, remove from debmonitor, etc... without the risk of any race or re-appearance.

@MoritzMuehlenhoff the dry-run mode is passed to all modules and all modules must implement unless they only do RO actions.
The difference in logging is because dry-run sets automatically debug logging to stdout, if you look at the file logs (normal and detailed) they show you all the steps.
The "skip" part was not added to the logging given the DRY-RUN in front, but ofc could be done if this is confusing.
PuppetDB has a queue and sure thing it has an issue to be investigated, see https://grafana.wikimedia.org/d/000000477/puppetdb?panelId=19&fullscreen&orgId=1&from=1564236470272&to=1565167416489

@Dzahn have you tried to follow https://wikitech.wikimedia.org/wiki/Management_Interfaces#Troubleshooting_Commands, in particular Management_Interfaces#Does_IPMI_works_but_SSH_to_the_management_console_doesn't? ?

@Joe any feedback on the above proposal? I'd really like to split the users ASAP given that dbctl is being deployed.

host downtimed on icinga until Friday ~15UTC. chatted with @Marostegui and the host is due decommission, so no hurry, he'll take a look tomorrow.

Unable to run hpssacli utility due to I/O error, I've depooled the host on dbctl and from db-codfw.php with the above patch (shortly). I'll look into logs after that.

If we go on the one file per host approach then I'd say we can read the file before writing so that we write/overwrite only if it's not there or has the wrong info. This should limit the re-write operations that in turn should reduce the race conditions of puppet non finding the file at the exact moment it's reading them to a negligible amount.
FYI row/rack don't change for most hosts during their lifetime, but in some cases we move hosts around, so it's a use case to take into account.

It seems to me that the simplest option would be #1, it would also be the one that optimizes API calls to Netbox (just one per puppetmaster every X minutes) and has a natural caching mechanism.
In addition we could add an alert if the file is stale (too old).

What was the issue?

errata corrige, I run the above with 'A:wdqs-internal and not P{wdqs1003.eqiad.wmnet}' instead to avoid to restart again 1003 that was already manually tested

In T97972#5352851, @Joe wrote:

IIRC we already have an account specialized for accessing only mwconfig, we could expand on the concept.

Resolving as this was merged already. The release process is out of scope for this task.

[stretch] Evaluate Netbox to store network secrets

Ack, thanks.

@hashar debmonitor uses Dpkg::Pre-Install-Pkgs for this feature because it's the only available hook from APT/DPKG that gives us the details of the operation that is made.
Any other option seemed very suboptimal. Things like save some temporary data with all the concurrency and stale risks involved + run something in Post-Invoke or send the full list each time in Post-Invoke or having a 2-way commit on the server side.

@wiki_willy sorry but cannot help as I've no special knowledge of this host or backup1001.

@Cmjohnson I've nothing to do with this host, I just commented because unable to remove a package as part of the conftool upgrade.

I had a chat on IRC with @Eevans, here some additional proposal that came up:

First of all I've a some questions due to my lack of knowledge of Cassandra specifics:

• Is there any configuration that could require a code change in the application? Can those be changed at will without any coordination with the application?
• Is there an easy way to connect to the cluster apart from localhost?
• If this configuration is not managed by Puppet how it will be managed? Would this system have a way to automatically apply those or we'll prefer to still do it manually for safety reasons?

@ayounsi I'm not sure the last two added in the last update should not alarm. What is the criteria used? According to the proposed document for incident response only incidents of level 5 should open tasks instead of alerting IMHO.

Most things were indeed fixed, I'm not sure on the status of the last 2 in the description checkboxes list. But they shouldn't affect anymore reboots/restarts but at most new installations. I'll rename it

All patches for v1 of dbconfig are merged, including the ones to make a new conftool release. I'll rebuild the packages on Monday and upload them to our APT and I'll start testing the upgrade of python3-conftool (the base package, pretty much untouched).
The testing of python3-conftool-dbctl will follow after.

The data model is now part of the software and will evolve with it, wikitech documentation will be provided for it. I'm resolving this.

@Gehel I'll leave the task open if you want to investigate more tomorrow for potential hardware parts to replace. (see above for hardware logs).

Both clusters back to green:

 elastic2054  0 ~$ curl -s localhost:9600/_cluster/health?pretty
{
  "cluster_name" : "production-search-psi-codfw",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 15,
  "number_of_data_nodes" : 15,
  "active_primary_shards" : 1450,
  "active_shards" : 4349,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
 elastic2054  0 ~$ curl -s localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "production-search-codfw",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 30,
  "number_of_data_nodes" : 30,
  "active_primary_shards" : 1254,
  "active_shards" : 3741,
  "relocating_shards" : 2,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Nothing in syslog.
It first detected a CPU error and then a memory one, here the hardware logs:

-------------------------------------------------------------------------------
Record:      4
Date/Time:   07/04/2019 21:12:40
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------
Record:      5
Date/Time:   07/04/2019 21:12:40
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      6
Date/Time:   07/04/2019 21:12:41
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   07/04/2019 21:12:41
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      8
Date/Time:   07/04/2019 21:12:41
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      9
Date/Time:   07/04/2019 21:15:21
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      10
Date/Time:   07/04/2019 21:15:21
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
-------------------------------------------------------------------------------

The host is part of the main and psi clusters:

$ confctl --quiet select name="elastic2054.codfw.wmnet" get
{"elastic2054.codfw.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=codfw,cluster=elasticsearch,service=elasticsearch"}
{"elastic2054.codfw.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=codfw,cluster=elasticsearch,service=elasticsearch-psi-ssl"}
{"elastic2054.codfw.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=codfw,cluster=elasticsearch,service=elasticsearch-ssl"}

I assume that this host will be reimaged, but in case it's not, please manually run:

apt-get remove python-conftool

once fixed.

In T226965#5295407, @Joe wrote:

What we need to do is:

• Upgrade python3-etcd to the latest version
• Upgrade python3-conftool to the latest version
• Remove python-conftool if present

The blocker above has been fixed in scap, released and rolled out to the fleet. I'll proceed with the removal of python-conftool.

Scap has been rolled out to the fleet, I've tested a scap pull on a mwdebug host and there was a scap sync-file deployment for mediawiki-config (DB stuff).

@akosiaris the "plan" was partially explained as part of the bare metal/host provisioning breakout session at the SRE Summit. You can find more details in the notes of the summit but basically the TL;DR is that as part of the effort to automate host provisioning we're aiming to have a system in which we don't need to hardcode MAC addresses anymore.
The details of the plan are evolving with the plan itself but the gist is that it will involve DHCP option 82 (or IPv6 autoconf alternatively) and iPXE (or equivalent) to dynamically map a physical host to data available in Netbox and from there drive the whole installation process with the required parameters.
Ping me offline if you want more details.

Thanks a lot @elukey. I've just added a small detail and formatted hosts and paths.
For the status of the task, the port of makevm script is done, the remaining additional parts are still pending.

@elukey: yes because of the temporary suppression of cumin's default output, to allow each cookbook to decide what to do with it, this specific one is printing its output at the end. This will soon-ish be more flexible on cumin side and should allow spicerack to expose it in a way that each cookbook can choose what to do with it. Sorry for the non-optimal experience for now.

Not yet as the script has clearly not been tested:

$ sudo cookbook sre.ganeti.makevm -h
Exception raised while parsing arguments for cookbook sre.ganeti.makevm:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/cookbook.py", line 460, in _parse_args
    args = self.module.argument_parser().parse_args(self.args)
  File "/srv/deployment/spicerack/cookbooks/sre/ganeti/makevm.py", line 37, in argument_parser
    clusters_and_rows = [cluster + '_' + row for cluster, row in CLUSTERS_AND_ROWS.items()]
  File "/srv/deployment/spicerack/cookbooks/sre/ganeti/makevm.py", line 37, in <listcomp>
    clusters_and_rows = [cluster + '_' + row for cluster, row in CLUSTERS_AND_ROWS.items()]
TypeError: Can't convert 'tuple' object to str implicitly

Yes it's confirmed that the Icinga check flaps between critical and unknown due to time outs and as a result the even handler created the dupes. See more specific info in T224794#5295606, basically megacli takes very long time to gather info from the broken disk.

The automatic gathering times out because megacli takes ~3 minutes to return the status of the disks, it blocks at PD7 (the one broken) and takes very long time to get info from that disk.

For debmonitor it connects to m2-master.eqiad.wmnet and I'm not sure if Django's connection pooling would be smart enough to reconnect given that the old one will still work, just RO. It might need a:

sudo cumin 'A:debmonitor' 'systemctl restart uwsgi-debmonitor.service'

just after the switch.
Alternatively if you plan to kill all existing connections to the old master that would do the trick already, because debmonitor will automatically reconnect and the proxy will send it to the new one.

Sorry for the spam. My guess is that the check is flapping between critical and unknown. The script ignores the unknowns but it doesn't know if there is already a task opened (long story).
I can have a check tomorrow, I'm without laptop at the moment. (It might also be related to the CPU governor task).
For now I've disabled the event handler on icinga for that check on that host so it should not spam anymore. Let me know in case it generate any additional noise and I can try to have a deeper luck tonight or silent it even more.

@elukey you can disable icinga notification on a cluster via hiera. Alternatively to disable only this kind of check you can disable event handler from Icinga UI. I'm not sure if we have any more fine-tuned way to skip this ones.