Running decom cookbook for wdqs100[3,4]. Dc-ops ticket up here: https://phabricator.wikimedia.org/T346699

Removed subtask because I think the scap ticket is not directly related to this one.

The reindex, even though we had to terminate it before it finished, had already gotten to enwiki_content. So this is done. Here's what the value on both eqiad and codfw looks like:

Host was reimaged but patch wasn't yet merged. Merging patch and rolling the re-image again.

Oops, meant this to be in progress. Changing status to Open from previous state of Resolved.

We made a slight mistake: one of these hosts needed to be in wdqs-internal since wdqs1003 (one of the hosts replaced by these new hosts) is.

The work for this ticket is done, but I made the actual decom ticket a subtask so I'll leave this in blocked/waiting until that's done

Generated new cergen certs for wdqs.discovery.wmnet that include wdqs1016 in the alt_names instead of wdqs1005. Followed the steps below:

Some observations from last two patches, tested on wdqs2007 before reverting due to issues:

Built wmf-elasticsearch-search-plugins_7.10.2-9 and wmf-elasticsearch-search-plugins_7.10.2-9~bullseye (https://apt.wikimedia.org/wikimedia/pool/thirdparty/elastic710/w/wmf-elasticsearch-search-plugins/); installed on all elastic* hosts (incl. relforge* and cloudelastic*). Rolling restarts not completed yet. relforge* can be restarted at any time, but elastic* and cloudelastic* must wait till after an ongoing reindex of all wikis has completed.

Will be in blocked/waiting for a few days while a reindex of all wikis completes to apply the newest settings.

Patch was merged here: https://gerrit.wikimedia.org/r/947928

Looks like we lost track of this a bit. @bking and I can work this this week.

Just some investigation we did to understand where the metrics come from: probe_ssl_earliest_cert_expiry comes from the blackbox exporter (see random docs). That metric is used by the alerting repo here: https://github.com/wikimedia/operations-alerts/blob/4ecc222e95710395a6f9a7039e487186d2264323/team-sre/probes.yaml#L55

Checked like so:

In T338159#9002663, @EBernhardson wrote:

It looks like we added only the link, could we add a paragraph about how to use this dashboard as well?

Decom cookbook finished, and dc-ops ticket created (see ticket desc AC section for ticket #)

wdqs202[1-2] have been brought into service. With teh merging of https://gerrit.wikimedia.org/r/c/operations/puppet/+/940272, all hosts are now in service and have alerting enabled.

With the new hosts in service, we can now begin decom'ing these hosts at our convenience.

All of these hosts except wdqs202[1-2] are in service. Those last two hosts will be brought in service after a final data xfer (ongoing).

In T342162#9025774, @thcipriani wrote:

I think I have the context to understand this.

It looks like /srv/deployment/wdqs/wdqs-cache/revs/$CURRENT_DEPLOY_COMMIT_HASH/.git/config-files/etc/query_service is symlinked at /etc/query_service/ldf-config.json is that true?

I see this line in log output:

/etc/query_service/ldf-config.json is already linked to current rev(use --force to override)

Which is exactly what you're describing. It comes from the check in scap deploy here: https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/deploy.py#L212

The assumption is that if a file is already symlinked, there's no need to regenerate the file. But it sounds like it's a bad assumption in this case, is that true?

First draft of this ticket up. There's a couple things that aren't perfect:

Merged patch (had wrong ticket in commit message): https://gerrit.wikimedia.org/r/c/operations/puppet/+/934403

This should be done, but I haven't yet ran a validation command to sanity check that the correct version is in place.

Should be deployed as of today.

Thanks for the patience on this! This is getting deployed today.

Documentation aspect of this ticket's already done. Basically two things left to do to close this ticket out: