Page MenuHomePhabricator
Create Task
Maniphest T430023

Versitygw certificate update doesn't get picked by the service
Open, HighPublic
Actions

Description

From 2026-06-12 until 2026-06-24 on codfw and from 2026-05-19 until 2026-06-24 on eqiad, mediabackups stopped being updated with newly uploaded files. Restoration or deletion of files (but not querying) would have failed also due to an expired TLS certificate, due to a missed refresh dependency on puppet code.

Original info:

My guess it is due to some of the following:

  • The puppet code has a bug where refresh of the certificate doesn't trigger an automatic reload (just a mistake due to setup being done in a hurry)
  • A reload is not enough to refresh the presented certificate and it requires a full restart (which will be an issue for ensuring continuous availability)

As automatic certificated renewal is there.

Two actionables are needed:

  1. Monitor freshness of service certificate so it alerts if invalid or expired
  2. Debug and fix the above issue so new certificates are picked right away, with minimal downtime

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
mediabackups: Update versitygw systemd unit file to SIGHUP on reloadJcrespooperations/puppetproduction+15 -2
versitygw: Fix service not reloading after certificate changeJcrespooperations/puppetproduction+4 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• jcrespo created this task.Jun 24 2026, 9:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2026, 9:16 AM
• jcrespo triaged this task as High priority.Jun 24 2026, 9:16 AM
gerritbot added a comment.Jun 24 2026, 9:28 AM

Change #1305356 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] versitygw: Fix service not reloading after certificate change

https://gerrit.wikimedia.org/r/1305356

gerritbot added a project: Patch-For-Review.Jun 24 2026, 9:28 AM
Tgr removed a project: MediaWiki-Core-Platform-Team.Jun 24 2026, 10:16 AM
Tgr subscribed.

Wrong team, I think?

• jcrespo added a comment.Jun 24 2026, 11:44 AM

Apologies @Tgr, that was supposed to add Data-Persistence-Backup not platform.

• jcrespo added a project: Data-Persistence-Backup.Jun 24 2026, 11:44 AM
• jcrespo removed a subscriber: Tgr.
gerritbot added a comment.Jun 24 2026, 11:46 AM

Change #1305356 merged by Jcrespo:

[operations/puppet@production] versitygw: Fix service not reloading after certificate change

https://gerrit.wikimedia.org/r/1305356

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2026, 12:33 PM
• jcrespo closed this task as Resolved.Jun 26 2026, 10:13 AM

I am going to resolve it because it was a one time mistake, and while monitoring is still needed, we will need a separate ticket and extensive approach to it.

• jcrespo added a project: Incident Severity 3.Jun 29 2026, 1:25 PM
• jcrespo mentioned this in T430526: Setup basic media backups alerting.Jun 29 2026, 1:36 PM
• jcrespo added a project: Wikimedia-Incident.
• jcrespo updated the task description. (Show Details)Jun 29 2026, 1:53 PM
MLechvien-WMF moved this task from Active investigation to Resolved on the Wikimedia-Incident board.Jul 1 2026, 7:53 AM
• jcrespo reopened this task as Open.EditedAug 4 2026, 9:16 PM

This happened again- but I caught it quickly this time. Either the reload didn't work (certainly the puppet patch is not enough) or it requires a full service restart, which opens questions on how to do it without causing downtime.

A rolling restart of all daemons made it work again- until next month.

• jcrespo added a comment.Aug 4 2026, 9:21 PM

I think we need to hardcode a SIGHUP signal on reload for the systemd unit: https://github.com/versity/versitygw/commit/0c520a30cff3bffeacb42c20616b24229675ed2b

• jcrespo added a comment.Aug 4 2026, 9:33 PM

Type=notify-reload seems to be the right way to go about this.

gerritbot added a comment.Aug 5 2026, 8:14 AM

Change #1321123 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackups: Update versitygw systemd unit file to SIGHUP on reload

https://gerrit.wikimedia.org/r/1321123

gerritbot added a project: Patch-For-Review.Aug 5 2026, 8:14 AM
• jcrespo removed • jcrespo as the assignee of this task.Aug 13 2026, 8:53 AM

This will break again on Sep 5 12:44:00 2026 GMT.

Aklapper added a comment.Sep 1 2026, 3:28 PM
In T430023#12211111, @jcrespo wrote:

This will break again on Sep 5 12:44:00 2026 GMT.

That will be in 4 days.

MLechvien-WMF added a subtask: T430526: Setup basic media backups alerting.Tue, Sep 8, 7:32 AM
Marostegui edited subscribers, added: CWilliams-WMF, KOfori, MatthewVernon, Marostegui; removed: • jcrespo.Thu, Sep 10, 3:25 PM

We should make this UBN I think

CWilliams-WMF added a comment.Tue, Sep 15, 1:11 PM

I have tested the HUP, all works as expected:

cwilliams@backup2019 ~ % echo -n | openssl s_client -connect $(hostname -f):9000 | openssl x509 -noout -dates                                                                                                                                                         
Connecting to 10.192.47.6
depth=2 C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc, OU=Cloud Services, CN=Wikimedia_Internal_Root_CA
verify return:1
depth=1 C=US, L=San Francisco, O=Wikimedia Foundation, Inc, OU=SRE Foundations, CN=discovery2026
verify return:1
depth=0 CN=backup2019.codfw.wmnet
verify return:1
DONE
notBefore=Aug 25 12:10:00 2026 GMT
notAfter=Sep 22 12:10:00 2026 GMT

cwilliams@backup2019 ~ % pgrep -f /srv/objectstorage00 | xargs ps -opid,ppid,etime,cmd | tee                 
    PID    PPID     ELAPSED CMD
3438848       1  4-02:24:55 /usr/bin/versitygw --cert /etc/versitygw/ssl/discovery2026__backup2019_codfw_wmnet.chained.pem --key /etc/versitygw/ssl/discovery2026__backup2019_codfw_wmnet-key.pem posix /srv/objectstorage00

cwilliams@backup2019 ~ % sudo pkill -HUP -f /srv/objectstorage00

cwilliams@backup2019 ~ % echo -n | openssl s_client -connect $(hostname -f):9000 | openssl x509 -noout -dates
Connecting to 10.192.47.6
depth=2 C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc, OU=Cloud Services, CN=Wikimedia_Internal_Root_CA
verify return:1
depth=1 C=US, L=San Francisco, O=Wikimedia Foundation, Inc, OU=SRE Foundations, CN=discovery2026
verify return:1
depth=0 CN=backup2019.codfw.wmnet
verify return:1
DONE
notBefore=Sep 11 11:40:00 2026 GMT
notAfter=Oct  9 11:40:00 2026 GMT

cwilliams@backup2019 ~ % sudo journalctl -u versitygw@objectstorage00.service --since "2 minutes ago" --no-pager
Sep 15 13:08:10 backup2019 versitygw[3438848]: caught signal hangup
Sep 15 13:08:10 backup2019 versitygw[3438848]: srv cert reloaded (cert: /etc/versitygw/ssl/discovery2026__backup2019_codfw_wmnet.chained.pem, key: /etc/versitygw/ssl/discovery2026__backup2019_codfw_wmnet-key.pem)

I will update the MR for this/replace it with a new one.

CWilliams-WMF added a comment.Tue, Sep 15, 1:31 PM

For now, as the certs were already regenerated, I have issued a HUP across all of the instances:

cwilliams@cumin1004 ~ % sudo cumin --batch-size=1 --batch-sleep=5 --mode=sync 'P{O:mediabackup::new_storage}' 'pkill -HUP -f "^/usr/bin/versitygw"'                               
12 hosts will be targeted:
backup[2015-2020].codfw.wmnet,backup[1015-1020].eqiad.wmnet
OK to proceed on 12 hosts? Enter the number of affected hosts to confirm or "q" to quit: 12
===== NO OUTPUT =====                                                                                                                                                                                                                                                  
PASS | 100% (12/12) [01:09<00:00,  5.82s/hosts]
FAIL |  0% (0/12) [01:09<?, ?hosts/s]
100.0% (12/12) success ratio (>= 100.0% threshold) for command #1: 'pkill -HUP -f "^/usr/bin/versitygw"'.
100.0% (12/12) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

cwilliams@cumin1004 ~ % sudo cumin 'P{O:mediabackup::new_storage}' 'echo -n | openssl s_client -connect $(hostname -f):9000 | openssl x509 -noout -enddate'
....
===== NODE GROUP =====                                                                                                                                                                                                                                                 
(1) backup1018.eqiad.wmnet                                                                                                                                                                                                                                             
----- OUTPUT for command #1: 'echo -n | openss... -noout -enddate' -----                                                                                                                                                                                               
Connecting to 10.64.185.4                                                                                                                                                                                                                                              
depth=2 C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, Inc, OU=Cloud Services, CN=Wikimedia_Internal_Root_CA
verify return:1
depth=1 C=US, L=San Francisco, O=Wikimedia Foundation, Inc, OU=SRE Foundations, CN=discovery2026
verify return:1
depth=0 CN=backup1018.eqiad.wmnet
verify return:1
DONE
notAfter=Oct  9 11:25:00 2026 GMT
================                                                                                                                                                                                                                                                       
PASS | 100% (12/12) [00:01<00:00, 10.66hosts/s]
FAIL |   0% (0/12) [00:01<?, ?hosts/s]
100.0% (12/12) success ratio (>= 100.0% threshold) for command #1: 'echo -n | openss... -noout -enddate'.
100.0% (12/12) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
gerritbot added a comment.Thu, Sep 17, 8:43 AM

Change #1321123 merged by CWilliams:

[operations/puppet@production] mediabackups: Update versitygw systemd unit file to SIGHUP on reload

https://gerrit.wikimedia.org/r/1321123

Maintenance_bot removed a project: Patch-For-Review.Thu, Sep 17, 9:31 AM
CWilliams-WMF added a comment.Thu, Sep 17, 11:23 AM

The MR has been merged with its original intent, to SIGHUP on reload. However, a new MR will be needed to do the correct checks to determine if the content changed, as the File subscription does not relate to the content, merely its permissions, ownership, etc.

CWilliams-WMF closed subtask T430526: Setup basic media backups alerting as Resolved.Fri, Sep 18, 9:33 AM
MLechvien-WMF subscribed.Wed, Sep 30, 10:25 AM

Does this need to continue being tracked as an open Wikimedia-Incident ?

CWilliams-WMF added a comment.Wed, Sep 30, 10:32 AM

@MLechvien-WMF I don't believe so. We have monitoring now, the certificates can be reloaded with systemctl reload, so it is just the automated reload that is missing as that needs so more work.

Marostegui removed projects: Wikimedia-Incident, Incident Severity 3.Mon, Oct 5, 5:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits