Page MenuHomePhabricator
Create Task
Maniphest T262816

The certificate for en.wikipedia.beta.wmflabs.org expired on 2020-09-14
Closed, ResolvedPublic
Actions

Description

Firefox:

Warning: Potential Security Risk Ahead

Firefox detected an issue and did not continue to en.wikipedia.beta.wmflabs.org. The website is either misconfigured or your computer clock is set to the wrong time.

It’s likely the website’s certificate is expired, which prevents Firefox from connecting securely. If you visit this site, attackers could try to steal information like your passwords, emails, or credit card details.

What can you do about it?

The issue is most likely with the website, and there is nothing you can do to resolve it. You can notify the website’s administrator about the problem.

Learn more…

Websites prove their identity via certificates, which are valid for a set time period. The certificate for en.wikipedia.beta.wmflabs.org expired on 9/14/2020.
 
Error code: SEC_ERROR_EXPIRED_CERTIFICATE
 
View Certificate

firefox.png (910×1 px, 226 KB)

Chrome:

Your connection is not private
Attackers might be trying to steal your information from en.wikipedia.beta.wmflabs.org (for example, passwords, messages or credit cards). Learn more
NET::ERR_CERT_DATE_INVALID

Help improve security on the web for everyone by sending URLs of some pages that you visit, limited system information, and some page content to Google. Privacy policy
This server could not prove that it is en.wikipedia.beta.wmflabs.org; its security certificate expired in the last day. This may be caused by a misconfiguration or an attacker intercepting your connection. Your computer's clock is currently set to Monday, 14 September 2020. Does that look right? If not, you should correct your system's clock and then refresh this page.

Proceed to en.wikipedia.beta.wmflabs.org (unsafe)

chrome.png (1×1 px, 241 KB)

Safari:

This Connection Is Not Private

This website may be impersonating “en.wikipedia.beta.wmflabs.org” to steal your personal or financial information. You should close this page.

Safari warns you when a website has an expired certificate. This website’s certificate expired 1 day ago. This may happen if the website is misconfigured, an attacker has compromised your connection or your system clock is incorrect. Your system clock is set to Monday, 14 September 2020. If this is not right, fixing the clock may address this warning.

To learn more, you can view the certificate. If you understand the risks involved, you can visit this website.

safari.png (821×1 px, 811 KB)

Related Objects
Search...

Event Timeline

zeljkofilipin created this task.Sep 14 2020, 11:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 14 2020, 11:23 AM
DannyS712 added a subscriber: DannyS712.Sep 14 2020, 11:23 AM
zeljkofilipin triaged this task as Unbreak Now! priority.Sep 14 2020, 11:24 AM
zeljkofilipin moved this task from Backlog ⏪ to Waiting... 🐢 on the User-zeljkofilipin board.
zeljkofilipin updated the task description. (Show Details)Sep 14 2020, 11:29 AM
zeljkofilipin updated the task description. (Show Details)Sep 14 2020, 11:32 AM
zeljkofilipin updated the task description. (Show Details)Sep 14 2020, 11:35 AM
zeljkofilipin updated the task description. (Show Details)
MarcoAurelio added subscribers: Krenair, MarcoAurelio.Sep 14 2020, 11:58 AM

I think this is managed via Acme-chief which I also think @Krenair is familiar with. Not sure if Traffic needs to be involved. Some docs at Wikitech mention purchases, etc. I hope that Alex can clarify.

Urbanecm added a subscriber: Urbanecm.Sep 14 2020, 12:05 PM

I logged into deployment-cache-upload06.deployment-prep.eqiad1.wikimedia.cloud and it says The last Puppet run was at Tue Sep 8 22:31:55 UTC 2020 (8012 minutes ago).. Can that be the issue?

Krenair added a comment.Sep 14 2020, 12:12 PM

It's possible - if acme chief has got a new cert issued but the cache-text
box hasn't run puppet since, you'll see this. Check whether acme-chief has
a new one and if it does, fix puppet on cache-text. If not investigate why.
Am having lunch and then working again but I can look this evening if no
one has fixed it by then.

(Almost?) all references to purchasing certificates will be out of date
legacy things

Aklapper renamed this task from The certificate for en.wikipedia.beta.wmflabs.org expired on 9/14/2020 to The certificate for en.wikipedia.beta.wmflabs.org expired on 2020-09-14.Sep 14 2020, 12:18 PM
RhinosF1 added a subscriber: RhinosF1.Sep 14 2020, 1:08 PM
Aklapper mentioned this in T262806: Beta cluster certificates have expired (September 2020).Sep 14 2020, 1:22 PM
Kizule merged a task: T262806: Beta cluster certificates have expired (September 2020).Sep 14 2020, 1:24 PM
Kizule added subscribers: Mainframe98, Kizule.
Urbanecm added a comment.Sep 14 2020, 1:35 PM

/var/lib/acme-chief/certs/unified/live/ec-prime256v1.chained.crt at acme-chief is the new certificate, so the issue is probably broken puppet at cache-text.

Trying to run Puppet on cache-text manually does the following:

urbanecm@deployment-cache-text06:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, No rule found for citoid.wikimedia.org in profile::trafficserver::backend::mapping_rules (file: /etc/puppet/modules/profile/functions/trafficserver_caching_rules.pp, line: 12, column: 17) on node deployment-cache-text06.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
urbanecm@deployment-cache-text06:~$

I attempted to fix that by removing .wikimedia.org domains from hiera's cache::req_handling (and setting it to {}). That did make Puppet to partially run (certificate is renewed in /etc/acmecerts), but it ended with a different error:

urbanecm@deployment-cache-text06:/etc/varnish$ sudo puppet agent -tv                                                                                                                                       [57/896]Info: Using configured environment 'production'                                                                                                                                                                    Info: Retrieving pluginfacts                                                                                                                                                                                       Info: Retrieving plugin                                                                                                                                                                                            Info: Retrieving locales                                                                                                                                                                                           Info: Loading facts                                                                                                                                                                                                Info: Caching catalog for deployment-cache-text06.deployment-prep.eqiad.wmflabs                                                                                                                                    Info: Applying configuration version '(e005d5552d) root - [WIP] arclamp: serve SVGs, compressed logs from Swift'                                                                                                   Notice: The LDAP client stack for this host is: sssd/sudo                                                                                                                                                          Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'                                                       Notice: /Stage[main]/Varnish::Logging/Exec[mask_default_mtail]/returns: executed successfully                                                                                                                      Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Mtail/Systemd::Service[mtail]/Service[mtail]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Mtail/Systemd::Service[mtail]/Service[mtail]: Unscheduling refresh on Service[mtail]
Notice: /Stage[main]/Confd/Base::Service_unit[confd]/Service[confd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Confd/Base::Service_unit[confd]/Service[confd]: Unscheduling refresh on Service[confd]
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Command failed with error code 106
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Message from VCC-compiler:
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: No backends or directors found in VCL program, at least one is necessary.
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Running VCC-compiler failed, exited with 2
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: VCL compilation failed
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-17c9be31-f60d-426$-b624-9a3e09869e44 /etc/varnish/wikimedia_misc-frontend.vcl"
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Traceback (most recent call last):                                         Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 179, in <module>
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     main()
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 138, in main
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     separate_vcl_ids = [load(vadm_cmd, vcl_file) for vcl_file in args.sepa$ate_vcl_files]
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 138, in <listcomp>
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     separate_vcl_ids = [load(vadm_cmd, vcl_file) for vcl_file in args.sepa$ate_vcl_files]
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 123, in load
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     do_cmd(vcl_load_cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 63, in do_cmd
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     subprocess.check_call(cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     raise CalledProcessError(retcode, cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'fro$tend', 'vcl.load', 'vcl-17c9be31-f60d-4260-b624-9a3e09869e44', '/etc/varnish/wikimedia_misc-frontend.vcl']' returned non-zero exit status 1.
Error: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_text-frontend.vcl -d 2 -a -s /etc/varnish/wikimedia_misc-frontend.vcl && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 inste$d of one of [0]
Error: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/reload-vcl -n fronte$d -f /etc/varnish/wikimedia_text-frontend.vcl -d 2 -a -s /etc/varnish/wikimedia_misc-frontend.vcl && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0]
Info: Stage[main]: Unscheduling all events on Stage[main]                                                                                                                                                          Notice: Applied catalog in 15.49 seconds
urbanecm@deployment-cache-text06:/etc/varnish$

I'm not sure what to do with this one, so I reverted my hiera changes.

Majavah added a subscriber: Majavah.Sep 14 2020, 1:39 PM

see also T257968: Certificate for *.beta.wmflabs.org has expired (July 2020)

Urbanecm added a comment.Sep 14 2020, 1:44 PM

Something similar to the second error from T262816#6458824 happens at cache-upload too:

urbanecm@deployment-cache-upload06:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-cache-upload06.deployment-prep.eqiad.wmflabs
Info: Applying configuration version '(563d2323a2) root - [WIP] arclamp: serve SVGs, compressed logs from Swift'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: /Stage[main]/Varnish::Logging/Exec[mask_default_mtail]/returns: executed successfully
Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter]
Notice: /Stage[main]/Mtail/Systemd::Service[mtail]/Service[mtail]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Confd/Base::Service_unit[confd]/Service[confd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Confd/Base::Service_unit[confd]/Service[confd]: Unscheduling refresh on Service[confd]
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Command failed with error code 106
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Message from VCC-compiler:
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: No backends or directors found in VCL program, at least one is necessary.Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Running VCC-compiler failed, exited with 2
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: VCL compilation failed
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-a25d593e-7bc8-4478-a9b3-7aaf94c96fad /etc/varnish/wikimedia_upload-frontend.vcl"
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 179, in <module>
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     main()
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 162, in main
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     main_vcl_id = load(vadm_cmd, args.vcl_file)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 123, in load
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     do_cmd(vcl_load_cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 63, in do_cmd
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     subprocess.check_call(cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     raise CalledProcessError(retcode, cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-a25d593e-7bc8-4478-a9b3-7aaf94c96fad', '/etc/varnish/wikimedia_upload-frontend.vcl']' returned non-zero exit status 1.
Error: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_upload-frontend.vcl -d 2 -a && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_upload-frontend.vcl -d 2 -a && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 14.76 seconds
urbanecm@deployment-cache-upload06:~$
Urbanecm added a comment.EditedSep 14 2020, 1:53 PM

Following T257968#6306273, I did the cert-reload, and both en.wikipedia.beta.wmflabs.org and upload.beta.wmflabs.org seems to work. We should still fix the puppet errors quoted above.

Urbanecm lowered the priority of this task from Unbreak Now! to High.Sep 15 2020, 1:45 AM
Urbanecm added a subscriber: • ema.

Not UBN anymore, but still high. @Krenair could you look at the puppet failure? Maybe @ema can also help, as it should be similar to prod's set up.

zeljkofilipin moved this task from Waiting... 🐢 to Watching 🕵️‍♂️ on the User-zeljkofilipin board.Sep 30 2020, 2:31 PM
Mainframe98 removed a subscriber: Mainframe98.Sep 30 2020, 2:35 PM
zeljkofilipin removed a project: User-zeljkofilipin.Dec 22 2020, 7:04 PM
Reedy closed this task as Resolved.Mar 27 2021, 12:38 AM
Reedy added a subscriber: Reedy.

I'm guessing this didn't regress...

AlexisJazz added a parent task: T293585: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy.Oct 17 2021, 10:04 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL