Page MenuHomePhabricator

mediawiki-vagrant fails provisioning (because of missing root cert)
Open, Needs TriagePublicBUG REPORT

Description

List of steps to reproduce (step by step, including full links if applicable):
Following the standard steps: https://www.mediawiki.org/wiki/MediaWiki-Vagrant

What happens?:

==> default: Error: Could not set 'link' on ensure: Protocol error @ rb_file_s_symlink - (/vagrant/mediawiki/COPYING, /vagrant/srv/docroot/w/COPYING)
==> default: Error: /Stage[main]/Npm/Exec[downgrade_npm]: Could not evaluate: Could not find command '/usr/bin/npm'
==> default: Notice: /Stage[main]/Service/File[/etc/mw-vagrant/services]: Dependency Exec[downgrade_npm] has failures: true
==> default: Warning: /Stage[main]/Service/File[/etc/mw-vagrant/services]: Skipping because of failed dependencies

and a long list of everything else failing.

$ vagrant git-update
/usr/bin/xauth:  file /home/vagrant/.Xauthority does not exist
bash: run-git-update: command not found

What should have happened instead?:

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc:

  • Vagrant
  • Virtualbox 6.1.26
  • Windows 10 Pro Version 10.0.19042 Build 19042
  • Most recent mediawiki-vagrant

Event Timeline

Seddon renamed this task from mediawiki-vagrant fails provisioning (missing dependency for downgrade_npm to mediawiki-vagrant fails provisioning (missing dependency for downgrade_npm).Oct 1 2021, 11:45 PM
Seddon updated the task description. (Show Details)
Seddon updated the task description. (Show Details)

Existing boxes are also borked.

$ vagrant git-update
==> Updating arcanist ...
[*] Updating repo in /vagrant/srv/arcanist ...
fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
 ***** GIT PULL failed in /vagrant/srv/arcanist for branch 'wmf/stable'

The proper fix is presumably T256822: [EPIC] Upgrade MediaWiki-Vagrant to Debian Buster.

The quick fix is adding the new root cert to the cert store and removing the old one. The tricky part is doing it early enough that nothing else breaks.

Tgr renamed this task from mediawiki-vagrant fails provisioning (missing dependency for downgrade_npm) to mediawiki-vagrant fails provisioning (because of missing root cert).Oct 2 2021, 9:21 AM

From T283165, the issue happens with:

OpenSSL < 1.1.0
LibreSSL < 3.2.0
GnuTLS < 3.6.14

Full details at https://lists.debian.org/debian-lts/2021/09/msg00008.html

Debian has updated them, may you check the version of libssl? On an up-to-date Stretch I got:

libssl1.0.21.0.2u-1~deb9u6
libssl1.11.1.0l-1~deb9u4
$ dpkg -l libssl*|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version         Architecture Description
+++-=================-===============-============-===============================================
ii  libssl1.0.2:amd64 1.0.2u-1~deb9u6 amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64   1.1.0l-1~deb9u4 amd64        Secure Sockets Layer toolkit - shared libraries

They come from security updates:

$ apt-cache madison libssl*|grep security
libssl1.0.2 | 1.0.2u-1~deb9u6 | http://security.debian.org stretch/updates/main amd64 Packages
libssl1.1 | 1.1.0l-1~deb9u4 | http://security.debian.org stretch/updates/main amd64 Packages

So maybe it is sufficient to update those?

sudo apt update
apt list --upgradable  # to list pending upgrades
apt-get -y install libssl1.0.2 libssl1.1
==> default: Error: Could not set 'link' on ensure: Protocol error @ rb_file_s_symlink - (/vagrant/mediawiki/COPYING, /vagrant/srv/docroot/w/COPYING)

I have no lead about this one :(

vagrant@growth:~$ dpkg -l libssl*|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version         Architecture Description
+++-=================-===============-============-================================================
ii  libssl-dev:amd64  1.1.0l-1~deb9u1 amd64        Secure Sockets Layer toolkit - development files
un  libssl-doc        <none>          <none>       (no description available)
un  libssl1.0-dev     <none>          <none>       (no description available)
ii  libssl1.0.2:amd64 1.0.2u-1~deb9u1 amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64   1.1.0l-1~deb9u1 amd64        Secure Sockets Layer toolkit - shared libraries
$ apt-cache madison libssl*|grep security
==> default: Error: Could not set 'link' on ensure: Protocol error @ rb_file_s_symlink - (/vagrant/mediawiki/COPYING, /vagrant/srv/docroot/w/COPYING)

I have no lead about this one :(

Presumably some secondary error caused by a failed git clone or such.

vagrant@growth:~$ curl -v https://phabricator.wikimedia.org/diffusion/ARC/arcanist.git/
*   Trying 208.80.154.224...
* TCP_NODELAY set
* Connected to phabricator.wikimedia.org (208.80.154.224) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
vagrant@growth:~$ GIT_CURL_VERBOSE=1 git ls-remote https://gerrit.wikimedia.org/r/mediawiki/services/jobrunner.git/
* Couldn't find host gerrit.wikimedia.org in the .netrc file; using defaults
*   Trying 208.80.154.137...
* TCP_NODELAY set
* Connected to gerrit.wikimedia.org (208.80.154.137) port 443 (#0)
* found 151 certificates in /etc/ssl/certs/ca-certificates.crt
* found 606 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_CHACHA20_POLY1305
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/services/jobrunner.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
vagrant@growth:~$ openssl version
OpenSSL 1.1.0l  10 Sep 2019

vagrant@growth:~$ ldd /usr/lib/git-core/git-http-fetch | grep libcurl
	libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f8bc2d04000)

vagrant@growth:~$ curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2u zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3

So OpenSSL 1.1.0 is installed but libcurl is linked with 1.0.2. And git is linked with the gnutls version of libcurl?
The issue is also mentioned in the linked conversation (which says this is not a problem anymore on buster).

Change 725444 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/vagrant@master] Temporary fix for OpenSSL roo CA issue

https://gerrit.wikimedia.org/r/725444

libcurl-gnutls.so.4 links to /usr/lib/x86_64-linux-gnu/libgnutls.so.30 which belongs to .Debian package libgnutls30. Its latest Debian changelog entry does mention Let's Encrypt:

gnutls28 (3.5.8-5+deb9u6) stretch-security; urgency=high

  * Non-maintainer upload by the LTS Security Team.
  * Fix verification error with alternate chains. Closes: #961889
    Addresses issue with Let's Encrypt certificates starting 2021-10-01.
    https://lists.debian.org/debian-lts/2021/09/msg00008.html

 -- Sylvain Beucler <beuc@debian.org>  Sat, 11 Sep 2021 20:07:51 +0200

In my previous comment I solely mentioned libssl. Looks like updating libgnutls30 will solve it:

sudo apt update
sudo apt -y install libgnutls30

Using the new patch

    default: Guest Additions Version: 5.2.24
    default: VirtualBox Version: 6.1
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /vagrant => x:/Users/xxx/vagrant
    default: /vagrant/logs => x:/Users/xxx/vagrant/logs
==> default: Running provisioner: lsb_check...
==> default: Running provisioner: file_perms...
==> default: Running provisioner: shell...
    default: Running: x:/Users/xxx/AppData/Local/Temp/vagrant-shell20211003-15324-186362s.sh
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment vagrant...
==> default: Info: Loading facts
==> default: Notice: Compiled catalog for vagrant.localdomain in environment vagrant in 4.20 seconds
==> default: Info: Applying configuration version '1633301965.no-git'
==> default: Error: /Stage[first]/Mwv::Hack/Exec[disable broken letsencrypt cert]: Could not evaluate: Could not find command 'ack'
==> default: Notice: /Stage[first]/Mwv::Hostname/Exec[set-hostname]/returns: executed successfully
==> default: Notice: /Stage[first]/Mwv::Hostname/Host[vagrant.mediawiki-vagrant.dev]/ensure: created
==> default: Info: Computing checksum on file /etc/hosts
==> default: Info: Stage[first]: Unscheduling all events on Stage[first]
==> default: Notice: /Stage[main]/Apt/File[/etc/apt/.update]: Dependency Exec[disable broken letsencrypt cert] has failures: true
==> default: Warning: /Stage[main]/Apt/File[/etc/apt/.update]: Skipping because of failed dependencies

Change 725444 merged by jenkins-bot:

[mediawiki/vagrant@master] Temporary fix for OpenSSL root CA issue

https://gerrit.wikimedia.org/r/725444