AdminCc jeremyb added by jeremyb

Bugzilla ticket 71686 added by jeremyb

from apache2 error.log
[Sun Oct 05 06:25:11 2014] [error] python_init: Python version mismatch,
expected '2.7.2+', fou
nd '2.7.3'.
[Sun Oct 05 06:25:11 2014] [error] python_init: Python executable found
'/usr/bin/python'.
[Sun Oct 05 06:25:11 2014] [error] python_init: Python path being used
'/usr/lib/python2.7/:/us
r/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/pytho
n2.7/lib-dynload'.
[Sun Oct 05 06:25:11 2014] [notice] mod_python: Creating 8 session mutexes
based on 150 max pro
cesses and 0 max threads.
[Sun Oct 05 06:25:11 2014] [notice] mod_python: using mutex_directory /tmp
[Sun Oct 05 06:25:11 2014] [error] Failed to configure CA certificate chain!
[Sun Oct 05 06:33:50 2014] [error] Failed to configure CA certificate chain!
[Sun Oct 05 06:54:02 2014] [error] Failed to configure CA certificate chain!
and the same message repeated again
On Mon Oct 06 00:04:52 2014, <christian at quelltextlich>

[1]

---

<christian at spencer jobs: 0 time: 02:00:02 // exit code: 0
cwd: ~
wget http://stats> failed:
Connection refused.

Status changed from 'new' to 'open' by RT_System

root cause is that the certificate for stats.wikimedia.org is missing a
trailing newline, thus when it gets concatenated the resulting chained file is
invalid
On Mon Oct 06 08:37:25 2014, fgiunchedi wrote:

from apache2 error.log

[Sun Oct 05 06:25:11 2014] [error] python_init: Python version
mismatch,
expected '2.7.2+', fou
nd '2.7.3'.
[Sun Oct 05 06:25:11 2014] [error] python_init: Python executable
found
'/usr/bin/python'.
[Sun Oct 05 06:25:11 2014] [error] python_init: Python path being used
'/usr/lib/python2.7/:/us
r/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-
tk:/usr/lib/python2.7/lib-old:/usr/lib/pytho
n2.7/lib-dynload'.
[Sun Oct 05 06:25:11 2014] [notice] mod_python: Creating 8 session
mutexes
based on 150 max pro
cesses and 0 max threads.
[Sun Oct 05 06:25:11 2014] [notice] mod_python: using mutex_directory
/tmp
[Sun Oct 05 06:25:11 2014] [error] Failed to configure CA certificate
chain!
[Sun Oct 05 06:33:50 2014] [error] Failed to configure CA certificate
chain!
[Sun Oct 05 06:54:02 2014] [error] Failed to configure CA certificate
chain!

and the same message repeated again

On Mon Oct 06 00:04:52 2014, <christian at quelltextlich>

[1]

---

<christian at spencer jobs: 0 time: 02:00:02 // exit code: 0
cwd: ~
wget http://stats> failed:
Connection refused.

On Mon Oct 06 08:46:13 2014, fgiunchedi wrote:

root cause is that the certificate for stats.wikimedia.org is missing
a
trailing newline, thus when it gets concatenated the resulting chained
file is
invalid

fixed by qchris in https://gerrit.wikimedia.org/r/#/c/164914/
the code that concatenates certs should be inserting a newline anyway, I think
an additional newline between certs doesn't cause problems with parsing the
chain

Subject changed from 'Webserver for stats.wikimedia.org and datasets.wikimedia.org not running' to 'more robust certificate chain creation in puppet' by fgiunchedi

Queue changed from ops-requests to core-ops by fgiunchedi

So this would be something like
command => "/bin/cat /etc/ssl/localcerts/${certname}.crt
/etc/ssl/private/${certname}.key > ${location}/${certname}.crt && echo '' >>
${location}/${certname}.crt"
?

On Mon Oct 06 15:04:54 2014, abogott wrote:

So this would be something like

command => "/bin/cat /etc/ssl/localcerts/${certname}.crt
/etc/ssl/private/${certname}.key > ${location}/${certname}.crt && echo '' >>
${location}/${certname}.crt"

?

something to that effect, yes but we should guarantee that the certs are
separated by at least one newline so sth like
cat cert1 > chain ; echo >> chain ; cat cert2 >> chain

57     exec { "${name}_create_chained_cert":
58         creates => "${location}/${certname}.chained.crt",
59         command => "/bin/cat /etc/ssl/localcerts/${certname}.crt ${ca} > ${location}/${certname}.chained.crt",
60         cwd     => '/etc/ssl/certs',

^ Current code. It has just used cat all the time. The problem was that when we add certificates to the repos we had missing newlines. Another way to approach it would be trying to detect that when certificates are first uploaded, say if jenkins would -2 missing newline when files end in .crt

another question. i would have made this ticket public, but i did not for one reason. Christian's mail footer with the private street address, phone numbers, etc. can we just delete that?

@QChris see my comment above. and what you get when you hit "(Show Details)" immediately above this comment next to "Dzahn edited the task description". I would like to make the ticket public but there is your private data and when i edit the ticket there is still the diff in history.

(Why is that an issue in Phab but not on publicly archived mailing lists like wikitech-l or analytics-l?
For the records, Phab admins can hide complete comments and users can hide their own comments.)

In T84543#1091241, @Dzahn wrote:

@QChris [...] I would like to make the ticket public but there
is your private data and when i edit the ticket there is still the diff in history.

Thanks for caring about such things. That's really great!

For me, it's fine to make this ticket public.

It's always ok to publish my emails (including the added street address, phone numbers and some such).

(Stupid local law requires me to add street address and all these things to work-related
emails. „Impressumspflicht” ftw! :-/ )

@QChris thanks for confirming. i made it public.

and here's a related Gerrit change:

https://gerrit.wikimedia.org/r/#/c/194455/

Change 194455 abandoned by Dzahn:
ensure there is always a newline in chained certs

https://gerrit.wikimedia.org/r/194455

I think this old ticket imported from RT times can be resolved. But i would say the authority on this should be @BBlack