Page MenuHomePhabricator

Extend NEL headers to sites not fronted by CDN
Open, LowPublic

Description

We have a bunch of sites that are not fronted by the CDN. This also means they generally only live in eqiad and/or codfw, which also means that users in EMEA/Asia/etc take a totally different path to these sites than they do to the CDN.

Several times a year, some SREs and other technical contributors report being unable to reach sites like Gerrit, GitLab, lists.wm.o, integration.wm.o, icinga.wm.o, etc.

Let's make these sites also serve the same NEL headers as production.

Probably the best way to do this is to define some top-level hiera with a map of names and values for headers (we need to serve both NEL: and Report-To:) and then modify other Puppet code to reference this hiera as needed (whether Apache or Nginx config snippets).

We can also make the current gross mess that is in wikimedia-frontend.vcl.erb be generated from this hiera as well.

It's been almost five years since we deployed NEL on all CDN domains, and almost three years since this task was filed. In that time we've not changed the NEL configuration once. Copying and pasting is fine actually :)

To add NEL to your service

Set response headers like the following:

Report-To: {"group": "wm_nel", "max_age": 604800, "endpoints": [{"url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0"}]}
NEL: {"report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}

You may wish to increase failure_fraction for low-traffic services, but 0.05 should always be a safe starting point even for high-traffic services: it's what we use on-wiki.

  • gerrit
  • lists
  • integration
  • icinga
  • everything on WMCS / toolserver
  • gitlab

Event Timeline

As a note, such sites also include "everything on WMCS / toolserver" and it would probably be good to extend NEL to that as well.

CDanis updated the task description. (Show Details)

Change #1175552 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: add NEL headers to apache

https://gerrit.wikimedia.org/r/1175552

Change #1175552 merged by Dzahn:

[operations/puppet@production] gerrit: add NEL headers to apache

https://gerrit.wikimedia.org/r/1175552

@CDanis Deployed on Gerrit. It should be sending the headers now. I added some check boxes for other services to the ticket description. Should some more be added?

Change #1177452 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] icinga: add NEL headers to httpd config

https://gerrit.wikimedia.org/r/1177452

Change #1177455 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] lists: add NEL headers to apache

https://gerrit.wikimedia.org/r/1177455

Change #1177456 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint/integration: add NEL headers to apache

https://gerrit.wikimedia.org/r/1177456

Change #1177452 merged by Dzahn:

[operations/puppet@production] icinga: add NEL headers to httpd config

https://gerrit.wikimedia.org/r/1177452

Change #1177455 merged by Dzahn:

[operations/puppet@production] lists: add NEL headers to apache

https://gerrit.wikimedia.org/r/1177455

Change #1178032 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] lists: add NEL headers to apache.conf.epp template

https://gerrit.wikimedia.org/r/1178032

Change #1177456 merged by Dzahn:

[operations/puppet@production] contint/integration: add NEL headers to apache

https://gerrit.wikimedia.org/r/1177456

deployed on Icinga and integration

Change #1178489 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::proxy: Collect network error reports

https://gerrit.wikimedia.org/r/1178489

Change #1178032 merged by Dzahn:

[operations/puppet@production] lists: add NEL headers to apache.conf.epp template

https://gerrit.wikimedia.org/r/1178032

Assuming the subtask T400994 being resolved means the checkbox "everything on WMCS/toolserver" here can be checked as well.

That would leave only GitLab which isn't as straight-forwarded as the other services because the webserver is nginx and it comes from the gitlab-omnibus upstream bundle. (nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx)

There might be ways to inject some config but we would have to look into that some more.

But also.. gerrit/gitlab/lists will likely move behind CDN and then they would be out of scope for this ticket.

BTW, after looking at a few weeks of data, I suggest increasing the failure sampling fraction for these services. 10% or 20% would be absolutely fine, given their usage, and would give much more signal.

@CDanis & @Dzahn thank you very much for adding the NEL. That has proven helpful to investigate an issue we had with accessing Gerrit. 🎉

Change #1240357 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: increase failure_fraction for NEL to 20%

https://gerrit.wikimedia.org/r/1240357

BTW, after looking at a few weeks of data, I suggest increasing the failure sampling fraction for these services. 10% or 20% would be absolutely fine, given their usage, and would give much more signal.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1240357

Change #1240357 abandoned by Dzahn:

[operations/puppet@production] gerrit: increase failure_fraction for NEL to 20%

Reason:

per comments above

https://gerrit.wikimedia.org/r/1240357

Since this is about sites NOT fronted by the CDN - I think we reject doing it for Gitlab. Because it's not as easy to do like for other services and we will just also move it behind the CDN as well.

LSobanski subscribed.

No longer applicable for GitLab, untagging Collab.