In T247653#6814030, @Krinkle wrote:

I merged https://gerrit.wikimedia.org/r/662785 (T273247) and deployed it with Scap, but https://doc.wikimedia.org/ keeps showing the previous content.

== DEFAULT ==
:* doc1001.eqiad.wmnet
:* contint2001.wikimedia.org
:* contint1001.wikimedia.org

I confirmed on doc1001, that /srv/deployment/integration/docroot contains the commit. and org/wikimedia/doc/opensource.yaml contains the […] change.

But https://doc.wikimedia.org/ and https://doc.wikimedia.org/?random=1234 keep showing outdated content. I've also checked locally on doc1001:

$ krinkle@doc1001:~$ curl 'http://doc1001.eqiad.wmnet' -H 'Host: doc.wikimedia.org' 

And strangely that also shows outdated content. This suggests one of two things to me:

1. […]
2. Or, there is some kind of additional caching locally to this server that I'm not familiar with and that Scap does not know to reload or purging.

In T247653#6817313, @Krinkle wrote:

[…] But where was it cached? Does the local Apache have an HTTP cache proxy that serves stale responses if it gets 500 from PHP?

This is still consistently an issue. E.g. after deploying https://gerrit.wikimedia.org/r/666245, the server keeps producing fresh HTTP responses from Apache with old PHP code. Same as last time, it does not appear to be correcting itself, it just stays there presumably until a root does a hard Apache or php-fpm restart.

Maybe it has something to do with the symlinks used by /srv/deployment? Or maybe opcache is configured on this host to cache forever and never look at file paths?

Tagging SRE and RelEng. I don't know who the first respondent is for this service. It seems that since Dec 2020 (ref T149924) it is basically become impossible to deploy changes to integration.wm.o and doc.wm.o. The only way changes become applies is if a root SRE restarts Apache and/or php-fpm locally on the doc1001 backend.