Change Details

>>! In T247653#6814030, @Krinkle wrote: > > I merged <https://gerrit.wikimedia.org/r/662785> (T273247) and deployed it with Scap, but <https://doc.wikimedia.org/> keeps showing the previous content. > > ``` > == DEFAULT == > :* doc1001.eqiad.wmnet > :* contint2001.wikimedia.org > :* contint1001.wikimedia.org > ``` > > I confirmed on doc1001, that `/srv/deployment/integration/docroot` contains the commit. and `org/wikimedia/doc/opensource.yaml` contains the […] change. > > But <https://doc.wikimedia.org/> and <https://doc.wikimedia.org/?random=1234> keep showing outdated content. I've also checked locally on doc1001: > > ``` > $ krinkle@doc1001:~$ curl 'http://doc1001.eqiad.wmnet' -H 'Host: doc.wikimedia.org' > ``` > > And strangely that also shows outdated content. This suggests one of two things to me: > > 1. […] > 2. Or, there is some kind of additional caching locally to this server that I'm not familiar with and that Scap does not know to reload or purging. ------- >>! In T247653#6817313, @Krinkle wrote: > […] But where was it cached? Does the local Apache have an HTTP cache proxy that serves stale responses if it gets 500 from PHP? This is still consistently an issue. E.g. after deploying <https://gerrit.wikimedia.org/r/666245>, the server keeps producing fresh HTTP responses from Apache with old PHP code. Same as last time, it does not appear to be correcting itself, it just stays there presumably until a root does a hard Apache or php-fpm restart. Maybe it has something to do with the symlinks used by `/srv/deployment`? Or maybe opcache is configured on this host to cache forever and never look at file paths? Tagging SRE and RelEng. I don't know who is first respondent for this service. It seems that since Dec 2020 (ref T149924) it is basically become impossible to deploy changes to integration.wm.o and doc.wm.o. The only way changes become applies is if a root SRE restarts Apache and/or php-fpm locally on the doc1001 backend.

>>! In T247653#6814030, @Krinkle wrote: > > I merged <https://gerrit.wikimedia.org/r/662785> (T273247) and deployed it with Scap, but <https://doc.wikimedia.org/> keeps showing the previous content. > > ``` > == DEFAULT == > :* doc1001.eqiad.wmnet > :* contint2001.wikimedia.org > :* contint1001.wikimedia.org > ``` > > I confirmed on doc1001, that `/srv/deployment/integration/docroot` contains the commit. and `org/wikimedia/doc/opensource.yaml` contains the […] change. > > But <https://doc.wikimedia.org/> and <https://doc.wikimedia.org/?random=1234> keep showing outdated content. I've also checked locally on doc1001: > > ``` > $ krinkle@doc1001:~$ curl 'http://doc1001.eqiad.wmnet' -H 'Host: doc.wikimedia.org' > ``` > > And strangely that also shows outdated content. This suggests one of two things to me: > > 1. […] > 2. Or, there is some kind of additional caching locally to this server that I'm not familiar with and that Scap does not know to reload or purging. ------- >>! In T247653#6817313, @Krinkle wrote: > […] But where was it cached? Does the local Apache have an HTTP cache proxy that serves stale responses if it gets 500 from PHP? This is still consistently an issue. E.g. after deploying <https://gerrit.wikimedia.org/r/666245>, the server keeps producing fresh HTTP responses from Apache with old PHP code. Same as last time, it does not appear to be correcting itself, it just stays there presumably until a root does a hard Apache or php-fpm restart. Maybe it has something to do with the symlinks used by `/srv/deployment`? Or maybe opcache is configured on this host to cache forever and never look at file paths? Tagging SRE and RelEng. I don't know who the first respondent is for this service. It seems that since Dec 2020 (ref T149924) it is basically become impossible to deploy changes to integration.wm.o and doc.wm.o. The only way changes become applies is if a root SRE restarts Apache and/or php-fpm locally on the doc1001 backend.

>>! In T247653#6814030, @Krinkle wrote: > > I merged <https://gerrit.wikimedia.org/r/662785> (T273247) and deployed it with Scap, but <https://doc.wikimedia.org/> keeps showing the previous content. > > ``` > == DEFAULT == > :* doc1001.eqiad.wmnet > :* contint2001.wikimedia.org > :* contint1001.wikimedia.org > ``` > > I confirmed on doc1001, that `/srv/deployment/integration/docroot` contains the commit. and `org/wikimedia/doc/opensource.yaml` contains the […] change. > > But <https://doc.wikimedia.org/> and <https://doc.wikimedia.org/?random=1234> keep showing outdated content. I've also checked locally on doc1001: > > ``` > $ krinkle@doc1001:~$ curl 'http://doc1001.eqiad.wmnet' -H 'Host: doc.wikimedia.org' > ``` > > And strangely that also shows outdated content. This suggests one of two things to me: > > 1. […] > 2. Or, there is some kind of additional caching locally to this server that I'm not familiar with and that Scap does not know to reload or purging. ------- >>! In T247653#6817313, @Krinkle wrote: > […] But where was it cached? Does the local Apache have an HTTP cache proxy that serves stale responses if it gets 500 from PHP? This is still consistently an issue. E.g. after deploying <https://gerrit.wikimedia.org/r/666245>, the server keeps producing fresh HTTP responses from Apache with old PHP code. Same as last time, it does not appear to be correcting itself, it just stays there presumably until a root does a hard Apache or php-fpm restart. Maybe it has something to do with the symlinks used by `/srv/deployment`? Or maybe opcache is configured on this host to cache forever and never look at file paths? Tagging SRE and RelEng. I don't know who isthe first respondent is for this service. It seems that since Dec 2020 (ref T149924) it is basically become impossible to deploy changes to integration.wm.o and doc.wm.o. The only way changes become applies is if a root SRE restarts Apache and/or php-fpm locally on the doc1001 backend.