Page MenuHomePhabricator

Eliminate symlinks in mediawiki-config (as much as possible)
Open, LowPublic

Description

Steps to reproduce:

  1. edit something that's symlinked (e.g. PrivateSettings.php)
  2. scap
  3. CLI scripts will use the updated code but the web page will not, for hours at least; presumably because the symlink's timestamp did not change so HHVM uses its cached version
  4. touching the symlink will make the webpage update

Observed on the beta cluster.

  • Scap should make sure the last-modified date of the symlink is not older than the last-modified date of the original, to prevent this.
  • It is far from straightforward to
    1. Find the symlinks pointing to any given file
    2. Modify the symlink's mtime
      • This isn't even supported on some platforms
      • The expectation is that you resolve the symlink and take the mtime of the target, not the symlink it's self, so hhvm is doing it wrong.
      • There doesn't seem to be a native python method to modify the timestamp of a symlink
      • The functionality was added to touch only somewhat recently
  • A much more productive thing to work on is eliminating symlinks - many of them are not needed.

Details

Related Gerrit Patches:
operations/puppet : productionMove wiktionary and foundationwiki docroots to standard docroot
operations/mediawiki-config : masterDrop unused docroots
operations/puppet : productionBeta: Moving all docroots to standard-docroot
operations/mediawiki-config : masterScap prep: Clean up everything, fix up StartProfiler symlink mess

Event Timeline

Tgr created this task.Feb 9 2016, 4:15 AM
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added a project: Scap.
Tgr added a subscriber: Tgr.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 9 2016, 4:15 AM
bd808 added a subscriber: bd808.Feb 9 2016, 4:26 AM

Doing this for the general case would be computationally expensive. There is no "what links here" functionality in the POSIX filesystem standard. All files in /srv/mediawiki-staging would need to be stated to determine if they were symlinks and then the target files of those files found to be symlinks would need to be stated to compare the mtimes. Is there anything other than wmf-config/PrivateSettings.php that would really benefit from this protection?

Tgr added a comment.Feb 11 2016, 7:16 AM

Is there anything other than wmf-config/PrivateSettings.php that would really benefit from this protection?

Not that I know of.

bd808 added a comment.Feb 11 2016, 4:04 PM

How about instead of a fancy generic mechanism we take the current "touch CommmonSettings.php" code and turn it into something that touches a list of files that can be configured in the scap.cfg file?

dduvall triaged this task as Low priority.Feb 12 2016, 7:40 PM
dduvall set Security to None.
dduvall moved this task from Needs triage to MediaWiki MVP on the Scap board.
mmodell edited projects, added Scap (Scap3-MediaWiki-MVP); removed Scap.

The more I think about it, the more I believe that scap should really be implemented like a build system - or it should call some kind of build system during each deployment. Or T120684: Build a dependency graph resolver for deployment stages and tasks

Is there anything other than wmf-config/PrivateSettings.php that would really benefit from this protection?

Not that I know of.

Yes, wmf-config/StartProfiler.php is also affected. It is symlinked from within individual MediaWiki-php directories. A scap change to it did not work as expected. Even 12 hours after the deployment and manual verification that the files on disk of the app servers were in fact up to date. I eventually suspected HHVM's in-memory compilation cache may've missed the change and did a touch. But that didn't help since I was touching the original file, not the symlink.

Even touching the symlink didn't work at first as one must do touch -h to make touch not follow the symlink, but oh well.

demon raised the priority of this task from Low to High.Apr 25 2017, 6:19 PM

@Jdlrobson this is probably what was causing us trouble during SWAT yesterday.

demon added a subscriber: demon.Sep 18 2017, 11:10 PM

PrivateSettings shouldn't happen anymore. CommonSettings just uses PrivateSettings directly now instead of a silly symlink.

For the StartProfiler example, we could just require it from the docroot version rather than symlinking.

Krinkle removed a subscriber: Krinkle.Sep 19 2017, 6:55 PM
greg added a subscriber: greg.

Adding our Release-Engineering-Team (Kanban) project as we would like to work on this in the coming quarter or two (no promises though, this is not a "goal" only "other hoped for work").

mmodell claimed this task.Oct 2 2017, 4:12 PM
mmodell renamed this task from Scap should touch symlinks when originals are touched to Eliminate symlinks in mediawiki-config (as much as possible).Oct 11 2017, 4:59 PM
mmodell updated the task description. (Show Details)

Change 384898 had a related patch set uploaded (by Chad; owner: Chad):
[operations/mediawiki-config@master] Scap prep: Clean up everything, fix up StartProfiler symlink mess

https://gerrit.wikimedia.org/r/384898

Change 384898 merged by Chad:
[operations/mediawiki-config@master] Scap prep: Clean up everything, fix up StartProfiler symlink mess

https://gerrit.wikimedia.org/r/384898

mmodell reassigned this task from mmodell to demon.Nov 6 2017, 5:32 PM

Chad's the one who's leading the way on this task so I'll assign it properly.

Is this one resolved now? I know you made some progress this quarter.

demon lowered the priority of this task from High to Low.Dec 4 2017, 6:21 PM

Not quite, it's still a WIP there's a long tail.

Change 394203 had a related patch set uploaded (by Chad; owner: Chad):
[operations/puppet@production] Beta: Moving all docroots to standard-docroot

https://gerrit.wikimedia.org/r/394203

Change 402090 had a related patch set uploaded (by Chad; owner: Chad):
[operations/puppet@production] Moving all docroots to standard-docroot

https://gerrit.wikimedia.org/r/402090

Change 402091 had a related patch set uploaded (by Chad; owner: Chad):
[operations/mediawiki-config@master] Drop unused docroots

https://gerrit.wikimedia.org/r/402091

Change 394203 merged by Elukey:
[operations/puppet@production] Beta: Moving all docroots to standard-docroot

https://gerrit.wikimedia.org/r/394203

demon removed demon as the assignee of this task.Jan 31 2018, 12:04 AM

Won't be able to wrap this up soon, kinda busy.

Change 402090 abandoned by Chad:
Move wiktionary and foundationwiki docroots to standard docroot

Reason:
Actually, this isn't necessary. Having symlinks allows us to A) quickly see what document root we're working with (and honestly, we could do something based on the hostname anyway if the goal is to unify all the apache config here) and B) It allows us to back out of standardized docroots /quickly/ if something needs to be one-off again.

https://gerrit.wikimedia.org/r/402090

Change 402091 abandoned by Chad:
Drop unused docroots

Reason:
Per what I said on the other change

https://gerrit.wikimedia.org/r/402091

Change 402090 restored by Chad:
Move wiktionary and foundationwiki docroots to standard docroot

Reason:
Actually, ignore everything I said in the comment. We still want this.

https://gerrit.wikimedia.org/r/402090

Change 402090 abandoned by Chad:
Move wiktionary and foundationwiki docroots to standard docroot

https://gerrit.wikimedia.org/r/402090