Have a look at bug 45007 comment 4

reedy@terbium:/home/mwdeploy/updateSpecialPages$ ls -al s5@15-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 455 Aug 15 01:02 s5@15-DeadendPages.log

reedy@terbium:/home/mwdeploy/updateSpecialPages$ cat s5@15-DeadendPages.log

dewiki

Statistics completed in 10.83s
UnreviewedPages completed in 9.71s

ValidationStatistics completed in 7.51s

wikidatawiki

Statistics 2m completed in 13.16s

reedy@hume:/home/wikipedia/log/norotate$ flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /home/wikipedia/logs/norotate/updateSpecialPages.log 2>&1
reedy@hume:/home/wikipedia/log/norotate$

Change 80560 had a related patch set uploaded by Reedy:
Maintenance scripts should be run as Apache

https://gerrit.wikimedia.org/r/80560

Change 80560 abandoned by Reedy:
Maintenance scripts should be run as Apache

Reason:
user => "apache",

https://gerrit.wikimedia.org/r/80560

Monitor the current manual run via http://noc.wikimedia.org/~reedy/updateSpecialPages.log

So, it's going to be a while before we get to enwikt?

And a lot longer to zhwiki

No patch to review, so changing to NEW.

Created attachment 13170
Log of manual run

It has apparently run to completion on my manual run

Attached:

Apparently, no update by cron ran since 13/08. The last 24/08 run was a manual one (see comments above).

• Bug 45007 has been marked as a duplicate of this bug. ***

Reedy: Do you know what's the plan for this issue?

Also brought up at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Cached_special_pages_not_being_updated

Now the last run is a month ago (10.09)

Please have a look at the script/cronjob. Thanks.

Still not running successfully with cron, it seems.
I'm already afraid of asking for yet another manual run... :)

• Bug 55483 has been marked as a duplicate of this bug. ***

What's the current status of this issue? What is being done currently?

• Bug 55785 has been marked as a duplicate of this bug. ***

• Bug 55785 has been marked as a duplicate of this bug. ***

Change 90117 had a related patch set uploaded by Nemo bis:
Simplify misc::maintenance::update_special_pages a bit

https://gerrit.wikimedia.org/r/90117

• Bug 55910 has been marked as a duplicate of this bug. ***

Is anything being done about this? Reedy?

(In reply to comment #22)

Is anything being done about this? Reedy?

I just had a similar question. Check out comment 20? :-)

I can't tell what that patch is supposed to do, but it looks like some general cleanup, not a fix to this.

Are there relevant logs that someone can look at? It's unclear to me what the problem is.

(In reply to comment #24)

I can't tell what that patch is supposed to do, but it looks like some
general
cleanup, not a fix to this.

I agree. It would be helpful to see where the script stops running.

(In reply to comment #26)

I agree. It would be helpful to see where the script stops running.

Last time I checked, even the aa/ab wikis didn't have any update since September, so it fails in very early stages.

It sounds like a problem with the cron job, because the manuell run completed succesfully, but a automatic run does not start.

Mabye a operator can have a look here.

For the moment a new manually run where nice. Thanks for that.

For enWS it is reported to have started failing at 10 September, which aligns with something in 1.22wmf16 according to
https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap

• Bug 56364 has been marked as a duplicate of this bug. ***

Please run updateSpecialPages.php manually on all wikis or at least dewiki. Thanks!

wikipedia wrote:

Can someone please start to fix this? Thanks.

emailseelsorge wrote:

Special-page-updating is still non-functional.
I cannot tell, if this is a worldwide issue or just affecting dewiki, however dewiki is missing special page updates since September 2013.

Please take a look at the issue.
Thanks!

Sorry that this takes longer - higher priority issues got into the way for the last days.
I assume that Reedy will find time next week again to look into this again.

Please run updateSpecialPages.php manually on all wikis. Thanks!

Greg or Sam: my question in comment 25 is still unanswered.

(In reply to comment #36)

Greg or Sam: my question in comment 25 is still unanswered.

What a shame.

(In reply to comment #25)

Are there relevant logs that someone can look at? It's unclear to me what the
problem is.

Relevant logs for what exactly?

grep is such a useful tool

eedy@ubuntu64-web-esxi:~/git/operations/puppet$ grep -r updateSpecialPages *
files/misc/scripts/update-special-pages: php MWScript.php updateSpecialPages.php $db
manifests/misc/maintenance.pp: command => "flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1",
manifests/misc/maintenance.pp: # Include this to add cron jobs calling updateSpecialPages.php on all clusters.
manifests/misc/maintenance.pp: file { '/home/mwdeploy/updateSpecialPages':
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Ancientpages > /home/mwdeploy/updateSpecialPages/${name}-AncientPages.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Deadendpages > /home/mwdeploy/updateSpecialPages/${name}-DeadendPages.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Mostlinked > /home/mwdeploy/updateSpecialPages/${name}-MostLinked.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Mostrevisions > /home/mwdeploy/updateSpecialPages/${name}-MostRevisions.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Wantedpages > /home/mwdeploy/updateSpecialPages/${name}-WantedPages.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Fewestrevisions > /home/mwdeploy/updateSpecialPages/${name}-FewestRevisions.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Lonelypages > /home/mwdeploy/updateSpecialPages/${name}-LonelyPages.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostcategories > /home/mwdeploy/updateSpecialPages/${name}-MostCategories.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostlinkedcategories > /home/mwdeploy/updateSpecialPages/${name}-MostLinkedCategories.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostlinkedtemplates > /home/mwdeploy/updateSpecialPages/${name}-MostLinkedTemplates.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Uncategorizedcategories > /home/mwdeploy/updateSpecialPages/${name}-UncategorizedCategories.log 2>&1",
manifests/misc/maintenance.pp: command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Wantedtemplates > /home/mwdeploy/updateSpecialPages/${name}-WantedTemplates.log 2>&1",
reedy@ubuntu64-web-esxi:~/git/operations/puppet$

reedy@terbium:/home/mwdeploy/updateSpecialPages$ ls -al
total 1072
drwxrwxr-x 2 mwdeploy mwdeploy 4096 Oct 21 01:00 .
drwxrwxr-x 4 mwdeploy mwdeploy 4096 Jul 30 21:38 ..
-rw-rw-r-- 1 mwdeploy mwdeploy 183 Oct 11 05:02 s1@11-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 183 Oct 11 04:56 s1@11-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 11 22:23 s1@11-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 12 00:59 s1@11-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 184 Oct 11 10:54 s1@11-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 12 04:54 s1@11-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3075 Oct 12 04:17 s2@12-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3050 Oct 12 03:19 s2@12-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3084 Oct 12 06:43 s2@12-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3090 Oct 12 11:25 s2@12-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3082 Oct 12 06:37 s2@12-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 3082 Oct 12 06:29 s2@12-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151521 Oct 13 03:27 s3@13-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 150670 Oct 13 02:53 s3@13-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151457 Oct 13 03:52 s3@13-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151677 Oct 13 08:28 s3@13-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151455 Oct 13 03:54 s3@13-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151341 Oct 13 05:36 s3@13-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 14 01:06 s4@14-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 182 Oct 14 01:00 s4@14-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 186 Oct 14 01:11 s4@14-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 189 Oct 14 05:25 s4@14-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 14 01:08 s4@14-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 185 Oct 14 01:53 s4@14-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 369 Oct 15 02:46 s5@15-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 360 Oct 15 01:28 s5@15-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 370 Oct 15 05:12 s5@15-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 372 Oct 15 04:15 s5@15-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 372 Oct 15 05:11 s5@15-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 369 Oct 15 02:21 s5@15-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 543 Oct 16 02:00 s6@16-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 537 Oct 16 02:29 s6@16-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 544 Oct 16 03:50 s6@16-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 551 Oct 16 06:55 s6@16-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 546 Oct 16 03:47 s6@16-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 546 Oct 16 03:57 s6@16-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2170 Oct 17 02:22 s7@17-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2149 Oct 17 02:07 s7@17-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2172 Oct 17 03:40 s7@17-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2181 Oct 17 08:17 s7@17-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2173 Oct 17 03:40 s7@17-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 2179 Oct 17 05:09 s7@17-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 184 Oct 21 09:22 updatequerypages-enwiki-only-MostLinkedTemplates.log
reedy@terbium:/home/mwdeploy/updateSpecialPages$

(In reply to comment #35)

Please run updateSpecialPages.php manually on all wikis. Thanks!

Comments like that really aren't helping anyone.

From hume
-rw-rw-r-- 1 reedy wikidev 1614045 Sep 11 02:35 updateSpecialPages.log
-rw-rw-r-- 1 apache wikidev 951081 Feb 4 2013 updateSpecialPages-small.log

Does anyone actually know what isn't working? "It doesn't work" or "They're not being run" is no help to anyone

class misc::maintenance::update_special_pages( $enabled = false ) {

cron {
        update_special_pages:
                command => "flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1",
                user => "apache",
                monthday => "*/3",
                hour => 5,
                minute => 0,
                ensure => $enabled ?{
                        true => present,
                        false => absent,
                        default => absent
                };
        update_special_pages_small:
                ensure => absent;
}

file {
        "/usr/local/bin/update-special-pages":
                source => "puppet:///files/misc/scripts/update-special-pages",
                owner => apache,
                group => wikidev,
                mode => 0755,
                ensure => present;
        "/usr/local/bin/update-special-pages-small":
                ensure => absent;
}

}

Enabled on terbium according to site.pp

reedy@terbium:~$ ls -al /var/log/updateSpecialPages.log
ls: cannot access /var/log/updateSpecialPages.log: No such file or directory
reedy@terbium:~$
reedy@terbium:~$ ls -al /var/lock/update-special-pages
-rw-rw-r-- 1 apache apache 0 Apr 16 2013 /var/lock/update-special-pages
reedy@terbium:~$ ls -al /usr/local/bin/update-special-pages
-rwxr-xr-x 1 apache wikidev 196 Mar 26 2013 /usr/local/bin/update-special-pages
reedy@terbium:~$ cat /usr/local/bin/update-special-pages
#!/bin/bash

. /usr/local/lib/mw-deployment-vars.sh

cd "$MW_COMMON/multiversion"
for db in <"$MW_DBLISTS/all.dblist"; do

echo $db
php MWScript.php updateSpecialPages.php $db
echo
echo

done
reedy@terbium:~$

The job is in the crontab of the apache user

Puppet Name: update_special_pages

0 5 */3 * * flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1

The next place would be to check the cron logs, which I cannot do as I cannot read /var/log/syslog

• confirmed this is on terbium in the cronjob of apache
• but that logfile doesn't exist, so didn't look like it runs
• tried with "sudo -u apache .." manually, worked

• debugged a little and touched the logfile manually, let apache user write to it, changed cronjob to "in a minute" (i commented the line actually executing mwscript) and then it worked, as opposed to before, when i could see the cron job starting in syslog but no log appearing

so i say next time it should run now that i changed the logfile permissions, and the root cause would be having to puppetize this file and it's permissions

(In reply to comment #40)

so i say next time it should run now that i changed the logfile permissions,
and the root cause would be having to puppetize this file and it's
permissions

How about /var/lock/update-special-pages , did you change permissions for that too? It's not clear to me why we even need flock (see patch above).

Does anyone actually know what isn't working? "It doesn't work" or "They're
not
being run" is no help to anyone

Check https://pt.wiktionary.org/wiki/Especial:Categorias_pedidas?uselang=en . Last updated "10 September 2013". That's the farthest I can check.

Special pages ran on 8 November and again today.

Confirmed at enWS. Thanks to those involved.

Thanks from en.wikt for getting this done.

Were these two runs special, requiring intervention not part of routine operations?

(In reply to comment #45)

Were these two runs special, requiring intervention not part of routine
operations?

No, it was the automatic cronjob. As said above, Daniel fixed the permissions of some things.

Change 90117 abandoned by Nemo bis:
Simplify misc::maintenance::update_special_pages a bit

Reason:
not worth it

https://gerrit.wikimedia.org/r/90117