update.php broken in Beta
Closed, ResolvedPublic

Mattflaschen-WMF updated the task description. (Show Details)
Mattflaschen-WMF raised the priority of this task from to Needs Triage.
Mattflaschen-WMF added a subscriber: Mattflaschen-WMF.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2015, 10:38 PM
Mattflaschen-WMF triaged this task as High priority.Sep 2 2015, 10:38 PM
DannyH set Security to None.
DannyH edited projects, added WorkType-Maintenance; removed WorkType-NewFunctionality.
bd808 added a subscriber: bd808.
Traceback (most recent call last):
  File "/usr/local/bin/wmf-beta-update-databases.py", line 98, in <module>
    sys.exit(main())
  File "/usr/local/bin/wmf-beta-update-databases.py", line 95, in main
    run_updates(dblist, args.batch)
  File "/usr/local/bin/wmf-beta-update-databases.py", line 51, in run_updates
    do_wait(procs)
  File "/usr/local/bin/wmf-beta-update-databases.py", line 32, in do_wait
    raise Exception("command: ", cmd, "output: ", f.read())
Exception: ('command: ', '/usr/local/bin/mwscript update.php --wiki=enwiki --quick', 'output: ', '')
Build step 'Execute shell' marked build as failure

This means that /usr/local/bin/mwscript update.php --wiki=enwiki --quick ended with a >0 exit code (1). Not sure yet why.

bd808 added a comment.Sep 3 2015, 9:15 PM

Results from manual run on deployment-bastion:

$ sudo -u jenkins-deploy /usr/local/bin/mwscript update.php --wiki=enwiki --quick

Catchable fatal error: Argument 7 passed to Flow\Model\WikiReference::__construct() must be an instance of Title, null given, called in /mnt/srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/WikiReference.php on line 67 and defined in /mnt/srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/WikiReference.php on line 24
Krenair added a subscriber: Krenair.Sep 3 2015, 9:18 PM

Change 236071 had a related patch set uploaded (by Mattflaschen):
Add debugging code for invalid titles in WikiReference

https://gerrit.wikimedia.org/r/236071

Change 236071 merged by jenkins-bot:
Add debugging code for invalid titles in WikiReference

https://gerrit.wikimedia.org/r/236071

2015-09-04 19:26:30 deployment-bastion enwiki Flow INFO: Flow\Model\WikiReference::makeTitle: Invalid title.  Namespace: 100, Title text: Contents/Health_and_fitness {"private":false}
mysql> SELECT workflow_wiki, ref_src_object_id, ref_src_object_type, ref_src_workflow_id, ref_src_namespace, ref_src_title, ref_target_namespace, ref_target_title, ref_type, ref_src_wiki FROM flow_wiki_ref JOIN flow_workflow ON ref_src_workflow_id = workflow_id WHERE ref_target_namespace = 100 AND ref_target_title = 'Contents/Health_and_fitness';                                                                        
+---------------+-------------------+---------------------+---------------------+-------------------+---------------+----------------------+-----------------------------+----------+--------------+               
| workflow_wiki | ref_src_object_id | ref_src_object_type | ref_src_workflow_id | ref_src_namespace | ref_src_title | ref_target_namespace | ref_target_title            | ref_type | ref_src_wiki |               
+---------------+-------------------+---------------------+---------------------+-------------------+---------------+----------------------+-----------------------------+----------+--------------+               
| enwiki        | �n@��!w�            | post                | �븖�!w�              |                 1 | Flow          |                  100 | Contents/Health_and_fitness | link     | enwiki       |            
+---------------+-------------------+---------------------+---------------------+-------------------+---------------+----------------------+-----------------------------+----------+--------------+               
1 row in set (0.00 sec)
> var_export( gettype( Title::makeTitle( 100, 'Contents/Health_and_fitness' ) ) );
'object'
> var_export( Title::makeTitle( 100, 'Contents/Health_and_fitness' )->getArticleId() )
0
> var_export( gettype( Flow\Model\WikiReference::makeTitle( 100, 'Contents/Health_and_fitness' ) ) )
'NULL'

It doesn't exist, but that doesn't explain the error.

Title::makeTitleSafe fails, but Title::makeTitle returns a Title. This is because MWNamespace::exists( 100 ) is false (there is no such namespace on enwiki Beta currently).

Change 236206 had a related patch set uploaded (by Mattflaschen):
Add missing extra namespaces from prod config to labs version

https://gerrit.wikimedia.org/r/236206

It does exist on prod, so that patch should restore the config on Labs so that's a valid title again.

Change 236206 merged by jenkins-bot:
Add missing extra namespaces from prod config to labs version

https://gerrit.wikimedia.org/r/236206

Mattflaschen-WMF added a comment.EditedSep 5 2015, 4:02 AM

This is weird. It switched from red to gray (https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/).

In https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/2692/consoleFull (the last red), it failed on enwiki, which is what led to me fixing https://gerrit.wikimedia.org/r/236206 .

In https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/2693/consoleFull the last output is:

00:20:18 ...Update 'FlowUpdateWorkflowPageId' already logged as completed.
00:20:18 ...Update 'FlowCreBuild timed out (after 45 minutes). Marking the build as aborted.

This sounds like FlowCreateTemplates.php (which is also immediately after FlowUpdateWorkflowPageId.php in Hooks.php).

But then it gets truncated (due to the build timing out?). I think it may actually be another wiki (enwiki?) timing out (probably on FlowFixLinks), not en_rtlwiki, despite the last "Going to run" being en_rtlwiki. They run in parallel.

I just started:

sudo -u jenkins-deploy /usr/local/bin/mwscript update.php --wiki=enwiki --quick

(this time in a screen session).

It successfully completed in the screen session (1 h 27 min 11 s. for the whole update.php run), so it should be marked as completed, and won't run automatically in update.php again.

I also triggered a run through the Jenkins interface, which was successful.

However, it looks like my manual shell run was running at the same time as the hourly Jenkins update.php one. So it may be in an inconsistent state.

I'm re-running it now in shell with force. It shouldn't conflict this time because it's marked as completed so update.php skips it:

sudo -u jenkins-deploy /usr/local/bin/mwscript extensions/Flow/maintenance/FlowFixLinks.php --wiki=enwiki --force
In T111267#1611382, @Mattflaschen wrote:

I'm re-running it now in shell with force.

Finished successfully.

DannyH closed this task as Resolved.Sep 8 2015, 5:32 PM
DannyH added a subscriber: DannyH.