Page MenuHomePhabricator
Create Task
Maniphest T104581

Protect background jobs from unhandled exceptions
Closed, ResolvedPublic
Actions

Description

Each time there is an update or insert in the storage back-end, we fire off background-update jobs for revision policy enforcement and secondary-indices updates. However, these asynchronous jobs are executed in parallel with the normal request execution pipeline, which means they are left unmonitored and unprotected. We need to make sure they are bullet-proof and that their execution cannot get interrupted by unhandled (and possibly synchronously-thrown) exceptions as that might leave the process hanging.

Details

SubjectRepoBranchLines +/-
Update restbase to 03ed384mediawiki/services/restbase/deploymaster+187 -124
Customize query in gerrit

Related Objects

Event Timeline

• mobrovac created this task.Jul 2 2015, 3:53 PM
• mobrovac raised the priority of this task from to High.
• mobrovac updated the task description. (Show Details)
• mobrovac added projects: RESTBase, RESTBase-Cassandra, Services.
• mobrovac subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2015, 3:53 PM
• mobrovac updated the task description. (Show Details)Jul 2 2015, 3:55 PM
• mobrovac set Security to None.
mark added a project: acl*sre-team.Jul 3 2015, 10:18 AM
mark subscribed.
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 3 2015, 10:18 AM
• mobrovac added a project: Patch-For-Review.Jul 3 2015, 12:03 PM

The PR for increasing robustness is here.

• mobrovac added a comment.Jul 3 2015, 12:05 PM

Since it is friday and the background update jobs keep leaving RESTBase processes in a comatose state, I am going to redeploy the last-deployed version of RESTBase, and only rebuild the module dependencies, which will pick up the new version.

• mobrovac added a comment.Jul 3 2015, 1:08 PM

This has been deployed @ 12:43 UTC, the system seems stable and functional. So far so good - no unhandled exception errors in local syslogs.

gerritbot subscribed.Jul 7 2015, 2:06 PM

Change 223297 had a related patch set uploaded (by Mobrovac):
Update restbase to 03ed384

https://gerrit.wikimedia.org/r/223297

gerritbot added a comment.Jul 9 2015, 11:13 AM

Change 223297 merged by Mobrovac:
Update restbase to 03ed384

https://gerrit.wikimedia.org/r/223297

• mobrovac mentioned this in rGRBD5d3cbf90ae4f: Update restbase to 03ed384.Jul 9 2015, 11:39 AM
• mobrovac closed this task as Resolved.Jul 10 2015, 4:08 PM
• mobrovac claimed this task.

This has been fixed and confirmed to work in production.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits