Page MenuHomePhabricator
Create Task
Maniphest T99017

Orphan slayer reads from frack Redis rather than ActiveMQ
Closed, ResolvedPublic4 Estimated Story Points
Actions

Description

Prerequisites:

  • Store limbo message IDs into buckets so we can pull arbitrarily. Implement in php-queue. Don't hit the 1MB default size limit, store the bucket in chunks if necessary.
  • Deploy, so we're filling this bucket.
  • Set limbo message expiry to a lower value, like 1 hour.

This task:

  • Finish porting the orphan slayer.

Details

SubjectRepoBranchLines +/-
WIP tests for the orphan rectifiermediawiki/extensions/DonationInterfacemaster+83 -0
Message helpers can handle both transaction or message-keyed source datamediawiki/extensions/DonationInterfacemaster+20 -7
Orphan slayer reads from frack Redismediawiki/extensions/DonationInterfacemaster+128 -51
Customize query in gerrit

Related Objects
Search...

Event Timeline

awight created this task.May 13 2015, 9:35 PM
awight claimed this task.
awight raised the priority of this task from to Medium.
awight updated the task description. (Show Details)
awight added projects: Fundraising Sprint James Brown, MediaWiki-extensions-DonationInterface, Fundraising-Backlog-Old, Fundraising Sprint Kraftwerk.
awight added subscribers: AndyRussG, • atgo, gerritbot and 2 others.
• atgo removed a project: Fundraising Sprint James Brown.May 13 2015, 9:42 PM
• atgo set Security to None.
awight edited a custom field.May 13 2015, 11:11 PM
awight updated the task description. (Show Details)May 14 2015, 9:34 PM
awight moved this task from Backlog to Doing on the Fundraising Sprint Kraftwerk board.May 14 2015, 10:47 PM
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)May 14 2015, 10:52 PM
awight updated the task description. (Show Details)May 14 2015, 10:54 PM
gerritbot added a comment.May 14 2015, 11:33 PM

Change 211062 had a related patch set uploaded (by Awight):
Orphan slayer reads from Memcache

https://gerrit.wikimedia.org/r/211062

gerritbot added a project: Patch-For-Review.May 14 2015, 11:33 PM
awight moved this task from Doing to Review on the Fundraising Sprint Kraftwerk board.May 14 2015, 11:38 PM
awight added a comment.May 14 2015, 11:50 PM

Successful local tests, for status 25 and 600, and that time buffer works.

gerritbot added a comment.May 15 2015, 12:11 AM

Change 211065 had a related patch set uploaded (by Awight):
WIP tests for the orphan rectifier

https://gerrit.wikimedia.org/r/211065

awight moved this task from Review to Doing on the Fundraising Sprint Kraftwerk board.May 22 2015, 12:49 AM
awight added a project: Fundraising Sprint Lou Reed.May 27 2015, 8:42 PM
awight moved this task from Backlog to Doing on the Fundraising Sprint Lou Reed board.May 27 2015, 8:47 PM
awight moved this task from Doing to Review on the Fundraising Sprint Lou Reed board.May 30 2015, 12:02 AM
awight renamed this task from Orphan slayer reads from Memcache rather than ActiveMQ to Orphan slayer reads from frack Redis rather than ActiveMQ.Jun 5 2015, 8:17 PM
awight added a parent task: T99018: Stop writing to ActiveMQ once orphan slaying on Redis has been tested .Jun 8 2015, 6:46 PM
gerritbot added a comment.Jun 8 2015, 8:26 PM

Change 211062 merged by jenkins-bot:
Orphan slayer reads from frack Redis

https://gerrit.wikimedia.org/r/211062

Diffusion mentioned this in rMEXT29a00de85926: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Jun 8 2015, 8:26 PM
awight mentioned this in rEDOI791d581a826c: Orphan slayer reads from frack Redis.Jun 8 2015, 8:33 PM
awight moved this task from Review to Pending Deployment on the Fundraising Sprint Lou Reed board.Jun 10 2015, 7:03 PM
awight added a project: Fundraising Sprint Miles Davis.Jun 10 2015, 11:12 PM
awight moved this task from Backlog to Pending Deployment on the Fundraising Sprint Miles Davis board.Jun 10 2015, 11:18 PM
awight moved this task from Backlog to PCI on the Fundraising-Backlog-Old board.Jun 23 2015, 1:09 AM
• DStrine added a project: Fundraising Sprint N*E*R*D.Jun 24 2015, 11:05 PM
• DStrine moved this task from Backlog to Pending Deployment on the Fundraising Sprint N*E*R*D board.
• atgo added a project: Fundraising-Backlog.Jun 25 2015, 4:36 PM
• atgo moved this task from Triage to Closed Sprint Work Q1 1516 on the Fundraising-Backlog board.
awight removed a project: Fundraising Sprint N*E*R*D.Jun 26 2015, 7:26 PM
• atgo moved this task from Closed Sprint Work Q1 1516 to Q3 2021-2022 on the Fundraising-Backlog board.Jun 30 2015, 10:39 PM
awight moved this task from Q3 2021-2022 to Sprint +3 on the Fundraising-Backlog board.Jul 22 2015, 7:53 PM
awight moved this task from Sprint +3 to Closed Sprint Work Q1 1516 on the Fundraising-Backlog board.
• DStrine added a project: Fundraising Sprint The Pogues.Jul 22 2015, 8:02 PM
awight moved this task from Backlog to Doing on the Fundraising Sprint The Pogues board.Jul 22 2015, 8:03 PM
awight moved this task from Doing to Backlog on the Fundraising Sprint The Pogues board.Jul 22 2015, 8:20 PM
awight closed subtask T99152: Provide access to limbo messages without knowing keys as Resolved.Jul 24 2015, 10:52 PM
gerritbot added a comment.Jul 25 2015, 1:43 AM

Change 226947 had a related patch set uploaded (by Awight):
WIP Orphan slayer reads from frack Redis

https://gerrit.wikimedia.org/r/226947

awight moved this task from Backlog to Doing on the Fundraising Sprint The Pogues board.Jul 25 2015, 1:54 AM
awight moved this task from Doing to Review on the Fundraising Sprint The Pogues board.Jul 29 2015, 10:14 PM
awight closed subtask T106906: Populate an hour of limbo queue in the new Redis store as Resolved.Aug 3 2015, 7:16 PM
Ejegg mentioned this in rEDOIdd047eededa4: Orphan slayer reads from frack Redis.Aug 3 2015, 7:34 PM
awight mentioned this in rMEXTf170bb1666a3: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 3 2015, 7:34 PM
awight moved this task from Review to Done on the Fundraising Sprint The Pogues board.Aug 3 2015, 8:55 PM
awight moved this task from Done to Doing on the Fundraising Sprint The Pogues board.Aug 3 2015, 10:42 PM
awight edited a custom field.
gerritbot added a comment.Aug 3 2015, 11:42 PM

Change 229027 had a related patch set uploaded (by Awight):
Message helpers can handle both transaction or message-keyed source data

https://gerrit.wikimedia.org/r/229027

awight added a comment.Aug 3 2015, 11:43 PM

This was deployed for a while today, but there was at least one blocker: it looks like we corrupt messages when we put them back on the queue. This happens when we pop a message that is too new (under 20 minutes), then push it back on.

Currently, this code is not deployed.

gerritbot added a comment.Aug 3 2015, 11:48 PM

Change 229027 merged by jenkins-bot:
Message helpers can handle both transaction or message-keyed source data

https://gerrit.wikimedia.org/r/229027

awight mentioned this in rMEXT13a367d48a7e: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 3 2015, 11:48 PM
awight mentioned this in rEDOI9e455d144ab5: Message helpers can handle both transaction or message-keyed source data.
awight mentioned this in rMW847d91fb404b: Updated mediawiki/core Project: mediawiki/extensions/DonationInterface….Aug 3 2015, 11:59 PM
awight mentioned this in rMW5fe9824ecc0c: Updated mediawiki/core Project: mediawiki/extensions/DonationInterface….
awight mentioned this in rEDOIb673eb4f9099: Orphan slayer reads from frack Redis.
awight closed this task as Resolved.Aug 4 2015, 12:12 AM
awight moved this task from Doing to Done on the Fundraising Sprint The Pogues board.

Deployed.

awight reopened this task as Open.Aug 4 2015, 6:12 AM

On second thought, let me keep this open until all the little fires burn out.

awight closed subtask T107848: Spike: Orphan logline conspicuously missing as Resolved.Aug 4 2015, 11:45 PM
• DStrine closed this task as Resolved.Aug 5 2015, 10:04 PM
awight added a parent task: Restricted Task.Aug 13 2015, 10:31 PM
• DStrine edited a custom field.Aug 18 2015, 3:37 PM
• DStrine edited a custom field.Aug 18 2015, 6:49 PM
awight closed subtask T107852: Fix cheesiness: be more transactional when popping messages from the limbo queue as Resolved.Aug 18 2015, 9:27 PM
• DStrine added a project: OKR-Work.Aug 25 2015, 9:24 PM
awight closed subtask T107958: Reduce cheesiness: Abort orphan slayer job when we hit a total elapsed time like 4 minutes as Invalid.Sep 15 2015, 5:43 PM
• mmodell removed a subscriber: awight.Jun 22 2017, 9:38 PM
gerritbot added a comment.Sep 5 2017, 10:38 PM

Change 211065 abandoned by Eileen:
WIP tests for the orphan rectifier

Reason:
old, & unloved

https://gerrit.wikimedia.org/r/211065

• DStrine edited projects, added Fr-tech-archived-from-FY-2015/16; removed Fundraising-Backlog.Jan 11 2018, 9:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL