Page MenuHomePhabricator
Create Task
Maniphest T134916

another round of activemq queue decrufting needed
Closed, ResolvedPublic
Actions

Description

We're over 30% store percent utilization again which is perturbing our early warning alerts, could we do another round of queue cleanup?

Details

SubjectRepoBranchLines +/-
Validate banner history messages before insertwikimedia/fundraising/crmmaster+56 -1
Customize query in gerrit

Event Timeline

Jgreen created this task.May 10 2016, 6:29 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 10 2016, 6:29 PM
• DStrine added a project: Fundraising-Backlog.May 18 2016, 8:24 PM
gerritbot subscribed.May 18 2016, 9:45 PM

Change 289566 had a related patch set uploaded (by Ejegg):
Validate banner history messages before insert

https://gerrit.wikimedia.org/r/289566

gerritbot added a project: Patch-For-Review.May 18 2016, 9:45 PM
Jgreen triaged this task as Unbreak Now! priority.May 19 2016, 12:59 AM
Restricted Application added subscribers: Luke081515, TerraCodes, Urbanecm. · View Herald TranscriptMay 19 2016, 12:59 AM
Jgreen added a comment.May 19 2016, 1:03 AM

Looks like the offending queue is banner-history which has been growing steadily since the weekend and is now up to 150K messages.

ActiveMQ crashed earlier today and from the ramp-up in RAM utilization I think we're headed for another soon. We are going to take banners down until the situation can be fixed.

Jgreen added a comment.May 19 2016, 1:13 AM
In T134916#2307618, @Jgreen wrote:

Looks like the offending queue is banner-history which has been growing steadily since the weekend and is now up to 150K messages.

ActiveMQ crashed earlier today and from the ramp-up in RAM utilization I think we're headed for another soon. We are going to take banners down until the situation can be fixed.

https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Fundraising+eqiad&h=silicon.frack.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=151484&m=ActiveMQ+QueueSize+banner-history&vl=Messages

gerritbot added a comment.May 19 2016, 1:58 AM

Change 289566 merged by Ejegg:
Validate banner history messages before insert

https://gerrit.wikimedia.org/r/289566

• DStrine closed this task as Resolved.May 24 2016, 8:10 PM
• DStrine added a project: FR-ActiveMQ.
• DStrine moved this task from Triage to Current Sprint & Completed in Q4 1516 on the Fundraising-Backlog board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits