fundraising hosts use nsca + icinga, and ganglia for monitoring and reporting
nsca seems to be poorly maintained, currently broken for Trusty hosts
as an org we're talking about moving away from ganglia
this task is to fix and update monitoring to be reliable and better supported going forward
random notes:
It is time to promote Wikimedia-Hackathon-2015 activities in the program (training sessions and meetings) and main wiki page (hacking projects and other ongoing activities). Follow the instructions, please. If you have questions, about this message, ask here.
Did anyone work on this task during Wikimedia-Hackathon-2015? If so, please post an update with the results. Otherwise remove the label.
Looked into why ganglia has stopped reporting for most of the fundraising cluster, determined pfw config is likely blocking multicast, opened T100150.
Spent some time researching monitoring options and spoke to Filippo about his research toward a strategy for production, and specifically re. promethius which sounds promising.
Looked at possibilities for pulling data from ganglia aggregators into promethius. Ganglia's aggregator listens on a tcp socket and responds with an XML report of metric data for all hosts in the aggregated cluster, studied that report format, wrote a simple python XML parser, thinking toward reformatting that data to a format promethius can use.
We are trying to help all open tasks listed under "Work continues after Lyon" at the Wikimedia Hackathon 2015 workboard finding their best way forward. * If you are participating in Wikimania, consider adding the #Wikimania-Hackathon-2015 project to get this task in that loop, which is about to start. * If you think this project could welcome help from a dedicated Google Summer of Code or Outreachy intern, or from an Individual Engagement Grant, add the Possible-Tech-Projects project. * If you would like to receive some other type of support (organizing a Tech Talk, establishing contacts with existing developer teams in Wikimedia or elsewhere, travel sponsorship for a related activity... you name it), please create a subtask explaining your request and associate it with #Engineering-Community (or you can start by commenting here if you prefer). * Keeping the description, priority and assigned fields up to date always helps. :) For some context about this message, see T101151: Evaluate which projects showcased at the Wikimedia Hackathon should be supported further. It is the last communication related to Wikimedia-Hackathon-2015 that we will post here.
Please confirm and promote this activity by assigning it to its owner, listing it or scheduling it at the Hackathon wiki page and by placing it in the right column at #Wikimania-Hackathon-2015. Thank you!