Page MenuHomePhabricator
Create Task
Maniphest T89323

Monitor the up-to-date status of wikitech-static
Closed, ResolvedPublic
Actions

Description

Occasionally one of the jobs that updates wikitech-static breaks, and the static site gets out of sync with the actual wikitech. We need to detect and report this, somehow -- often it stays broken for weeks before I notice.

I can think of a few solutions but nothing that I love.

For instance, a cron on the wikitech host could make a daily edit that adds a date stamp to a pre-arranged page, and then shinken could check that same page on wikitech-static and confirm that it contains yesterday's date. Is there a better way?

Details

SubjectRepoBranchLines +/-
wikitech-static monitoring,fix check command setupoperations/puppetproduction+8 -0
wikitech-static monitoring, install pluginoperations/puppetproduction+6 -0
wikitech-static monitoring, add check commandoperations/puppetproduction+4 -0
wikitech-static monitoring, ? is illegal characteroperations/puppetproduction+1 -1
wikitech: add monitoring::service for static syncoperations/puppetproduction+9 -3
nagios plugin checks if wikitech-static is in syncoperations/puppetproduction+40 -0
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.Feb 12 2015, 1:35 AM
Andrew raised the priority of this task from to Medium.
Andrew updated the task description. (Show Details)
Andrew added projects: acl*sre-team, observability.
Andrew subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2015, 1:35 AM
Andrew set Security to None.Feb 12 2015, 1:58 AM
Andrew added a subscriber: yuvipanda.
Dzahn subscribed.Feb 12 2015, 3:00 AM

maybe just check the mtime of one of the files in the filesystem, to make sure files have been written on the remote host without having to go through actual wiki. there's check_file_age on neon in the nagios plugin dir.

or, does the job write logs? could also use check_log to detect an error pattern there and be notified when the job fails

Liuxinyu970226 subscribed.Feb 12 2015, 3:11 AM
fgiunchedi subscribed.Feb 12 2015, 9:53 AM

how does wikitech-static generation work? is there a job on wikitech that generates a static copy to be pushed to an external host or it gets pulled from the external host?

Dzahn mentioned this in T87202: wikitech-static is out of date.Feb 14 2015, 12:57 AM
Andrew added a comment.Feb 22 2015, 1:38 AM

@fgiunchedi a daily cron runs on wikitech which wikitech-static fetches. Details here: https://wikitech.wikimedia.org/wiki/Wikitech-static

Andrew added a comment.Feb 22 2015, 1:39 AM

I propose that we have a monitoring test which uses api calls to compare the most recent edit date of

https://wikitech.wikimedia.org/wiki/Server_Admin_Log

with

https://wikitech-static.wikimedia.org/wiki/Server_Admin_Log

And alerts if they are more than 25 hours different. That's easy, I think?

fgiunchedi added a comment.Feb 23 2015, 12:48 PM

ack, thanks! yep comparing the two sound easy enough

Dzahn added a comment.May 7 2015, 12:31 AM

https://wikitech-static.wikimedia.org/w/api.php?action=ask&query=[[Modification%20date::%2B]]|%3FModification%20date|sort%3DModification%20date|order%3Ddesc

https://wikitech.wikimedia.org/w/api.php?action=ask&query=[[Modification%20date::%2B]]|%3FModification%20date|sort%3DModification%20date|order%3Ddesc

Dzahn added a comment.EditedMay 7 2015, 2:45 AM
In T89323#1057032, @Andrew wrote:

compare the most recent edit date
And alerts if they are more than 25 hours different

#!/bin/bash

API_QUERY="action=query&titles=Server_Admin_Log&list=recentchanges&format=xml"
WIKITECH="https://wikitech.wikimedia.org/w/api.php"
WIKITECHSTATIC="https://wikitech-static.wikimedia.org/w/api.php"

TS_W=$(curl $WIKITECH/?$API_QUERY 2> /dev/null | grep -o "timestamp.*" | cut -d\" -f2)
TS_S=$(curl $WIKITECHSTATIC/?$API_QUERY 2> /dev/null | grep -o "timestamp.*" | cut -d\" -f2)

echo "W: $TS_W"
echo "S: $TS_S"

W: 2015-05-07T02:43:26Z
S: 2015-04-30T01:18:12Z

gerritbot subscribed.May 13 2015, 1:56 AM

Change 210637 had a related patch set uploaded (by Dzahn):
nagios plugin checks if wikitech-static is in sync

https://gerrit.wikimedia.org/r/210637

gerritbot added a project: Patch-For-Review.May 13 2015, 1:56 AM
gerritbot added a comment.May 13 2015, 2:02 AM

Change 210637 merged by Dzahn:
nagios plugin checks if wikitech-static is in sync

https://gerrit.wikimedia.org/r/210637

Dzahn mentioned this in rOPUPa383a6e09bb7: nagios plugin checks if wikitech-static is in sync.May 13 2015, 2:03 AM
gerritbot added a comment.May 13 2015, 2:12 AM

Change 210638 had a related patch set uploaded (by Dzahn):
wikitech: add monitoring::service for static sync

https://gerrit.wikimedia.org/r/210638

Dzahn claimed this task.May 13 2015, 2:22 AM
gerritbot added a comment.May 13 2015, 2:53 PM

Change 210638 merged by Andrew Bogott:
wikitech: add monitoring::service for static sync

https://gerrit.wikimedia.org/r/210638

Andrew mentioned this in rOPUPf4ab0d1f71bd: wikitech: add monitoring::service for static sync.May 13 2015, 2:53 PM
Andrew added a comment.May 13 2015, 2:58 PM

I guess now we should break it on purpose to test the test?

gerritbot added a comment.May 13 2015, 5:54 PM

Change 210721 had a related patch set uploaded (by Dzahn):
wikitech-static monitoring, ? is illegal character

https://gerrit.wikimedia.org/r/210721

gerritbot added a comment.May 13 2015, 5:55 PM

Change 210721 merged by Dzahn:
wikitech-static monitoring, ? is illegal character

https://gerrit.wikimedia.org/r/210721

Dzahn mentioned this in rOPUP8d43e6f1a22b: wikitech-static monitoring, ? is illegal character.May 13 2015, 5:55 PM
gerritbot added a comment.May 13 2015, 5:59 PM

Change 210724 had a related patch set uploaded (by Dzahn):
wikitech-static monitoring, add check command

https://gerrit.wikimedia.org/r/210724

gerritbot added a comment.May 13 2015, 6:11 PM

Change 210730 had a related patch set uploaded (by Dzahn):
wikitech-static monitoring, install plugin

https://gerrit.wikimedia.org/r/210730

gerritbot added a comment.May 13 2015, 6:11 PM

Change 210724 merged by Dzahn:
wikitech-static monitoring, add check command

https://gerrit.wikimedia.org/r/210724

Dzahn mentioned this in rOPUPa81b80a102ed: wikitech-static monitoring, add check command.May 13 2015, 6:11 PM
gerritbot added a comment.May 13 2015, 6:12 PM

Change 210730 merged by Dzahn:
wikitech-static monitoring, install plugin

https://gerrit.wikimedia.org/r/210730

Dzahn mentioned this in rOPUP82ab8b8ab23d: wikitech-static monitoring, install plugin.May 13 2015, 6:13 PM
gerritbot added a comment.May 13 2015, 6:24 PM

Change 210739 had a related patch set uploaded (by Dzahn):
wikitech-static monitoring, fix template name

https://gerrit.wikimedia.org/r/210739

gerritbot added a comment.May 13 2015, 6:29 PM

Change 210739 merged by Dzahn:
wikitech-static monitoring,fix check command setup

https://gerrit.wikimedia.org/r/210739

Dzahn mentioned this in rOPUPb45abe6ee8fb: wikitech-static monitoring,fix check command setup.May 13 2015, 6:30 PM
Dzahn closed this task as Resolved.May 13 2015, 6:45 PM

it works now:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=silver&service=are+wikitech+and+wt-static+in+sync

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=silver&nostatusheader

Liuxinyu970226 unsubscribed.May 13 2015, 10:40 PM
Dzahn mentioned this in T100485: wikitech-static CRIT - wikitech and wikitech-static out of sync (94951s > 90000s).May 27 2015, 5:13 AM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL