We've recovered old coal data in T161538: Plug in ex-graphite2001 SSDs to recover coal data and it is now on graphite2001:~filippo/coal.tar.gz. Next step is to merge that data with the current whisper files in /var/lib/coal on graphite1001 and graphite2001. For that we can use whisper-fill which does exactly that: backfill old data into a new whisper database.
Description
Description
Related Objects
Related Objects
- Mentioned In
- T161538: Plug in ex-graphite2001 SSDs to recover coal data
- Mentioned Here
- T161538: Plug in ex-graphite2001 SSDs to recover coal data
Event Timeline
Comment Actions
I tried out whisper-fill in my home directory on a copy of the coal.loadEventEnd metric and copied it to graphite1001 as coal.merged_loadEventEnd. Looks like it's working fine.
Before:
https://graphite.wikimedia.org/render/?target=movingAverage(coal.loadEventEnd,30)&from=-1year&width=1000&height=600&yMin=0&yMax=2500
https://graphite.wikimedia.org/render/?target=movingAverage(coal.loadEventEnd,30)&from=20170101&to=20170215&width=1000&height=600&yMin=0&yMax=2500
 |  |
After:
https://graphite.wikimedia.org/render/?target=movingAverage(coal.merged_loadEventEnd,30)&from=-1year&width=1000&height=600&yMin=0&yMax=2500
https://graphite.wikimedia.org/render/?target=movingAverage(coal.merged_loadEventEnd,30)&from=20170101&to=20170215&width=1000&height=600&yMin=0&yMax=2500
 |  |
Comment Actions
Starting with merging data on graphite2001.
Backup current data
[22:28 UTC] krinkle at graphite2001.codfw.wmnet in ~ tar -C /var/lib -cvzpf coal-graphite2001-20170424.tar.gz coal/ coal/ coal/domContentLoadedEventStart.wsp coal/redirectEnd.wsp coal/unloadEventStart.wsp coal/secureConnectionStart.wsp coal/domContentLoadedEventEnd.wsp coal/saveTiming.wsp coal/responseStart.wsp coal/redirectStart.wsp coal/redirecting.wsp coal/dnsLookup.wsp coal/connectEnd.wsp coal/mediaWikiLoadEnd.wsp coal/firstPaint.wsp coal/connectStart.wsp coal/mediaWikiLoadStart.wsp coal/loadEventStart.wsp coal/redirectCount.wsp coal/loadEventEnd.wsp coal/domComplete.wsp coal/requestStart.wsp coal/fetchStart.wsp coal/mediaWikiLoadComplete.wsp coal/responseEnd.wsp coal/domainLookupEnd.wsp coal/domainLookupStart.wsp coal/domInteractive.wsp coal/unloadEventEnd.wsp
Test for loop
$ for path in coal-archive/*; do file=$(basename "$path"); echo whisper-fill "$path" "coal-cur/$file"; done; whisper-fill coal-archive/connectEnd.wsp coal-cur/connectEnd.wsp whisper-fill coal-archive/connectStart.wsp coal-cur/connectStart.wsp whisper-fill coal-archive/dnsLookup.wsp coal-cur/dnsLookup.wsp whisper-fill coal-archive/domainLookupEnddomComplete.wsp coal-cur/domainLookupEnddomComplete.wsp whisper-fill coal-archive/domainLookupEnd.wsp coal-cur/domainLookupEnd.wsp whisper-fill coal-archive/domainLookupStart.wsp coal-cur/domainLookupStart.wsp whisper-fill coal-archive/domComplete.wsp coal-cur/domComplete.wsp whisper-fill coal-archive/domContentLoadedEventEnd.wsp coal-cur/domContentLoadedEventEnd.wsp whisper-fill coal-archive/domContentLoadedEventStart.wsp coal-cur/domContentLoadedEventStart.wsp whisper-fill coal-archive/domInteractive.wsp coal-cur/domInteractive.wsp whisper-fill coal-archive/domLoading.wsp coal-cur/domLoading.wsp whisper-fill coal-archive/fetchStart.wsp coal-cur/fetchStart.wsp whisper-fill coal-archive/firstPaint.wsp coal-cur/firstPaint.wsp whisper-fill coal-archive/loadEventEnd.wsp coal-cur/loadEventEnd.wsp whisper-fill coal-archive/loadEventStart.wsp coal-cur/loadEventStart.wsp whisper-fill coal-archive/mediaWikiLoadComplete.wsp coal-cur/mediaWikiLoadComplete.wsp whisper-fill coal-archive/mediaWikiLoadEnd.wsp coal-cur/mediaWikiLoadEnd.wsp whisper-fill coal-archive/mediaWikiLoadStart.wsp coal-cur/mediaWikiLoadStart.wsp whisper-fill coal-archive/redirectCount.wsp coal-cur/redirectCount.wsp whisper-fill coal-archive/redirectEnd.wsp coal-cur/redirectEnd.wsp whisper-fill coal-archive/redirecting.wsp coal-cur/redirecting.wsp whisper-fill coal-archive/redirectStart.wsp coal-cur/redirectStart.wsp whisper-fill coal-archive/requestStart.wsp coal-cur/requestStart.wsp whisper-fill coal-archive/responseEnd.wsp coal-cur/responseEnd.wsp whisper-fill coal-archive/responseStart.wsp coal-cur/responseStart.wsp whisper-fill coal-archive/saveTiming.wsp coal-cur/saveTiming.wsp whisper-fill coal-archive/secureConnectionStart.wsp coal-cur/secureConnectionStart.wsp whisper-fill coal-archive/unloadEventEnd.wsp coal-cur/unloadEventEnd.wsp whisper-fill coal-archive/unloadEventStart.wsp coal-cur/unloadEventStart.wsp
First test
$ for path in coal-archive/*; do file=$(basename "$path"); whisper-fill "$path" "coal-cur/$file"; done; IOError: [Errno 2] No such file or directory: 'coal-cur/domainLookupEnddomComplete.wsp' IOError: [Errno 2] No such file or directory: 'coal-cur/domLoading.wsp'
Existence of domainLookupEnddomComplete is a bug. Manually excluded from the real merge.
Metric domLoading is real, but it was discontinued some time last year. I've copied it over to the live dataset without any merge.
Second test
[22:35 UTC] krinkle at graphite2001.codfw.wmnet in ~ $ rm coal-archive/domainLookupEnddomComplete.wsp $ cp coal-archive/domLoading.wsp /var/lib/coal/ $ rm coal-archive/domLoading.wsp # .. Delete and re-create coal-archive and coal-cur from backups [22:38 UTC] krinkle at graphite2001.codfw.wmnet in ~ $ for path in coal-archive/*; do file=$(basename "$path"); whisper-fill "$path" "coal-cur/$file"; done;
Finishes now without errors. Applied to the live data on graphite2001 and graphite1001.
Merge on graphite2001
$ sudo -su coal $ for path in /home/krinkle/coal-archive/*; do file=$(basename "$path"); whisper-fill "$path" "$file"; done;