Page MenuHomePhabricator

Toolforge: Build dashboard that breaks down webservice type usage by tool
Closed, ResolvedPublic

Description

I want

1docker-registry.tools.wmflabs.org/toollabs-jdk8-web:latest
2==========================================================
3corenlp
4mw2sparql
5simplewd
6----
7docker-registry.tools.wmflabs.org/toollabs-nodejs-web:latest
8============================================================
9autodesc
10embeddeddata
11hatjitsu
12hub
13nodejs-mw-oauth-tool
14sql-optimizer
15tesseract-ocr-service
16translate-bot
17unique
18video2commons-socketio
19w-slackbot
20wd-rank
21wikifactmine-api
22wm-commons-emoji-bot
23www-portal-builder
24----
25docker-registry.tools.wmflabs.org/toollabs-php-web:latest
26=========================================================
27add-information
28addshore-dev
29admin
30admin-beta
31anagrimes
32android-maven-repo
33article
34articlerequest
35articlerequest-dev
36ato
37autolist
38bash
39bd808-test
40blankpages
41bldrwnsch
42blog
43book2scroll
44catfood
45catnap
46cats-php
47catscan2
48cdnjs
49cdnjs-beta
50citations
51citations-dev
52cite-o-meter
53commons-app-stats
54commonsedge
55commonshelper
56commonsinterwiki
57comprende
58copypatrol
59costar
60coverage
61data-design-demo
62delinker
63deskana
64dewkin
65dnbtools
66durl-shortener
67earwig-dev
68earwigbot
69etytree
70everythingisconnected
71featured-article
72file-reuse
73file-reuse-piwik
74file-reuse-test
75file-siblings
76filedupes
77fist
78five-million
79flickr2commons
80forrestbot
81geograph2commons
82geohack
83gerrit-reviewer-bot
84giraffe
85glamtools
86gmt
87grantmetrics
88grantmetrics-test
89guc
90hartman
91hasteurbot
92hennalabs
93heritage
94icommons
95interaction-timeline
96intuition
97ios-crashes
98ircredirect
99isbn-usage
100isbn2wiki
101isin
102joanjoc
103kasper-data-translator
104langviews
105langviews-test
106lingua-libre
107list
108listeria
109locator
110lolrrit-wm
111lziad
112magnus-toolserver
113magnustools
114magog
115makeref
116maplayers-demo
117massviews
118massviews-test
119matthewrbowker
120matthewrbowker-dev
121media-reports
122mediaviews
123mediawiki-mirror
124meetbot
125metamine
126metaviews
127metricslibrary
128missingtopics
129mix-n-match
130most-readable-pages
131most-wanted
132multidesc
133mzmcbride
134nagf
135niosh
136noclaims
137nppdash
138oauth-hello-world
139olympics
140oojs-ui
141opendatasets
142ores
143orphantalk
144osm-gadget-leaflet
145pagecounts
146pagepile
147pageviews
148pageviews-test
149paste
150paws-support
151peachy-docs
152phabulous
153piagetbot
154plagiabot
155precise-tools
156pub
157pywikibot
158pywikibot-testwiki
159pywikipedia
160quarrybot-enwiki
161query
162quick-intersection
163quickstatements
164rank
165reasonator
166redirectviews
167redirtest
168replag
169ricordisamoa
170sal
171sau226test
172shields
173sighting
174siteviews
175snapshots
176stashbot
177static
178supercount
179superzerocool
180tabletop
181templatecheck
182templator
183tfaprotbot
184toolscript
185toolserver
186toolserver-home-archive
187tooltranslate
188topicmatcher
189topviews
190topviews-test
191tour
192tptools
193typoscan
194url2commons
195usage
196userrank
197userviews
198usualsuspects
199vendor
200versions
201warped-to-iframe
202watroles
203wdq2sparql
204widar
205wiki-todo
206wikidata-exports
207wikidata-game
208wikidata-reconcile
209wikidata-terminator
210wikidata-todo
211wikidipendenza
212wikilovesdownloads
213wikipedia-readability
214wikiradio
215wikishootme
216wikisoba
217wikisource-penguin-classics
218wits
219wlmuk
220wmf-task-samtar
221ws-google-ocr
222ws2wd
223wscontest
224xtools-autoedits
225xtools-dev
226xtools-ec
227xtools-pages
228yellowbot
229zppixbot
230----
231docker-registry.tools.wmflabs.org/toollabs-php72-web:latest
232===========================================================
233coverme
234krinkle-redirect
235wikiinfo
236----
237docker-registry.tools.wmflabs.org/toollabs-python-web:latest
238============================================================
239apt-browser
240ash-django
241awmd-stats
242category-by-uploaders
243checker
244commons-mass-description
245commons-mass-description-test
246contribgraph
247csfd
248csp-report
249deep-learning-services
250deprecated-fixer-bot
251detox
252devys
253editgroups
254extreg-wos
255faces
256farhangestan
257fatameh
258fireflytools
259firmenbuch
260gerrit-newcomer-bot
261github-notif-bot
262glam2commons
263globalprefs
264grid-jobs
265gsoc
266gsoc-petscan-query-articles
267gsoc-worklist-tool
268harvesting-data-refinery
269import-500px
270interactoa
271ipwatcher
272kokolores
273ksamsok-rest
274lexeme-forms
275massmailer
276mathqa
277mediaviews-api
278merge2pdf
279mitmachen
280mwpackages
281my-first-django-oauth-app
282my-first-flask-oauth-tool
283my-threads
284newusers
285ninthcircuit
286nli-wiki
287oabot-wd-game
288oauthtest
289openrefine-wikidata
290openstack-browser
291order-user-by-reg
292ordia
293ores-support-checklist
294outreachy-user-contribution-tool
295outreachy-user-ranking-tool
296outreachy-wikicv
297phab-ban
298phpcs
299platypus-qa
300prism
301proxies
302ptable
303readmore
304recitation-bot
305refill-api
306reviewers
307sibu
308sibutest
309similarity
310sistercities
311sphinxcapt-leaderboard
312statistics
313thankyou
314tool-db-usage
315toolviews
316translatemplate
317twltools
318uploadhelper-ir
319visualcategories
320wdmm
321wdq-checker
322weapon-of-mass-description
323weapon-of-mass-description-test
324wembedder
325wikigrade
326wikitext-deprecation
327worklist-tool
328wprequests
329yabbr
330zhaofeng-test
331zoomproof
332----
333docker-registry.tools.wmflabs.org/toollabs-python2-web:latest
334=============================================================
335fab-proxy
336hashtags-test
337ifttt
338ifttt-testing
339ircredirector
340montage
341monumental
342phabricator-bug-status
343proneval-gsoc17
344spiarticleanalyzer
345strephit
346threed2commons
347video2commons
348video2commons-test
349wikiloves
350----
351docker-registry.tools.wmflabs.org/toollabs-ruby-web:latest
352==========================================================
353position-holder-history
354prompter
355verification-pages
356----
357docker-registry.tools.wmflabs.org/toollabs-tcl-web:latest
358=========================================================
359giftbot
360----
but in a web interface/dashboard.

I ran kubectl get pods --all-namespaces --output=json on tools-k8s-master-01 to get a JSON file, and then ran the following to parse it:

#!/usr/bin/python3
# (C) 2018 Kunal Mehta <legoktm@member.fsf.org>

from collections import defaultdict
import json

with open('pods.json') as f:
    d = json.load(f)

tools = defaultdict(set)

for info in d['items']:
    try:
        name = info['metadata']['namespace']
    except KeyError:
        print(info)
        raise
    for container in info['spec']['containers']:
        if container['name'] != 'webservice':
            continue
        tools[container['image']].add(name)

for image, tool_names in sorted(tools.items()):
    print(image)
    print('=' * len(image))
    print('\n'.join(sorted(tool_names)))
    print('----')

My main motivation is to track the rollout of the new php7.2 image.

Event Timeline

I got stuck in implementing this in a web tool because only from the k8s-master can you see other tools' running images and stuff. Maybe we could lift that restriction for the admin tool? I'm not too sure about the k8s security model and what potential problems that could expose.

I thought I had a related ticket for this, but I can't find it. A long, long time ago (T140255) I talked with @yuvipanda about figuring out how to create a read-only user for our Kubernetes cluster that would work similar to the "novaobserver" user we have on the OpenStack side that could be used for building tools that look at the the overall k8s cluster. I think this can be done with RBAC auth. Once we upgrade our k8s deploy to 1.9 or newer it may just be a matter of granting the "view" cluster role to a user.

The shorter term hack solution would be to setup a root cronjob on k8s-master to dump the json data to a file that you can read from a tool. You could probably get away with only updating the data every hour or two and still have a good idea of what's going on in the cluster.

We will probably want to make some more filtered/targeted tools in the future to drive people to newer runtimes, but we have a way to get that data from a tool now and https://tools.wmflabs.org/k8s-status/images/ as a high level overview.