Maniphest T201892

Toolforge: Build dashboard that breaks down webservice type usage by tool
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Legoktm
	Aug 14 2018, 3:00 AM

Description

I want

P7455 (An Untitled Masterwork)

1	docker-registry.tools.wmflabs.org/toollabs-jdk8-web:latest
2	==========================================================
3	corenlp
4	mw2sparql
5	simplewd
6	----
7	docker-registry.tools.wmflabs.org/toollabs-nodejs-web:latest
8	============================================================
9	autodesc
10	embeddeddata
11	hatjitsu
12	hub
13	nodejs-mw-oauth-tool
14	sql-optimizer
15	tesseract-ocr-service
16	translate-bot
17	unique
18	video2commons-socketio
19	w-slackbot
20	wd-rank
21	wikifactmine-api
22	wm-commons-emoji-bot
23	www-portal-builder
24	----
25	docker-registry.tools.wmflabs.org/toollabs-php-web:latest
26	=========================================================
27	add-information
28	addshore-dev
29	admin
30	admin-beta
31	anagrimes
32	android-maven-repo
33	article
34	articlerequest
35	articlerequest-dev
36	ato
37	autolist
38	bash
39	bd808-test
40	blankpages
41	bldrwnsch
42	blog
43	book2scroll
44	catfood
45	catnap
46	cats-php
47	catscan2
48	cdnjs
49	cdnjs-beta
50	citations
51	citations-dev
52	cite-o-meter
53	commons-app-stats
54	commonsedge
55	commonshelper
56	commonsinterwiki
57	comprende
58	copypatrol
59	costar
60	coverage
61	data-design-demo
62	delinker
63	deskana
64	dewkin
65	dnbtools
66	durl-shortener
67	earwig-dev
68	earwigbot
69	etytree
70	everythingisconnected
71	featured-article
72	file-reuse
73	file-reuse-piwik
74	file-reuse-test
75	file-siblings
76	filedupes
77	fist
78	five-million
79	flickr2commons
80	forrestbot
81	geograph2commons
82	geohack
83	gerrit-reviewer-bot
84	giraffe
85	glamtools
86	gmt
87	grantmetrics
88	grantmetrics-test
89	guc
90	hartman
91	hasteurbot
92	hennalabs
93	heritage
94	icommons
95	interaction-timeline
96	intuition
97	ios-crashes
98	ircredirect
99	isbn-usage
100	isbn2wiki
101	isin
102	joanjoc
103	kasper-data-translator
104	langviews
105	langviews-test
106	lingua-libre
107	list
108	listeria
109	locator
110	lolrrit-wm
111	lziad
112	magnus-toolserver
113	magnustools
114	magog
115	makeref
116	maplayers-demo
117	massviews
118	massviews-test
119	matthewrbowker
120	matthewrbowker-dev
121	media-reports
122	mediaviews
123	mediawiki-mirror
124	meetbot
125	metamine
126	metaviews
127	metricslibrary
128	missingtopics
129	mix-n-match
130	most-readable-pages
131	most-wanted
132	multidesc
133	mzmcbride
134	nagf
135	niosh
136	noclaims
137	nppdash
138	oauth-hello-world
139	olympics
140	oojs-ui
141	opendatasets
142	ores
143	orphantalk
144	osm-gadget-leaflet
145	pagecounts
146	pagepile
147	pageviews
148	pageviews-test
149	paste
150	paws-support
151	peachy-docs
152	phabulous
153	piagetbot
154	plagiabot
155	precise-tools
156	pub
157	pywikibot
158	pywikibot-testwiki
159	pywikipedia
160	quarrybot-enwiki
161	query
162	quick-intersection
163	quickstatements
164	rank
165	reasonator
166	redirectviews
167	redirtest
168	replag
169	ricordisamoa
170	sal
171	sau226test
172	shields
173	sighting
174	siteviews
175	snapshots
176	stashbot
177	static
178	supercount
179	superzerocool
180	tabletop
181	templatecheck
182	templator
183	tfaprotbot
184	toolscript
185	toolserver
186	toolserver-home-archive
187	tooltranslate
188	topicmatcher
189	topviews
190	topviews-test
191	tour
192	tptools
193	typoscan
194	url2commons
195	usage
196	userrank
197	userviews
198	usualsuspects
199	vendor
200	versions
201	warped-to-iframe
202	watroles
203	wdq2sparql
204	widar
205	wiki-todo
206	wikidata-exports
207	wikidata-game
208	wikidata-reconcile
209	wikidata-terminator
210	wikidata-todo
211	wikidipendenza
212	wikilovesdownloads
213	wikipedia-readability
214	wikiradio
215	wikishootme
216	wikisoba
217	wikisource-penguin-classics
218	wits
219	wlmuk
220	wmf-task-samtar
221	ws-google-ocr
222	ws2wd
223	wscontest
224	xtools-autoedits
225	xtools-dev
226	xtools-ec
227	xtools-pages
228	yellowbot
229	zppixbot
230	----
231	docker-registry.tools.wmflabs.org/toollabs-php72-web:latest
232	===========================================================
233	coverme
234	krinkle-redirect
235	wikiinfo
236	----
237	docker-registry.tools.wmflabs.org/toollabs-python-web:latest
238	============================================================
239	apt-browser
240	ash-django
241	awmd-stats
242	category-by-uploaders
243	checker
244	commons-mass-description
245	commons-mass-description-test
246	contribgraph
247	csfd
248	csp-report
249	deep-learning-services
250	deprecated-fixer-bot
251	detox
252	devys
253	editgroups
254	extreg-wos
255	faces
256	farhangestan
257	fatameh
258	fireflytools
259	firmenbuch
260	gerrit-newcomer-bot
261	github-notif-bot
262	glam2commons
263	globalprefs
264	grid-jobs
265	gsoc
266	gsoc-petscan-query-articles
267	gsoc-worklist-tool
268	harvesting-data-refinery
269	import-500px
270	interactoa
271	ipwatcher
272	kokolores
273	ksamsok-rest
274	lexeme-forms
275	massmailer
276	mathqa
277	mediaviews-api
278	merge2pdf
279	mitmachen
280	mwpackages
281	my-first-django-oauth-app
282	my-first-flask-oauth-tool
283	my-threads
284	newusers
285	ninthcircuit
286	nli-wiki
287	oabot-wd-game
288	oauthtest
289	openrefine-wikidata
290	openstack-browser
291	order-user-by-reg
292	ordia
293	ores-support-checklist
294	outreachy-user-contribution-tool
295	outreachy-user-ranking-tool
296	outreachy-wikicv
297	phab-ban
298	phpcs
299	platypus-qa
300	prism
301	proxies
302	ptable
303	readmore
304	recitation-bot
305	refill-api
306	reviewers
307	sibu
308	sibutest
309	similarity
310	sistercities
311	sphinxcapt-leaderboard
312	statistics
313	thankyou
314	tool-db-usage
315	toolviews
316	translatemplate
317	twltools
318	uploadhelper-ir
319	visualcategories
320	wdmm
321	wdq-checker
322	weapon-of-mass-description
323	weapon-of-mass-description-test
324	wembedder
325	wikigrade
326	wikitext-deprecation
327	worklist-tool
328	wprequests
329	yabbr
330	zhaofeng-test
331	zoomproof
332	----
333	docker-registry.tools.wmflabs.org/toollabs-python2-web:latest
334	=============================================================
335	fab-proxy
336	hashtags-test
337	ifttt
338	ifttt-testing
339	ircredirector
340	montage
341	monumental
342	phabricator-bug-status
343	proneval-gsoc17
344	spiarticleanalyzer
345	strephit
346	threed2commons
347	video2commons
348	video2commons-test
349	wikiloves
350	----
351	docker-registry.tools.wmflabs.org/toollabs-ruby-web:latest
352	==========================================================
353	position-holder-history
354	prompter
355	verification-pages
356	----
357	docker-registry.tools.wmflabs.org/toollabs-tcl-web:latest
358	=========================================================
359	giftbot
360	----

but in a web interface/dashboard.

I ran kubectl get pods --all-namespaces --output=json on tools-k8s-master-01 to get a JSON file, and then ran the following to parse it:

#!/usr/bin/python3
# (C) 2018 Kunal Mehta <legoktm@member.fsf.org>

from collections import defaultdict
import json

with open('pods.json') as f:
    d = json.load(f)

tools = defaultdict(set)

for info in d['items']:
    try:
        name = info['metadata']['namespace']
    except KeyError:
        print(info)
        raise
    for container in info['spec']['containers']:
        if container['name'] != 'webservice':
            continue
        tools[container['image']].add(name)

for image, tool_names in sorted(tools.items()):
    print(image)
    print('=' * len(image))
    print('\n'.join(sorted(tool_names)))
    print('----')

My main motivation is to track the rollout of the new php7.2 image.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		bd808	T201892 Toolforge: Build dashboard that breaks down webservice type usage by tool
Resolved		• Bstorm	T233372 Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection
Resolved	Security	• aborrero	T346313 toolforge: k8s-status: prevent it from accessing some information

Event Timeline

Legoktm created this task.Aug 14 2018, 3:00 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 14 2018, 3:00 AM

I got stuck in implementing this in a web tool because only from the k8s-master can you see other tools' running images and stuff. Maybe we could lift that restriction for the admin tool? I'm not too sure about the k8s security model and what potential problems that could expose.

I thought I had a related ticket for this, but I can't find it. A long, long time ago (T140255) I talked with @yuvipanda about figuring out how to create a read-only user for our Kubernetes cluster that would work similar to the "novaobserver" user we have on the OpenStack side that could be used for building tools that look at the the overall k8s cluster. I think this can be done with RBAC auth. Once we upgrade our k8s deploy to 1.9 or newer it may just be a matter of granting the "view" cluster role to a user.

The shorter term hack solution would be to setup a root cronjob on k8s-master to dump the json data to a file that you can read from a tool. You could probably get away with only updating the data every hour or two and still have a good idea of what's going on in the cluster.

bd808 merged a task: T140255: Add kubernetes status page to admin tool.Feb 10 2019, 7:38 PM

bd808 added a subscriber: Zppix.

bd808 added a subtask: T214513: Deploy and migrate tools to a Kubernetes v1.15 or newer cluster.Feb 10 2019, 7:44 PM

• GTirloni added a project: cloud-services-team (Kanban).Mar 21 2019, 5:01 PM

Xinbenlv subscribed.Jul 1 2019, 5:25 PM

bd808 mentioned this in T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection.Sep 19 2019, 11:45 PM

bd808 edited subtasks, added: T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection; removed: T214513: Deploy and migrate tools to a Kubernetes v1.15 or newer cluster.

• Bstorm closed subtask T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection as Resolved.Dec 19 2019, 7:34 PM

Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptDec 19 2019, 7:35 PM

Work in progress at https://tools.wmflabs.org/k8s-status/images/

We will probably want to make some more filtered/targeted tools in the future to drive people to newer runtimes, but we have a way to get that data from a tool now and https://tools.wmflabs.org/k8s-status/images/ as a high level overview.

Toolforge: Build dashboard that breaks down webservice type usage by toolClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Toolforge: Build dashboard that breaks down webservice type usage by tool
Closed, ResolvedPublic
Actions

Related Objects
Search...