Page MenuHomePhabricator

Find top 15 target projects that could use Selenium tests to prevent incidents
Closed, ResolvedPublic

Description

Review incidents from the past year to determine how many had a testable regression as the cause. From review identify the top 15 target projects.

Review existing e2e test coverage

List of repositories with at least one Selenium test.

RepositoryLanguage
mediawiki/coreJavaScript
mediawiki/extensions/AdvancedSearchJavaScript
mediawiki/extensions/CentralAuthRuby
mediawiki/extensions/CentralNoticeRuby
mediawiki/extensions/CirrusSearchJavaScript
mediawiki/extensions/EchoJavaScript
mediawiki/extensions/ElectronPdfServiceJavaScript
mediawiki/extensions/GettingStartedRuby
mediawiki/extensions/MathJavaScript
mediawiki/extensions/MobileFrontendRuby
mediawiki/extensions/MultimediaViewerRuby
mediawiki/extensions/NewsletterJavaScript
mediawiki/extensions/ORESJavaScript
mediawiki/extensions/PopupsJavaScript
mediawiki/extensions/QuickSurveysRuby
mediawiki/extensions/RelatedArticlesJavaScript
mediawiki/extensions/RevisionSliderRuby
mediawiki/extensions/TwoColConflictJavaScript, Ruby
mediawiki/extensions/WikibaseJavaScript, Ruby
mediawiki/extensions/WikibaseLexemeJavaScript, Ruby
mediawiki/extensions/WikimediaEventsPHP
mediawiki/skins/MinervaNeueRuby
phab-deploymentJavaScript
wikimedia/portals/deployJavaScript

Define prioritization scheme

How much change did happen

wikimedia.biterg.io report from 2017-11-01 to 2018-10-31.

RepositoryCommits
mediawiki/extensions11300
operations/puppet7988
mediawiki/core4590
operations/mediawiki-config4005
integration/config1652
operations/software/librenms1169
pywikibot/core927
mediawiki/extensions/Wikibase806
apps/android/wikipedia789
mediawiki/services/parsoid700
mediawiki/extensions/VisualEditor692
operations/dns653
VisualEditor/VisualEditor599
mediawiki/skins570
mediawiki/extensions/MobileFrontend504
mediawiki/extensions/ContentTranslation491
translatewiki486
oojs/ui469
wikimedia/fundraising/crm457
mediawiki/extensions/BlueSpiceFoundation414
mediawiki/extensions/CirrusSearch357
mediawiki/extensions/AbuseFilter306
phabricator/phabricator302
mediawiki/services/restbase290
mediawiki/extensions/Flow232
mediawiki/extensions/Echo223
mediawiki/vagrant221
mediawiki/extensions/Popups184
mediawiki/extensions/Translate182
mediawiki/extensions/DonationInterface180
analytics/refinery178
mediawiki/extensions/PageTriage177
mediawiki/extensions/Cargo176
mediawiki/tools/codesniffer156
mediawiki/extensions/TimedMediaHandler152
mediawiki/extensions/UniversalLanguageSelector142
mediawiki/vendor140
mediawiki/extensions/SocialProfile139
analytics/refinery/source138
operations/software137
mediawiki/services/restbase/deploy136
operations/debs/pybal123
mediawiki/extensions/CentralAuth116
mediawiki/tools/release116
mediawiki/services/cxserver112
mediawiki/extensions/BlueSpiceExtensions110
mediawiki/extensions/WikimediaEvents110
labs/private108
operations/debs/python-kafka104
labs/tools/heritage96

Incident Review 2017-11-01 to 2018-10-31

RepositoryIncidentsMaintainers
operations/puppet22SRE
mediawiki/core6Core Platform
operations/mediawiki-config4Release Engineering
mediawiki/extensions/Wikibase4WMDE
wikidata/query/rdf2WMDE
operations/debs/pybal2SRE
mediawiki/extensions/ORES2Growth
integration/config2Release Engineering
wikidata/query/blazegraph1WMDE
operations/software1SRE
operations/dns1SRE
mediawiki/vagrant1Release Engineering
mediawiki/tools/release1
mediawiki/services/ores/deploy1Growth
mediawiki/services/eventstreams1
mediawiki/extensions/WikibaseQualityConstraints1WMDE
mediawiki/extensions/PropertySuggester1WMDE
mediawiki/extensions/PageTriage1Growth
mediawiki/extensions/Cognate1WMDE
mediawiki/extensions/Babel1Editing
maps/tilerator/deploy1
maps/kartotherian/deploy1
integration/jenkins1Release Engineering
eventlogging1
analytics/refinery/source1Analytics
analytics/refinery1Analytics
All-Projects1

Repository, Change, Maintainers, Coverage, Incidents, Selected

RepositoryChangeMaintainersCoverageIncidentsSelected
mediawiki/extensions11300
operations/puppet7988SRE22
mediawiki/core4590Core PlatformJavaScript6
operations/mediawiki-config4005Release Engineering4
integration/config1652Release Engineering2
operations/software/librenms1169SRE
pywikibot/core927
mediawiki/extensions/Wikibase806WMDEJavaScript, Ruby4
apps/android/wikipedia789
mediawiki/services/parsoid700Parsing
mediawiki/extensions/VisualEditor692Editingโœ…
operations/dns653SRE1
VisualEditor/VisualEditor599Editing
mediawiki/skins570Reading
mediawiki/extensions/MobileFrontend504ReadingRuby
mediawiki/extensions/ContentTranslation491Language engineeringโœ…
translatewiki486
oojs/ui469
wikimedia/fundraising/crm457Fundraising tech
mediawiki/extensions/BlueSpiceFoundation414
mediawiki/extensions/CirrusSearch357Search PlatformJavaScript
mediawiki/extensions/AbuseFilter306Contributorsโœ…
phabricator/phabricator302Release Engineeringโœ…
mediawiki/services/restbase290Core Platform
mediawiki/extensions/Flow232Growthโœ…
mediawiki/extensions/Echo223GrowthJavaScript
mediawiki/vagrant221Release Engineering1
mediawiki/extensions/Popups184ReadingJavaScript
mediawiki/extensions/Translate182Language engineeringโœ…
mediawiki/extensions/DonationInterface180Fundraising techโœ…
analytics/refinery178Analytics1
mediawiki/extensions/PageTriage177Growth1โœ…
mediawiki/extensions/Cargo176
mediawiki/tools/codesniffer156
mediawiki/extensions/TimedMediaHandler152Readingโœ…
mediawiki/extensions/UniversalLanguageSelector142Language engineeringโœ…
mediawiki/vendor140
mediawiki/extensions/SocialProfile139
analytics/refinery/source138Analytics1
operations/software137SRE1
mediawiki/services/restbase/deploy136Core Platform
operations/debs/pybal123SRE2
mediawiki/extensions/CentralAuth116Ruby
mediawiki/tools/release1161
mediawiki/services/cxserver112
mediawiki/extensions/BlueSpiceExtensions110
mediawiki/extensions/WikimediaEvents110PHP
labs/private108
operations/debs/python-kafka104SRE
labs/tools/heritage96

git rev-list --all --after="11/1/2017" --before="10/31/2018" --count

RepositoryChangeMaintainersCoverageIncidentsSelected
mediawiki/extensions/WikibaseQualityConstraints910WMDE1โœ…
mediawiki/extensions/ORES364GrowthJavaScript2
wikidata/query/rdf204WMDE2
mediawiki/extensions/Babel146Editing1โœ…
mediawiki/services/ores/deploy84Growth1
maps/kartotherian/deploy801
mediawiki/extensions/PropertySuggester67WMDE1โœ…
maps/tilerator/deploy611
mediawiki/extensions/Cognate47WMDE1โœ…
eventlogging261
integration/jenkins19Release Engineering1
mediawiki/services/eventstreams161
wikidata/query/blazegraph10WMDE1
All-Projects1

Prioritize e2e testing gaps

RepositoryChangeMaintainersCoverageIncidentsSelected
mediawiki/extensions/VisualEditor692Editingโœ…
mediawiki/extensions/ContentTranslation491Language engineeringโœ…
mediawiki/extensions/AbuseFilter306Contributorsโœ…
phabricator/phabricator302Release Engineeringโœ…
mediawiki/extensions/Flow232Growthโœ…
mediawiki/extensions/Translate182Language engineeringโœ…
mediawiki/extensions/DonationInterface180Fundraising techโœ…
mediawiki/extensions/PageTriage177Growth1โœ…
mediawiki/extensions/TimedMediaHandler152Readingโœ…
mediawiki/extensions/UniversalLanguageSelector142Language engineeringโœ…
mediawiki/extensions/WikibaseQualityConstraints910WMDE1โœ…
mediawiki/extensions/Babel146Editing1โœ…
mediawiki/extensions/PropertySuggester67WMDE1โœ…
mediawiki/extensions/Cognate47WMDE1โœ…

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: jeblad. ยท View Herald TranscriptSep 20 2018, 5:41 PM
hashar added a subscriber: hashar.Sep 21 2018, 10:24 AM

A quick and dirty way to get the busiest repos per number of commits is to use git log on mediawiki/extensions:

1For https://phabricator.wikimedia.org/T199133
2
3Use mediawiki/extensions.git as the source of data, Gerrit craft a commit in that repo whenever a mediawiki extension has a new commit.
4
5The repository is already cloned for beta cluster on deployment-deploy01 in /srv/mediawiki-staging/php-master/extensions . So we can git log and process the result to get the busiest repos.
6
7
8Gather the modules updated by each commit made to mediawiki/extensions.git
9
10 ssh deployment-deploy01.deployment-prep.eqiad.wmflabs
11 cd /srv/mediawiki-staging/php-master/extensions
12 git log --format="" --stat --since "One year ago"|egrep -o '^ \w\w+' > ~/mwextscommits.log
13
14That is listing every commits, including l10n update and regular dependencies updates handled by bots.
15
16Process:
17
18 sort ~/mwextscommits.log | uniq -c | sort -rn
19
20Profit:
21
22 1103 Wikibase
23 930 VisualEditor
24 805 ContentTranslation
25 763 WikibaseLexeme
26 655 MobileFrontend
27 634 WikibaseQualityConstraints
28 564 AbuseFilter
29 537 CirrusSearch
30 497 BlueSpiceFoundation
31 451 AdvancedSearch
32 449 Translate
33 441 Flow
34 441 Echo
35 392 WikimediaMessages
36 377 FileImporter
37 330 CentralAuth
38 318 Popups
39 311 DonationInterface
40 309 UploadWizard
41 293 ORES
42 292 TwoColConflict
43 287 Cargo
44 284 Collection
45 283 TimedMediaHandler
46 281 PageTriage
47 281 Cite
48 276 UniversalLanguageSelector
49 276 Kartographer
50 258 Thanks
51 248 ProofreadPage
52 246 FlaggedRevs
53 245 PageForms
54 216 Linter
55 208 WikiEditor
56 202 GlobalPreferences
57 194 CentralNotice
58 190 ReadingLists
59 175 ConfirmEdit
60 174 Newsletter
61 174 Math
62 170 OAuth
63 169 TemplateData
64 168 LiquidThreads
65 164 CategoryTree
66 162 WikiLove
67 159 MultimediaViewer
68 157 WikimediaEvents
69 157 EducationProgram
70 154 SocialProfile
71 150 GlobalUsage
72 148 BetaFeatures
73 147 Babel
74 146 CodeMirror
75 143 Scribunto
76 138 CheckUser
77 135 Gadgets
78 133 Score
79 132 Citoid
80 131 MassMessage
81 128 LoginNotify
82 127 RevisionSlider
83 126 CommentStreams
84 125 TemplateWizard
85 125 ParserFunctions
86 125 GlobalBlocking
87 121 TwnMainPage
88 121 OATHAuth
89 120 Wikidata
90 118 3D
91 116 WikiLexicalData
92 114 ArticleFeedbackv5
93 103 BlueSpiceExtendedSearch
94 102 Renameuser
95...
96

Another note from a discussion with Zeljko: we have a third party gathering git/gerrit metrics. They store it in ElasticSearch and use Kibana for dashboarding: https://wikimedia.biterg.io/

That was build for technical community metrics.

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin added a comment.EditedSep 24 2018, 12:52 PM

Thanks @hashar, Bitergia is really useful! It's report for the last year:

1Repository Commits
2https://gerrit.wikimedia.org/r/mediawiki/extensions 35616
3https://gerrit.wikimedia.org/r/operations/puppet 7354
4https://gerrit.wikimedia.org/r/mediawiki/core 6288
5https://gerrit.wikimedia.org/r/operations/mediawiki-config 3160
6https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase 2319
7https://gerrit.wikimedia.org/r/mediawiki/skins 2098
8https://gerrit.wikimedia.org/r/apps/android/wikipedia 1959
9https://gerrit.wikimedia.org/r/phabricator/phabricator 1532
10https://gerrit.wikimedia.org/r/operations/software/librenms 1446
11https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor 1263
12https://gerrit.wikimedia.org/r/oojs/ui 1229
13https://gerrit.wikimedia.org/r/mediawiki/extensions/BlueSpiceExtensions 1176
14https://gerrit.wikimedia.org/r/integration/config 1160
15https://gerrit.wikimedia.org/r/pywikibot/core 1101
16https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend 923
17https://gerrit.wikimedia.org/r/mediawiki/extensions/ContentTranslation 915
18https://gerrit.wikimedia.org/r/VisualEditor/VisualEditor 856
19https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch 688
20https://gerrit.wikimedia.org/r/translatewiki 674
21https://gerrit.wikimedia.org/r/mediawiki/extensions/BlueSpiceFoundation 622
22https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm 621
23https://gerrit.wikimedia.org/r/mediawiki/extensions/Popups 603
24https://gerrit.wikimedia.org/r/mediawiki/extensions/Flow 563
25https://gerrit.wikimedia.org/r/operations/dns 539
26https://gerrit.wikimedia.org/r/mediawiki/extensions/Translate 510
27https://gerrit.wikimedia.org/r/mediawiki/extensions/Echo 504
28https://gerrit.wikimedia.org/r/mediawiki/extensions/DonationInterface 494
29https://gerrit.wikimedia.org/r/mediawiki/services/parsoid 444
30https://gerrit.wikimedia.org/r/mediawiki/extensions/WikimediaMessages 380
31https://gerrit.wikimedia.org/r/mediawiki/extensions/CentralAuth 376
32https://gerrit.wikimedia.org/r/mediawiki/vagrant 373
33https://gerrit.wikimedia.org/r/mediawiki/extensions/AbuseFilter 354
34https://gerrit.wikimedia.org/r/mediawiki/extensions/Collection 328
35https://gerrit.wikimedia.org/r/mediawiki/services/restbase 327
36https://gerrit.wikimedia.org/r/mediawiki/extensions/UploadWizard 311
37https://gerrit.wikimedia.org/r/mediawiki/extensions/FlaggedRevs 297
38https://gerrit.wikimedia.org/r/mediawiki/extensions/TimedMediaHandler 291
39https://gerrit.wikimedia.org/r/mediawiki/extensions/ProofreadPage 283
40https://gerrit.wikimedia.org/r/mediawiki/extensions/UniversalLanguageSelector 283
41https://gerrit.wikimedia.org/r/mediawiki/extensions/Newsletter 276
42https://gerrit.wikimedia.org/r/mediawiki/skins/Vector 275
43https://gerrit.wikimedia.org/r/mediawiki/extensions/Cite 271
44https://gerrit.wikimedia.org/r/mediawiki/extensions/CodeMirror 269
45https://gerrit.wikimedia.org/r/integration/jenkins-job-builder 252
46https://gerrit.wikimedia.org/r/mediawiki/extensions/CentralNotice 246
47https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikidata 227
48https://gerrit.wikimedia.org/r/mediawiki/extensions/ConfirmEdit 225
49https://gerrit.wikimedia.org/r/mediawiki/extensions/WikimediaEvents 219
50https://gerrit.wikimedia.org/r/mediawiki/extensions/Math 215
51https://gerrit.wikimedia.org/r/operations/debs/python-kafka 214

zeljkofilipin added a comment.EditedSep 24 2018, 12:56 PM

git effort results for mediawiki/extensions (git effort --above 100 -- --after="one year ago"'):

1 path commits active days
2
3 Wikibase.............................. 1091 341
4 VisualEditor.......................... 922 318
5 ContentTranslation.................... 790 328
6 WikibaseLexeme........................ 762 262
7 MobileFrontend........................ 649 308
8 WikibaseQualityConstraints............ 629 238
9 AbuseFilter........................... 562 306
10 CirrusSearch.......................... 531 247
11 BlueSpiceFoundation................... 489 264
12 AdvancedSearch........................ 448 256
13 Translate............................. 445 286
14 Echo.................................. 437 275
15 Flow.................................. 436 272
16 WikimediaMessages..................... 385 308
17 FileImporter.......................... 375 178
18 CentralAuth........................... 325 246
19 Popups................................ 315 196
20 UploadWizard.......................... 306 237
21 DonationInterface..................... 300 197
22 TwoColConflict........................ 288 182
23 ORES.................................. 287 173
24 Cargo................................. 285 182
25 PageTriage............................ 280 186
26 TimedMediaHandler..................... 279 192
27 Collection............................ 279 230
28 Cite.................................. 276 225
29 UniversalLanguageSelector............. 272 193
30 Kartographer.......................... 272 189
31 Thanks................................ 256 213
32 ProofreadPage......................... 246 190
33 FlaggedRevs........................... 245 204
34 PageForms............................. 244 159
35 Linter................................ 212 177
36 WikiEditor............................ 204 161
37 GlobalPreferences..................... 201 149
38 ReadingLists.......................... 190 139
39 CentralNotice......................... 190 158
40 Newsletter............................ 172 127
41 Math.................................. 171 141
42 ConfirmEdit........................... 171 157
43 OAuth................................. 169 146
44 LiquidThreads......................... 165 149
45 TemplateData.......................... 163 143
46 WikiLove.............................. 162 136
47 CategoryTree.......................... 160 139
48 MultimediaViewer...................... 157 132
49 SocialProfile......................... 154 95
50 EducationProgram...................... 154 134
51 WikimediaEvents....................... 151 121
52 BetaFeatures.......................... 146 128
53 GlobalUsage........................... 145 130
54 CodeMirror............................ 145 113
55 Babel................................. 144 131
56 Scribunto............................. 139 119
57 CheckUser............................. 137 116
58 Gadgets............................... 132 121
59 Score................................. 131 102
60 MassMessage........................... 130 109
61 Citoid................................ 129 117
62 RevisionSlider........................ 126 103
63 LoginNotify........................... 126 119
64 TemplateWizard........................ 125 93
65 ParserFunctions....................... 122 109
66 CommentStreams........................ 122 115
67 GlobalBlocking........................ 121 111
68 OATHAuth.............................. 120 104
69 .gitmodules........................... 119 67
70 TwnMainPage........................... 118 109
71 3D.................................... 117 85
72 WikiLexicalData....................... 115 107
73 ArticleFeedbackv5..................... 113 107
74 Renameuser............................ 102 89
75 BlueSpiceExtendedSearch............... 101 80

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Oct 3 2018, 4:18 PM
zeljkofilipin updated the task description. (Show Details)Oct 4 2018, 4:35 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Nov 7 2018, 7:11 PM
zeljkofilipin rescinded a token.
zeljkofilipin awarded a token.
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:18 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:20 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:35 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:41 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:44 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:51 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 1:56 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 3:46 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 3:55 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 3:57 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 4:10 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 4:15 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 5:15 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 5:28 PM
zeljkofilipin updated the task description. (Show Details)Nov 8 2018, 5:30 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Just saying, the "ORES" service and backend of the ORES extension is being maintained by scoring platform team not growth.

greg updated the task description. (Show Details)Nov 14 2018, 5:08 PM

Just saying, the "ORES" service and backend of the ORES extension is being maintained by scoring platform team not growth.

Thanks, but Developers/Maintainers says Growth. Should it be updated?

ExtensionDescriptionCode StewardsMaintainersIn training
ORESIntegrates data from the ORES project into the RecentChanges viewGrowth teamUnassignedโ€”

Thanks, but Developers/Maintainers says Growth. Should it be updated?

So there are three different things called ORES here:

  • ORES service (ores.wikimedia.org) that does the heavy lifting.
    • The code for that is in mediawiki/services/ores/deploy, it's python
    • Has one incident in the table but I suspect it's more
    • This is being maintained by Scoring platform team
  • ORES extension frontend:
    • The code is mostly in the ORES extension (The Recentchanges hook handlers basically), it's mostly PHP
    • Doesn't have any incidents
    • That is being maintained by Growth team
  • ORES extension backend (that gets the data from the service and store it in ores_classification table):
    • This code is also sitting in ORES extension (basically the rest of the extension), it's also mostly PHP
    • It had two incidents (It might the frontend part but my guess goes to the backend)
    • It's being maintained by Scoring platform team.

HTH

So there are three different things called ORES here:

  • ORES service...
    • This is being maintained by Scoring platform team
  • ORES extension frontend...
    • That is being maintained by Growth team
  • ORES extension backend...
    • It's being maintained by Scoring platform team.

The only mention of ORES at Developers/Maintainers page is about the extension. It does not mention frontend/backend or service. Should the page be updated? ๐Ÿ˜„

The only mention of ORES at Developers/Maintainers page is about the extension. It does not mention frontend/backend or service. Should the page be updated? ๐Ÿ˜„

Done now :D Thanks for pointing out.

greg closed this task as Resolved.Nov 27 2018, 9:13 PM
greg added a subscriber: greg.
RepositoryChangeMaintainersCoverageIncidentsSelected
mediawiki/extensions/VisualEditor692Editingโœ…
mediawiki/extensions/ContentTranslation491Language engineeringโœ…
mediawiki/extensions/AbuseFilter306Contributorsโœ…
phabricator/phabricator302Release Engineeringโœ…
mediawiki/extensions/Flow232Growthโœ…
mediawiki/extensions/Translate182Language engineeringโœ…
mediawiki/extensions/DonationInterface180Fundraising techโœ…
mediawiki/extensions/PageTriage177Growth1โœ…
mediawiki/extensions/TimedMediaHandler152Readingโœ…
mediawiki/extensions/UniversalLanguageSelector142Language engineeringโœ…
mediawiki/extensions/WikibaseQualityConstraints910WMDE1โœ…
mediawiki/extensions/Babel146Editing1โœ…
mediawiki/extensions/PropertySuggester67WMDE1โœ…
mediawiki/extensions/Cognate47WMDE1โœ…

14 is good enough.