Page MenuHomePhabricator

[QA] Exploratory testing: Vector 2022 night theme
Closed, ResolvedPublic

Assigned To
Authored By
Jdlrobson
Apr 29 2024, 11:16 PM
Referenced Files
F54798556: Screenshot 2024-05-31 at 8.37.46 AM.png
May 31 2024, 3:37 PM
F54693163: screenshot 457.png
May 30 2024, 5:47 AM
F54692474: screenshot 456.png
May 30 2024, 5:47 AM
F54694777: screenshot 461.png
May 30 2024, 5:46 AM
F54694772: screenshot 460.png
May 30 2024, 5:46 AM
F54694313: screenshot 459.png
May 30 2024, 5:46 AM
F54693753: screenshot 458.png
May 30 2024, 5:30 AM
F54691977: screenshot 455.png
May 30 2024, 5:30 AM

Description

Background

In T354889 Vector 2022 will add a night theme. We will want to explore every nook and cranny of the interface to identify any issues that were not problems in the mobile version of the site.

User story

As a product manager I want to understand how ready Vector 2022's night theme is before immediate release.

Acceptance criteria

  • Document any issues found

QA steps

On testwiki:

  • Turn on dark mode beta feature (under accessibility for reading)
  • Explore UI
  • Flag any issues with color contrast issues either in content or interface.

Sign off steps

  • Create tickets for any issues found during QA.

QA Results - Prod

ACStatusAreaDetails
1User Authentication and ManagementT363780#9844692
2Content ManagementT363780#9844863
3Navigation and SearchT363780#9844869
4Media HandlingT363780#9844909
5Discussion and CollaborationT363780#9844936
6LanguageT363780#9844963
7Visual ConsistencyT363780#9845011
8Content DisplayT363780#9845020
9Forms and InputsT363780#9845023
10Notifications and MessagesT363780#9845044
11User SettingsT363780#9845048
12User ProfileT363780#9845050
13User LinksT363780#9845054
14InfoboxesT363780#9845081
15MiscellaneousT363780#9845083

Event Timeline

Jdlrobson lowered the priority of this task from High to Medium.May 29 2024, 11:18 PM
Test Result - Prod (#1 of 15)

Status: ✅ PASS
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 1: User Authentication and Management

  1. Login/Logout
  2. Account Creation
  3. Password Reset
  4. Preferences
  5. AC1: There are no dark mode related issues.
a
screenshot 391.png (1×1 px, 171 KB)
b
screenshot 389.png (1×1 px, 177 KB)
c
screenshot 388.png (1×1 px, 184 KB)
d
screenshot 390.png (1×1 px, 266 KB)
Test Result - Prod (#2 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 2: Content Management

  1. Creating, editing, and deleting pages.
  2. Page history and revision control.
  3. Wiki markup and visual editor.
  4. Verify content readability and correct display.
  5. ❌❓ AC2: Content is readable and correctly displayed in dark mode.
aThe version notice sometimes appears different, for example in the VE screenshots in TestCase 2 (AC2)
screenshot 403.png (1×1 px, 331 KB)
screenshot 402.png (1×1 px, 329 KB)
screenshot 400.png (1×1 px, 325 KB)
screenshot 401.png (1×1 px, 331 KB)
b
screenshot 404.png (1×1 px, 255 KB)
This impacts all of the similar disabled buttons in VE
c
screenshot 408.png (1×1 px, 272 KB)
d
screenshot 409.png (1×1 px, 277 KB)
e
screenshot 410.png (1×1 px, 282 KB)
f
screenshot 411.png (1×1 px, 264 KB)
g
screenshot 412.png (1×1 px, 265 KB)
h
screenshot 413.png (1×1 px, 301 KB)
i
screenshot 414.png (1×1 px, 267 KB)
j
screenshot 415.png (1×1 px, 489 KB)
k
screenshot 416.png (1×1 px, 454 KB)
l
screenshot 417.png (1×1 px, 445 KB)
m
screenshot 418.png (1×1 px, 449 KB)
n
screenshot 419.png (1×1 px, 376 KB)
o
screenshot 420.png (1×1 px, 466 KB)
p
screenshot 420.png (1×1 px, 466 KB)
q
screenshot 421.png (1×1 px, 543 KB)
r
screenshot 422 question.png (1×1 px, 507 KB)
s
screenshot 422 question 2.png (1×1 px, 532 KB)
t
screenshot 422.png (1×1 px, 725 KB)
u
screenshot 423.png (1×1 px, 520 KB)
v
screenshot 425.png (919×1 px, 665 KB)
w
screenshot 424.png (919×1 px, 508 KB)
Test Result - Prod (#3 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 3: Navigation and Search

  1. Main page, category, and special pages navigation.
  2. Search functionality.
  3. Check visibility and usability of navigation menus and search bar.
  4. Verify appearance and functionality of navigation menus, sidebars, and dropdowns.
  5. Ensure hover and active states are distinguishable.
  6. Test collapsible navigation elements.
  7. AC3: Navigation elements are visible and usable, with no visual issues in dark mode.
a
screenshot 397.png (1×1 px, 585 KB)
b
screenshot 399.png (1×1 px, 628 KB)
c
screenshot 396.png (1×1 px, 620 KB)
d
screenshot 395.png (1×1 px, 677 KB)
e
screenshot 398.png (1×1 px, 802 KB)
f
screenshot 92.mov.gif (1×1 px, 2 MB)
Test Result - Prod (#4 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 4: Media Handling

  1. Uploading and managing images, videos, and other files.
  2. File versioning and metadata.
  3. Ensure media display correctly.
  4. AC4: Media files are displayed correctly and are not obscured in dark mode.
a
screenshot 435.png (898×1 px, 273 KB)
The text is also not white on white so it doesn't show up. Here it is highlighted. //
screenshot 436.png (898×1 px, 288 KB)
b
screenshot 427.png (913×1 px, 276 KB)
c
screenshot 429.png (898×1 px, 204 KB)
d
screenshot 430.png (898×1 px, 218 KB)
e
screenshot 431.png (898×1 px, 187 KB)
The text is also not white on white so it doesn't show up. Here it is highlighted.
screenshot 432.png (898×1 px, 193 KB)
f
screenshot 433.png (898×1 px, 230 KB)
screenshot 434.png (898×1 px, 229 KB)
g
screenshot 437.png (898×1 px, 202 KB)
h
screenshot 438.png (898×1 px, 225 KB)
i
screenshot 439.png (898×1 px, 223 KB)
j
screenshot 440.png (898×1 px, 230 KB)

I stopped at this point, this page is not dark mode on enwiki so I wasn't sure if I should continue. Ultimately a number of the license types get similar errors.

This issue is unrelated to dark mode. A few of the license types have odd strings appear similar to this. This isn't the case on enwiki

k
screenshot 428.png (898×1 px, 251 KB)
Test Result - Prod (#5 of 15)

Status: ❌ FAIL / ❓ Need More Info
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 5: Discussion and Collaboration

  1. Talk pages and discussion threads.
  2. Notifications and watchlist.
  3. Test display and readability of notifications and discussion threads.
  4. AC5: Notifications and discussion threads are readable and correctly displayed.
a
screenshot 443.png (1×1 px, 897 KB)
b
screenshot 444.png (1×1 px, 841 KB)

Not much here, Watchlists, and it appears the notification popup are excluded from dark mode. There are some WCAG issues on the notifications but those are the same in light mode. Here they are for reference.

c
screenshot 441.png (1×1 px, 973 KB)
d
screenshot 442.png (850×1 px, 776 KB)
Test Result - Prod (#6 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 6: Language

  1. AC6: Dark mode is applied consistently in the ULS.
a
screenshot 445.png (898×1 px, 444 KB)
b
screenshot 446.png (898×1 px, 464 KB)
c
screenshot 449.png (898×1 px, 320 KB)
Test Result - Prod (#7 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 7: Visual Consistency

  1. Check appearance of text, links, buttons, and other UI elements.
  2. Verify color contrasts and readability.
  3. AC7: Text and UI elements are visually consistent and readable.
a
screenshot 450.png (898×1 px, 385 KB)
b
screenshot 451.png (898×1 px, 706 KB)
c
screenshot 452.png (898×1 px, 307 KB)
Test Result - Prod (#8 of 15)

Status: ✅ PASS
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 8: Content Display

  1. Ensure images, videos, and other media display correctly.
  2. Check for visual artifacts or misalignment.
  3. AC8: Content is displayed correctly with no visual artifacts.
aNo issues observed
Test Result - Prod (#9 of 15)

Status: ✅ PASS
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 9: Forms and Inputs

  1. Test input fields, text areas, checkboxes, radio buttons, and other form elements.
  2. Verify placeholder text visibility and overall form usability.
  3. AC9: Forms and inputs are usable and visible in dark mode.
aNo issues observed
Test Result - Prod (#10 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 10: Notifications and Messages

  1. Check display of notifications, alerts, and error messages.
  2. Ensure readability and distinguishability from the background.
  3. AC10: Notifications and messages are readable and distinguishable.
a
screenshot 454.png (898×1 px, 278 KB)
screenshot 453.png (898×1 px, 262 KB)
screenshot 455.png (898×1 px, 252 KB)
screenshot 458.png (898×1 px, 529 KB)
Test Result - Prod (#11 of 15)

Status: ✅ PASS
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 11: User Settings

  1. Ensure users can easily find and change the dark mode setting in their preferences.
  2. Test if dark mode respects user system settings (if applicable).
  3. AC11: Dark mode settings are easily accessible and respected.
aNo issues observed
Test Result - Prod (#12 of 15)

Status: ✅ PASS
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 12: User Profile

  1. User profile page and related functionality.
  2. AC12: User profile page displays correctly in dark mode.
aNo issues observed
Test Result - Prod (#13 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 13: User Links

  1. User tools links in the top navigation (e.g., "My Talk", "Preferences").

2.❌AC13: User links are visible and functional in dark mode.

a
screenshot 422 usr menu.png (1×1 px, 596 KB)
Test Result - Prod (#14 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 14: Infoboxes

  1. Ensure Infoboxes render correctly in dark mode.
  2. Check for visual consistency and readability of text within Infoboxes.
  3. Verify that borders, backgrounds, and other styling elements are properly adapted to dark mode.
  4. Ensure all content within Infoboxes (text, images, links) is visible and legible.
  5. Test any interactive elements within Infoboxes, such as collapsible sections or links.
  6. Verify consistent styling and rendering of Infoboxes across different pages and namespaces.
  7. AC14: Infoboxes are correctly rendered and visually consistent in dark mode.
a
screenshot 459.png (898×1 px, 397 KB)
b
screenshot 460.png (898×1 px, 436 KB)
c
screenshot 461.png (1×1 px, 488 KB)
Test Result - Prod (#15 of 15)

Status: ❌ FAIL
Environment: Prod
OS: macOS Sonoma
Browser: Chrome
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 15: Miscellaneous

  1. AC15: User profile page displays correctly in dark mode.
a
screenshot 405.png (1×1 px, 307 KB)
screenshot 406.png (1×1 px, 310 KB)
screenshot 407.png (1×1 px, 295 KB)
b
screenshot 456.png (898×1 px, 264 KB)
c
screenshot 457.png (898×1 px, 467 KB)
Edtadros subscribed.

From T363780#9844863

❌❓ AC2: Content is readable and correctly displayed in dark mode.

a

b, c, d, f, g, l, r, u

  • This is tracked in T366197 and can be QAed as part of that task.

e, h
Fixed on wiki and added to https://www.mediawiki.org/wiki/Recommendations_for_night_mode_compatibility_on_Wikimedia_wikis#Templates_/_Modules

g, i, j, n
Issues with editor. Now tracked in T366336

k:
On wiki issue. Out of scope for team.

m, o, p
These seem like false positives to me.

q:
Opened T366337

t:
This is T365280

v:
This is T365089

w:
This is T363878

From T363780#9845054

2.❌AC13: User links are visible and functional in dark mode.

Tracked in T365580

From T363780#9845044

❌AC10: Notifications and messages are readable and distinguishable.

All of these are covered by T364798.

From T363780#9845081

❌AC14: Infoboxes are correctly rendered and visually consistent in dark mode.

These are all out of scope for web team and we need to defer to editors to fix these sort of problems.
(I did personally fix the cricket example on wiki https://en.wikipedia.org/w/index.php?title=Template%3ACricket_sidebar&diff=1226520938&oldid=1183270385 in volunteer capacity)

From T363780#9845011

❌ AC7: Text and UI elements are visually consistent and readable.

The coordinates icon likely needs replacing. I left a comment on the associated talk page: https://en.wikipedia.org/w/index.php?title=Template_talk%3ACoord&oldid=1226522348#c-Jdlrobson-20240531045500-Template_is_not_compatible_with_night_mode

The Wiktionary logo is already covered in T365622

Thanks for flagging the help icon - I opened T366358.

From T363780#9844869

❌AC3: Navigation elements are visible and usable, with no visual issues in dark mode.

a: Also an issue in the light theme, so out of scope for web team. Must be fixed on wiki.
b: Tracked in T365990
e: False positive

From T363780#9844909

❌AC4: Media files are displayed correctly and are not obscured in dark mode.

a: This is now fixed.
b: Tracked in T361717
c,d: Out of scope.
e,f,g,h,i,j,k: Out of scope, but added example https://www.mediawiki.org/wiki/Recommendations_for_night_mode_compatibility_on_Wikimedia_wikis#Templates_/_Modules

From T363780#9844936

❌AC5: Notifications and discussion threads are readable and correctly displayed.

a,b: To be reassessed in T365383
c,d: Tracked in T366322

From T363780#9845083

❌AC15: User profile page displays correctly in dark mode.

a: Tracked in T365383
b: Good catch! Opened T366362
c: No longer appears to be a problem.

Screenshot 2024-05-31 at 8.37.46 AM.png (396×1 px, 103 KB)

That's a wrap. To summarize the above:

The QA resulted in uncovering bugs with certain banners; help icon and contribution icons ULS and language button ; ULS; printing to PDF and provided many QA examples for testing the OOUI to CSS variables migration.

T366197, T366336, T366337, T365280, T365089, T363878, T365990, T364798, T365622, T366358, T365990. T361717, T366322, T365383, T366362