Page MenuHomePhabricator

schema_change_helper.py failure
Closed, ResolvedPublic

Description

[06:52:14] marostegui@cumin1003:~/git/T432200/schema-changes$ ./schema_change_helper.py 2026/add_ss_temp_users_T432200.py s6-codfw-replicas
Sequence:
s6 codfw replicas

[2026/add_ss_temp_users_T432200.py]
[s6 codfw replicas] Proceed on this section? Type y for yes, s to skip, ctrl-c to exit: yes

[2026/add_ss_temp_users_T432200.py]
[s6 codfw replicas] Proceed on this section? Type y for yes, s to skip, ctrl-c to exit: y
Starting new HTTPS connection (1): zarcillo.wikimedia.org:443
https://zarcillo.wikimedia.org:443 "POST /api/v1/acquire_lock HTTP/1.1" 200 62
[s6 codfw] Performing check on replicas
Running ['sudo', 'PYTHONPATH=../auto_schema/', 'python3', '-u', '2026/add_ss_temp_users_T432200.py', '--section', 's6', '--dc', 'codfw', '--check']
2026-07-21 06:52:42 INFO Starting schema change on db2158,db2169,db2180,db2193,db2197:3316,db2214,db2217,db2224
2026-07-21 06:52:42 INFO SQL of schema change:
ALTER TABLE /*_*/site_stats
  ADD ss_temp_users BIGINT UNSIGNED DEFAULT NULL,
  ADD ss_active_temp_users BIGINT UNSIGNED DEFAULT NULL;

2026-07-21 06:52:42 INFO Start of schema change sql on db2158
2026-07-21 06:52:42 INFO db-mysql db2158 -N -e "show slave hosts;"
2026-07-21 06:52:42 DEBUG Starting new HTTPS connection (1): noc.wikimedia.org:443
2026-07-21 06:52:42 DEBUG https://noc.wikimedia.org:443 "GET /conf/dblists/s6.dblist HTTP/1.1" 200 132
2026-07-21 06:52:43 INFO End of schema change sql on db2158
2026-07-21 06:52:43 INFO Start of schema change sql on db2169
2026-07-21 06:52:43 INFO db-mysql db2169 -N -e "show slave hosts;"
2026-07-21 06:52:44 INFO End of schema change sql on db2169
2026-07-21 06:52:44 INFO Start of schema change sql on db2180
2026-07-21 06:52:44 INFO db-mysql db2180 -N -e "show slave hosts;"
2026-07-21 06:52:45 INFO End of schema change sql on db2180
2026-07-21 06:52:45 INFO Start of schema change sql on db2193
2026-07-21 06:52:45 INFO db-mysql db2193 -N -e "show slave hosts;"
2026-07-21 06:52:45 INFO End of schema change sql on db2193
2026-07-21 06:52:45 INFO Start of schema change sql on db2197:3316
2026-07-21 06:52:45 INFO db-mysql db2197:3316 -N -e "show slave hosts;"
2026-07-21 06:52:46 INFO End of schema change sql on db2197:3316
2026-07-21 06:52:46 INFO Start of schema change sql on db2214
2026-07-21 06:52:46 INFO db-mysql db2214 -N -e "show slave hosts;"
2026-07-21 06:52:47 INFO End of schema change sql on db2214
2026-07-21 06:52:47 INFO Start of schema change sql on db2217
2026-07-21 06:52:47 INFO db-mysql db2217 -N -e "show slave hosts;"
2026-07-21 06:52:48 INFO End of schema change sql on db2217
2026-07-21 06:52:48 INFO Start of schema change sql on db2224
2026-07-21 06:52:48 INFO db-mysql db2224 -N -e "show slave hosts;"
2026-07-21 06:52:49 INFO End of schema change sql on db2224
2026-07-21 06:52:49 INFO End of schema change on db2158,db2169,db2180,db2193,db2197:3316,db2214,db2217,db2224
2026-07-21 06:52:49 INFO Result: {"needed in some dbs": ["db2158", "db2169", "db2180", "db2193", "db2197:3316", "db2214", "db2217", "db2224"]}
[2026/add_ss_temp_users_T432200.py]
[s6 codfw replicas] Removing section lock
Releasing lock by calling https://zarcillo.wikimedia.org/api/v1/release_lock/59381006-dae0-4ee2-be9c-926c7ff48039
Starting new HTTPS connection (1): zarcillo.wikimedia.org:443
https://zarcillo.wikimedia.org:443 "POST /api/v1/release_lock/59381006-dae0-4ee2-be9c-926c7ff48039 HTTP/1.1" 200 16
Lock released
Traceback (most recent call last):
  File "/home/marostegui/git/T432200/schema-changes/./schema_change_helper.py", line 291, in <module>
    main()
  File "/home/marostegui/git/T432200/schema-changes/./schema_change_helper.py", line 287, in main
    schema_change_section(section, dc, kind, args.path, args.check_only)
  File "/home/marostegui/git/T432200/schema-changes/./schema_change_helper.py", line 232, in schema_change_section
    res = perform_check(path, section, dc)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marostegui/git/T432200/schema-changes/./schema_change_helper.py", line 123, in perform_check
    return run_and_capture(path, section, dc, on_master=on_master)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marostegui/git/T432200/schema-changes/./schema_change_helper.py", line 115, in run_and_capture
    assert last.startswith("Result: ")
AssertionError

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update for new logging format from auto schemarepos/sre/schema-changes!67fcerattoT432682main
Customize query in GitLab

Event Timeline

As spoken on IRC, let's separate https://gitlab.wikimedia.org/repos/sre/schema-changes/-/merge_requests/64 into two. Because it not only fixes the bug but also introduces a new functionality.

FCeratto-WMF moved this task from Triage to Done on the DBA board.