Follow the procedure at https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Adding_a_wiki
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T414237 Post-creation work for kaiwiki | |||
| Resolved | BTullis | T414240 [wikireplicas] Create views for new wiki kaiwiki | |||
| Resolved | FCeratto-WMF | T414236 Prepare and check storage layer for kaiwiki |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-01-27T13:48:08Z] <fnegri@cumin1003> START - Cookbook sre.wikireplicas.add-wiki for database kaiwiki (T414240)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-01-27T13:48:18Z] <fnegri@cumin1003> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database kaiwiki (T414240)
Comment Actions
I used --skip-dns as I expected the DNS record had been created by T415041: [wikireplicas] Create views for new wiki kajwiki, but it wasn't :/
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-12T13:14:07Z] <fnegri@cumin1003> START - Cookbook sre.wikireplicas.add-wiki for database kaiwiki (T414240)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-12T13:14:11Z] <fnegri@cumin1003> END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) for database kaiwiki (T414240)
Comment Actions
Views were created on all clouddb*, but the cookbook failed halfway through, so I ran manually maintain-meta_p on clouddb1014 and clouddb1018 and ran wmcs-wikireplica-dns on cloudcontrol1007.
@BTullis the add-wiki cookbook failed on an-redacteddb1001:
===== NODE GROUP =====
(1) an-redacteddb1001.eqiad.wmnet
----- OUTPUT -----
Traceback (most recent call last):
File "/usr/bin/maintain-replica-indexes", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3/dist-packages/wikireplicas_utils/maintain_replica_indexes.py", line 176, in main
db_connections[instance] = pymysql.connect(
^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 353, in __init__
self.connect()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 633, in connect
self._request_authentication()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 907, in _request_authentication
auth_packet = self._read_packet()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user 'maintainindexes'@'localhost' (using password: YES)")After the permissions are fixed, you'll need to run this manually on an-redacteddb1001:
/usr/bin/maintain-replica-indexes --database kaiwiki /usr/bin/maintain-views --replace --databases kaiwiki /usr/bin/maintain-meta_p --databases kaiwiki
Comment Actions
Thanks @fnegri.
I have now run those commands.
btullis@an-redacteddb1001:~$ sudo /usr/bin/maintain-replica-indexes --database kaiwiki btullis@an-redacteddb1001:~$ sudo /usr/bin/maintain-views --replace --databases kaiwiki 2026-03-13 17:43:54,546 INFO Full views for s5.kaiwiki: 2026-03-13 17:43:54,548 INFO [s5.kaiwiki_p.site_identifiers] 2026-03-13 17:43:54,552 INFO [s5.kaiwiki_p.updatelog] 2026-03-13 17:43:54,555 INFO [s5.kaiwiki_p.user_former_groups] 2026-03-13 17:43:54,560 INFO [s5.kaiwiki_p.site_stats] 2026-03-13 17:43:54,564 INFO [s5.kaiwiki_p.change_tag] 2026-03-13 17:43:54,567 INFO [s5.kaiwiki_p.l10n_cache] 2026-03-13 17:43:54,570 INFO [s5.kaiwiki_p.redirect] 2026-03-13 17:43:54,574 INFO [s5.kaiwiki_p.pagelinks] 2026-03-13 17:43:54,578 INFO [s5.kaiwiki_p.templatelinks] 2026-03-13 17:43:54,582 INFO [s5.kaiwiki_p.imagelinks] 2026-03-13 17:43:54,585 INFO [s5.kaiwiki_p.langlinks] 2026-03-13 17:43:54,590 INFO [s5.kaiwiki_p.iwlinks] 2026-03-13 17:43:54,593 INFO [s5.kaiwiki_p.existencelinks] 2026-03-13 17:43:54,598 INFO [s5.kaiwiki_p.category] 2026-03-13 17:43:54,601 INFO [s5.kaiwiki_p.change_tag_def] 2026-03-13 17:43:54,605 INFO [s5.kaiwiki_p.ipblocks_restrictions] 2026-03-13 17:43:54,609 INFO [s5.kaiwiki_p.page_restrictions] 2026-03-13 17:43:54,613 INFO [s5.kaiwiki_p.user_groups] 2026-03-13 17:43:54,617 INFO [s5.kaiwiki_p.sites] 2026-03-13 17:43:54,621 INFO [s5.kaiwiki_p.interwiki] 2026-03-13 17:43:54,624 INFO [s5.kaiwiki_p.protected_titles] 2026-03-13 17:43:54,628 INFO [s5.kaiwiki_p.externallinks] 2026-03-13 17:43:54,631 INFO [s5.kaiwiki_p.page_props] 2026-03-13 17:43:54,637 INFO [s5.kaiwiki_p.slot_roles] 2026-03-13 17:43:54,641 INFO [s5.kaiwiki_p.content_models] 2026-03-13 17:43:54,645 INFO [s5.kaiwiki_p.categorylinks] 2026-03-13 17:43:54,649 INFO [s5.kaiwiki_p.page] 2026-03-13 17:43:54,653 INFO [s5.kaiwiki_p.user_autocreate_serial] 2026-03-13 17:43:54,657 INFO [s5.kaiwiki_p.searchindex] 2026-03-13 17:43:54,662 INFO [s5.kaiwiki_p.filetypes] 2026-03-13 17:43:54,666 INFO [s5.kaiwiki_p.collation] 2026-03-13 17:43:54,671 INFO [s5.kaiwiki_p.babel] 2026-03-13 17:43:54,739 INFO [s5.kaiwiki_p.geo_tags] 2026-03-13 17:43:54,744 INFO [s5.kaiwiki_p.global_block_whitelist] 2026-03-13 17:43:54,760 INFO [s5.kaiwiki_p.linter] 2026-03-13 17:43:54,779 INFO [s5.kaiwiki_p.transcode] 2026-03-13 17:43:54,813 INFO [s5.kaiwiki_p.wbc_entity_usage] 2026-03-13 17:43:54,827 INFO Custom views for s5.kaiwiki: 2026-03-13 17:43:54,830 INFO [s5.kaiwiki_p.abuse_filter] 2026-03-13 17:43:54,836 INFO [s5.kaiwiki_p.abuse_filter_action] 2026-03-13 17:43:54,843 INFO [s5.kaiwiki_p.abuse_filter_history] 2026-03-13 17:43:54,848 INFO [s5.kaiwiki_p.actor] 2026-03-13 17:43:54,857 INFO [s5.kaiwiki_p.actor_user] 2026-03-13 17:43:54,862 INFO [s5.kaiwiki_p.actor_archive] 2026-03-13 17:43:54,867 INFO [s5.kaiwiki_p.actor_block] 2026-03-13 17:43:54,871 INFO [s5.kaiwiki_p.actor_ipblocks] 2026-03-13 17:43:54,876 INFO [s5.kaiwiki_p.actor_image] 2026-03-13 17:43:54,879 INFO [s5.kaiwiki_p.actor_oldimage] 2026-03-13 17:43:54,883 INFO [s5.kaiwiki_p.actor_filearchive] 2026-03-13 17:43:54,887 INFO [s5.kaiwiki_p.actor_recentchanges] 2026-03-13 17:43:54,893 INFO [s5.kaiwiki_p.actor_logging] 2026-03-13 17:43:54,897 INFO [s5.kaiwiki_p.actor_revision] 2026-03-13 17:43:54,902 INFO [s5.kaiwiki_p.archive] 2026-03-13 17:43:54,913 INFO [s5.kaiwiki_p.archive_compat] 2026-03-13 17:43:54,918 INFO [s5.kaiwiki_p.archive_userindex] 2026-03-13 17:43:54,922 INFO [s5.kaiwiki_p.block] 2026-03-13 17:43:54,927 INFO [s5.kaiwiki_p.block_target] 2026-03-13 17:43:54,933 INFO [s5.kaiwiki_p.block_target_ipindex] 2026-03-13 17:43:54,938 INFO [s5.kaiwiki_p.comment] 2026-03-13 17:43:54,943 INFO [s5.kaiwiki_p.comment_block] 2026-03-13 17:43:54,948 INFO [s5.kaiwiki_p.comment_image] 2026-03-13 17:43:54,953 INFO [s5.kaiwiki_p.comment_filearchive] 2026-03-13 17:43:54,958 INFO [s5.kaiwiki_p.comment_ipblocks] 2026-03-13 17:43:54,965 INFO [s5.kaiwiki_p.comment_oldimage] 2026-03-13 17:43:54,970 INFO [s5.kaiwiki_p.comment_protected_titles] 2026-03-13 17:43:54,974 INFO [s5.kaiwiki_p.comment_recentchanges] 2026-03-13 17:43:54,979 INFO [s5.kaiwiki_p.comment_revision] 2026-03-13 17:43:54,984 INFO [s5.kaiwiki_p.comment_logging] 2026-03-13 17:43:54,989 INFO [s5.kaiwiki_p.content] 2026-03-13 17:43:54,993 INFO [s5.kaiwiki_p.file] 2026-03-13 17:43:54,998 INFO [s5.kaiwiki_p.filerevision] 2026-03-13 17:43:55,004 INFO [s5.kaiwiki_p.filearchive] 2026-03-13 17:43:55,013 INFO [s5.kaiwiki_p.filearchive_compat] 2026-03-13 17:43:55,019 INFO [s5.kaiwiki_p.filearchive_userindex] 2026-03-13 17:43:55,031 INFO [s5.kaiwiki_p.ip_changes] 2026-03-13 17:43:55,037 INFO [s5.kaiwiki_p.image] 2026-03-13 17:43:55,045 INFO [s5.kaiwiki_p.image_compat] 2026-03-13 17:43:55,051 INFO [s5.kaiwiki_p.linktarget] 2026-03-13 17:43:55,059 INFO [s5.kaiwiki_p.logging] 2026-03-13 17:43:55,067 INFO [s5.kaiwiki_p.logging_compat] 2026-03-13 17:43:55,073 INFO [s5.kaiwiki_p.logging_logindex] 2026-03-13 17:43:55,079 INFO [s5.kaiwiki_p.logging_userindex] 2026-03-13 17:43:55,085 INFO [s5.kaiwiki_p.oldimage] 2026-03-13 17:43:55,094 INFO [s5.kaiwiki_p.oldimage_compat] 2026-03-13 17:43:55,099 INFO [s5.kaiwiki_p.oldimage_userindex] 2026-03-13 17:43:55,111 INFO [s5.kaiwiki_p.page_compat] 2026-03-13 17:43:55,118 INFO [s5.kaiwiki_p.protected_titles_compat] 2026-03-13 17:43:55,123 INFO [s5.kaiwiki_p.recentchanges] 2026-03-13 17:43:55,131 INFO [s5.kaiwiki_p.recentchanges_compat] 2026-03-13 17:43:55,137 INFO [s5.kaiwiki_p.recentchanges_userindex] 2026-03-13 17:43:55,142 INFO [s5.kaiwiki_p.revision] 2026-03-13 17:43:55,156 INFO [s5.kaiwiki_p.revision_compat] 2026-03-13 17:43:55,162 INFO [s5.kaiwiki_p.revision_userindex] 2026-03-13 17:43:55,170 INFO [s5.kaiwiki_p.slots] 2026-03-13 17:43:55,175 INFO [s5.kaiwiki_p.user] 2026-03-13 17:43:55,182 INFO [s5.kaiwiki_p.user_properties] 2026-03-13 17:43:55,184 INFO Skipping user_properties_anon: DB size 1 < required 2. btullis@an-redacteddb1001:~$ sudo /usr/bin/maintain-meta_p --databases kaiwiki
I see that the authentication problems have come about because of the preliminary work that I did on T407485: Set up x1 replication to an-redacteddb1001
Where I left it was that we run an x1 section, but it has no content and is missing the correct grants.
However, puppet has added entries for this section into /etc/maintain-views.yaml and /etc/index-conf.yaml.
When attempting to run the commands above, they try to use the x1 section when it is not ready.
I had to remove these entries temporarily, before they would run.
I will close this ticket, since this wiki has been created, but the problem will come back so I will try to fix it under T407485.