Bring labswiki database tables up to date
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Reedy
	Jun 1 2021, 8:39 PM

Description

It seems labswiki has numerous long defunct (uninstalled extension, or no longer part of MW core) tables, and probably has a few schema drifts too along the way.

table	needed	to replicate?	comment
abuse_filter	yes
abuse_filter_action	yes
abuse_filter_history	yes
abuse_filter_log	yes
actor	yes
archive	yes
babel	yes
betafeatures_user_counts	yes
bv2017_edits		no	data is presumably no longer relevant
category	yes
categorylinks	yes
change_tag	yes
change_tag_def	yes
comment	yes
config	no
content	yes
content_models	yes
cu_changes	yes
cu_log	yes
echo_email_batch	yes
echo_event	yes
echo_notification	yes
echo_target_page	yes
external_user	no
externallinks	yes
filearchive	yes
flow_ext_ref	no		dropped
flow_revision	no		dropped
flow_subscription	no		dropped
flow_topic_list	no		dropped
flow_tree_node	no		dropped
flow_tree_revision	no		dropped
flow_wiki_ref	no		dropped
flow_workflow	no		dropped
geo_tags			Shouldn't be needed (GeoData not installed)
global_block_whitelist	yes		Will be needed when wiki is SUL'd
globalblocks			Shouldn't be needed (only for "central wiki")
globalimagelinks			Shouldn't be needed (only for "central wiki")
hidden	no		dropped
historical_thread			LQT
image	yes
imagelinks	yes
interwiki	yes
ip_changes	yes
ipblocks	yes
ipblocks_restrictions	yes
iwlinks	yes
job	yes
l10n_cache	yes
langlinks	yes
ldap_domains	yes
linter	yes
log_search	yes
logging	yes
mathoid	yes
module_deps	yes
msg_resource	no		dropped
msg_resource_links	no		dropped
oaiaudit	no		dropped
oaiharvest	no		dropped
oaiuser	no		dropped
oathauth_users	yes
oathauth_users_restore	yes
oauth2_access_tokens	yes
oauth_accepted_consumer	yes
oauth_registered_consumer	yes
objectcache	yes
oldimage	yes
openstack_notification_event	no		dropped
openstack_puppet_classes	no		dropped
openstack_puppet_groups	no		dropped
openstack_puppet_vars	no		dropped
openstack_tokens	no		dropped
page	yes
page_props	yes
page_restrictions	yes
pagelinks	yes
protected_titles	yes
querycache	yes
querycache_info	yes
querycachetwo	yes
recentchanges	yes
redirect	yes
revision	yes
revision_actor_temp	yes		T215466
revision_comment_temp	yes		T215466
searchindex	yes
site_identifiers	yes
site_stats	yes
sites	yes
slot_roles	yes
slots	yes
spoofuser	yes
templatelinks	yes
text	yes
thread			LQT
thread_history			LQT
thread_pending_relationship			LQT
thread_reaction			LQT
titlekey	no		dropped
trackbacks	no		dropped
transcache	no		dropped
updatelog	yes
updates	no		dropped
uploadstash	yes
user	yes
user_daily_contribs	no		dropped
user_former_groups	yes
user_groups	yes
user_message_state			LQT
user_newtalk	yes
user_openid	no		dropped
user_properties	yes
watchlist	yes
watchlist_expiry	yes
wikilove_log	yes

Related Objects
Search...

Status	Assigned	Task
Duplicate	None	T237771 Make Wikitech a normal wiki
Duplicate	None	T237772 Move Wikitech's database off of M5 and onto a normal production wiki DB server
Declined	None	T171570 Rename database 'labswiki' to 'wikitechwiki'
Resolved	Milimetric	T217792 Add wikitech (labswiki) to the sqoop list
		Restricted Task
Resolved	Legoktm	T281515 June 2021 Datacenter switchover
Resolved	Marostegui	T285139 DB maintenance work to do while eqiad is passive (June 2021)
Open	None	T189531 All Wikimedia developer services should use single sign-on
Resolved	None	T161859 Make Wikitech an SUL wiki
Duplicate	None	T237773 Move Wikitech onto the production MW cluster
Resolved	Marostegui	T167973 Move database for wikitech (labswiki) to a main cluster section
Resolved	Andrew	T284106 Replicate & sanitize wikitech data
Resolved	Andrew	T284108 Bring labswiki database tables up to date
Resolved	Ladsgroup	T269348 wikitech database has almost all of its varbinary fields wrong

Event Timeline

Reedy created this task.Jun 1 2021, 8:39 PM

Reedy updated the task description. (Show Details)

Ladsgroup subscribed.Jun 1 2021, 8:45 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 8:57 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 9:01 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 9:05 PM

Tables marked <dropped> are backed up temporarily on db1128 in /home/andrew/wikitechtables

Mentioned in SAL (#wikimedia-operations) [2021-06-01T21:09:49Z] <andrewbogott> dropping a bunch of tables from the labswiki db as per T284108

Andrew updated the task description. (Show Details)Jun 1 2021, 9:13 PM

Andrew updated the task description. (Show Details)

Andrew updated the task description. (Show Details)Jun 1 2021, 9:16 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 9:32 PM

Reedy updated the task description. (Show Details)

Reedy updated the task description. (Show Details)Jun 1 2021, 9:38 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 9:44 PM

Reedy updated the task description. (Show Details)Jun 1 2021, 9:49 PM

I know @Ladsgroup has filled some tasks regarding schema drifts on wikitech: T269348

Marostegui added a subtask: T269348: wikitech database has almost all of its varbinary fields wrong.Jun 2 2021, 4:43 AM

LSobanski subscribed.Jun 2 2021, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:19:53Z] <Amir1> setting charset of all tables in wikitech to binary (T284108 T269348)

btw I noticed this during the change, the actor table is empty, you need to run its migration script I think.

Ladsgroup closed subtask T269348: wikitech database has almost all of its varbinary fields wrong as Resolved.Jun 2 2021, 10:28 PM

In T284108#7130751, @Ladsgroup wrote:

btw I noticed this during the change, the actor table is empty, you need to run its migration script I think.

MariaDB [labswiki]> select count(*) from actor;
+----------+
| count(*) |
+----------+
|    25902 |
+----------+
1 row in set (0.015 sec)

oh my bad. It was rows affected and it was zero because I already set actor to binary charset. Sorry.

Andrew updated the task description. (Show Details)Jun 7 2021, 2:15 PM

Andrew updated the task description. (Show Details)Jun 7 2021, 2:18 PM

It's my understanding that this data is now ready for T167973 -- please lmk if anyone disagrees, otherwise I think we can close this.

So, it is fine to proceed with the normal filtering we have for the other wikis or do we need extra ones?:
These are the tables we currently ignore:

replicate-wild-ignore-table = %.__wmf_checksums
replicate-wild-ignore-table = %.accountaudit_login
replicate-wild-ignore-table = %.arbcom1_vote
replicate-wild-ignore-table = %.archive_old
replicate-wild-ignore-table = %.blob_orphans
replicate-wild-ignore-table = %.blob_tracking
replicate-wild-ignore-table = %.bot_passwords
replicate-wild-ignore-table = %.bv2009_edits
replicate-wild-ignore-table = %.categorylinks_old
replicate-wild-ignore-table = %.click_tracking
replicate-wild-ignore-table = %.cu_changes
replicate-wild-ignore-table = %.cu_log
replicate-wild-ignore-table = %.cur
replicate-wild-ignore-table = %.discussiontools_subscription
replicate-wild-ignore-table = %.echo_email_batch
replicate-wild-ignore-table = %.echo_event
replicate-wild-ignore-table = %.echo_target_page
replicate-wild-ignore-table = %.echo_unread_wikis
replicate-wild-ignore-table = %.echo_notification
replicate-wild-ignore-table = %.echo_push_subscription
replicate-wild-ignore-table = %.edit_page_tracking
replicate-wild-ignore-table = %.email_capture
replicate-wild-ignore-table = %.exarchive
replicate-wild-ignore-table = %.exrevision
replicate-wild-ignore-table = %.globalnames
replicate-wild-ignore-table = %.growthexperiments_link_recommendations
replicate-wild-ignore-table = %.growthexperiments_link_submissions
replicate-wild-ignore-table = %.growthexperiments_mentor_mentee
replicate-wild-ignore-table = %.growthexperiments_mentee_data
replicate-wild-ignore-table = %.hidden
replicate-wild-ignore-table = %.image_old
replicate-wild-ignore-table = %.job
replicate-wild-ignore-table = %.linkscc
replicate-wild-ignore-table = %.localnames
replicate-wild-ignore-table = %.log_search
replicate-wild-ignore-table = %.logging_old
replicate-wild-ignore-table = %.long_run_profiling
replicate-wild-ignore-table = %.migrateuser_medium
replicate-wild-ignore-table = %.moodbar_feedback
replicate-wild-ignore-table = %.moodbar_feedback_response
replicate-wild-ignore-table = %.msg_resource
replicate-wild-ignore-table = %.oathauth_users
replicate-wild-ignore-table = %.oauth_accepted_consumer
replicate-wild-ignore-table = %.oauth_ratelimit_client_tier
replicate-wild-ignore-table = %.oauth_registered_consumer
replicate-wild-ignore-table = %.oauth2_access_tokens
replicate-wild-ignore-table = %.objectcache
replicate-wild-ignore-table = %.old_growth
replicate-wild-ignore-table = %.oldimage_old
replicate-wild-ignore-table = %.optin_survey
replicate-wild-ignore-table = %.prefstats
replicate-wild-ignore-table = %.prefswitch_survey
replicate-wild-ignore-table = %.profiling
replicate-wild-ignore-table = %.querycache
replicate-wild-ignore-table = %.querycache_info
replicate-wild-ignore-table = %.querycache_old
replicate-wild-ignore-table = %.querycachetwo
replicate-wild-ignore-table = %.reading_list
replicate-wild-ignore-table = %.reading_list_entry
replicate-wild-ignore-table = %.securepoll_cookie_match
replicate-wild-ignore-table = %.securepoll_elections
replicate-wild-ignore-table = %.securepoll_entity
replicate-wild-ignore-table = %.securepoll_lists
replicate-wild-ignore-table = %.securepoll_msgs
replicate-wild-ignore-table = %.securepoll_options
replicate-wild-ignore-table = %.securepoll_properties
replicate-wild-ignore-table = %.securepoll_questions
replicate-wild-ignore-table = %.securepoll_strike
replicate-wild-ignore-table = %.securepoll_voters
replicate-wild-ignore-table = %.securepoll_votes
replicate-wild-ignore-table = %.spoofuser
replicate-wild-ignore-table = %.text
replicate-wild-ignore-table = %.titlekey
replicate-wild-ignore-table = %.transcache
replicate-wild-ignore-table = %.translate_cache
replicate-wild-ignore-table = %.uploadstash
replicate-wild-ignore-table = %.urlshortcodes
replicate-wild-ignore-table = %.user_newtalk
replicate-wild-ignore-table = %.vote_log
replicate-wild-ignore-table = %.watchlist
replicate-wild-ignore-table = %.watchlist_expiry
replicate-wild-ignore-table = %.wikimedia_editor_tasks_counts
replicate-wild-ignore-table = %.wikimedia_editor_tasks_keys
replicate-wild-ignore-table = %.wikimedia_editor_tasks_targets_passed

ldap_domains should be the only non standard table

I'm struggling to see any reason it should be replicated

CREATE TABLE /*_*/ldap_domains (
  domain_id INT NOT NULL,
  domain VARCHAR(255) NOT NULL,
  user_id INT NOT NULL,
  INDEX user_id (user_id),
  PRIMARY KEY(domain_id)
) /*$wgDBTableOptions*/;

bv2017_edits shouldn't be either, and I suspect can just be dropped anyway (what do we do about the board vote tables on other wikis?)

Reedy updated the task description. (Show Details)Jun 7 2021, 3:15 PM

Reedy updated the task description. (Show Details)Jun 7 2021, 3:19 PM

Reedy updated the task description. (Show Details)Jun 7 2021, 3:21 PM

ldap_domains has been excluded from replication: T284106#7142455

I'm about to close this unless someone asks me not to :)

Andrew closed this task as Resolved.Jul 13 2021, 2:20 PM

Bring labswiki database tables up to dateClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Bring labswiki database tables up to date
Closed, ResolvedPublic
Actions

Related Objects
Search...