Make wb_terms table fancy
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Mar 6 2018, 12:47 PM

Description

This table is 816 GB in production with 1.4 B rows. This has to change.

Investigate ways to optimize the table (T188279)
Replace term_search_key and term_weight with empty values (T188993)
Clean up the code base for term_entity_id (T188995)
Drop indexes (T194273 T194270)
Drop term_entity_id column
Normalize term_type column
Normalize term_text column

Related Objects
Search...

Status	Assigned	Task
Resolved	Addshore	T188992 Make wb_terms table fancy
Resolved	Lucas_Werkmeister_WMDE	T188993 Replace term_search_key and term_weight with empty values when wb_terms is not used for search
Resolved	Ladsgroup	T189776 Disable reading from term_search_key from wb_terms table in beta cluster and testwikidata
Resolved	Lucas_Werkmeister_WMDE	T189777 Disable reading from term_search_key from wb_terms table in wikidata
Declined	Ladsgroup	T189779 Run clearTermSqlIndexSearchFields on Wikidata
Resolved	Lucas_Werkmeister_WMDE	T191631 Add maintenance script to wipe term_search_key and term_weight columns
Resolved	Ladsgroup	T188995 Clean up the code base for term_entity_id
Resolved	None	T188279 Investigate optimzing wb_terms
Resolved	Ladsgroup	T191391 Apply schema changes to an isolated database and examine the results
Resolved	jcrespo	T196047 reimage db2083 back into wikidata (s8)
Declined	None	T191626 Remove term_entity_type from wb_terms
Resolved	• Marostegui	T194270 Drop 'tmp1' index from wb_terms table in production
Declined	Ladsgroup	T194273 Clean up indexes of wb_terms table
Resolved	Lucas_Werkmeister_WMDE	T195485 Don’t select term_entity_type in TermSqlIndex
Resolved	Lydia_Pintscher	T195490 Use CirrusSearch for PropertySuggester

Event Timeline

Ladsgroup created this task.Mar 6 2018, 12:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2018, 12:47 PM

Ladsgroup updated the task description. (Show Details)Mar 6 2018, 12:48 PM

Ladsgroup added a subtask: T188279: Investigate optimzing wb_terms.Mar 6 2018, 12:54 PM

Ladsgroup updated the task description. (Show Details)

Ladsgroup added a project: DBA.

Ladsgroup moved this task from Triage to Blocked external/Not db team on the DBA board.

Normalize term_text column

Should we normalize the text on its own or the tuple of text and language code? I feel like treating the text and language code as a single unit would make more sense, but I’m not sure if it makes a big difference.

It actually makes a big difference as there are lots of cases when someone adds label for all languages at the same time, for humans. Here's an interesting example.