Page MenuHomePhabricator
Create Task
Maniphest T228311

Find key patterns in Redis keys that indicate components using MainStash
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

We should be able to map key patterns in Redis to components in MediaWiki that use MainStash that we identified in T228309.

This task will involve dumping the keys from Redis, grouping them by some characteristic (usually a namespacing prefix followed by ":") and matching those to components we know about from code analysis.

If there are patterns that don't match components we know about, we need to identify those components.

Related Objects
Search...

Event Timeline

EvanProdromou created this task.Jul 17 2019, 5:01 PM
EvanProdromou edited projects, added Platform Team Workboards (Green); removed Platform Team Workboards.Jul 22 2019, 1:59 PM
EvanProdromou added a comment.Jul 22 2019, 2:04 PM

For reference, we need more of these!

https://phabricator.wikimedia.org/T212129#4948690

WDoranWMF moved this task from Backlog to Next Sprint on the Platform Team Workboards (Green) board.Jul 22 2019, 2:15 PM
WDoranWMF moved this task from Next Sprint to Create Engineering Task on the Platform Team Workboards (Green) board.Jul 22 2019, 2:36 PM
WDoranWMF reassigned this task from EvanProdromou to holger.knust.Jul 22 2019, 2:40 PM
WDoranWMF triaged this task as Medium priority.
WDoranWMF added a subscriber: WDoranWMF.

As reference, this was done previously for the top 7 parameters
https://phabricator.wikimedia.org/T212129#4948690

EvanProdromou added a subscriber: jijiki.Jul 22 2019, 2:41 PM

It would be helpful if @jijiki were able to share their tools or code (if any) that they did for that previous analysis.

holger.knust mentioned this in T228703: Keys from MediaWiki Redis Instances.Jul 22 2019, 9:50 PM
WDoranWMF set the point value for this task to 5.Jul 23 2019, 1:06 PM
EvanProdromou added a comment.Jul 23 2019, 1:18 PM

One algorithm for this might be the following:

  • Make a hashtable for the pattern counts, mapping a pattern string to an integer
  • For each line:
    • pattern = line
    • Replace substrings in pattern that match the code for of one of our wikis (like "en_wp") with "(WIKI)"
    • Replace everything after the last colon in pattern with "(ID)"
    • (Other transforms here)
    • Increment the count for this pattern in the big hashtable (or set to 1 if not exists)
  • Dump comma-separated values (CSV) of hashtable to stdout as "count,pattern"

We can stop adding transforms when the resulting dumped CSV is less than 1000 lines long. Any dupes after that I can deal with manually.

WDoranWMF moved this task from Create Engineering Task to User Stories in Progress on the Platform Team Workboards (Green) board.Jul 23 2019, 10:09 PM
WDoranWMF moved this task from User Stories in Progress to Ready on the Platform Team Workboards (Green) board.
holger.knust moved this task from Ready to Doing on the Platform Team Workboards (Green) board.Jul 24 2019, 1:07 PM
EvanProdromou closed this task as Resolved.Jul 25 2019, 6:45 PM
WDoranWMF moved this task from Doing to Done on the Platform Team Workboards (Green) board.Nov 11 2019, 4:53 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL