Page MenuHomePhabricator

Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org
Closed, ResolvedPublic

Description

SMW isn't properly reviewed and is overkill for our use case for the most part.

Current usages:

For querying

  • Analytics/EventLogging: Calls {{#ask: [[Category:EventLogging/Incident documentation]] | format=ul | limit=50 | order=desc }}. This could easily be replaced with Extension:DynamicPageList_(Wikimedia)
  • Portal:Wikimedia_Labs: Calls SMW to compute some statistics. Does anybody care about them. If they do, maybe someone could write a bot.
  • Resource_query_examples: Just some examples. It can be deleted
  • Labs_Server_Admin_Log: Transcludes all the server admin logs for labs. Is this even useful to anyone?
  • Help:Tool_Labs/Web: {{#ask: [[Resource Type::instance]][[Project::tools]][[Instance Name::~*proxy*]] | ?Private IP |format=list |headers=hide }} - this could probably be replaced with a manually updated list, or converted to categories and use DynamicPageList.
  • Nova Resource:Tools/Documentation: {{#ask:[[Category:Tools Access Requests]][[Is Completed::false||no]]|format=broadtable|sort=Modification date|order=asc|headers=show|searchlabel=Outstanding Requests|limit=0|default=(No outstanding requests)}} and {{#ask:[[Category:Tools Access Requests]][[Is Completed::true||yes]]|format=broadtable|sort=Modification date|order=asc|headers=show|searchlabel=Completed Requests|limit=0|default=(No completed requests)}} - T162508: Implement Tool Labs membership application and processing in Striker
  • ~~ Help:MediaWiki-Vagrant in Labs/Hosts: Doesn't even seem to work ~~
  • Things using the custom bug tracker {{Ops Project}} template. Doesn't seem to be used since 2012-ish
    • Category:All_projects: DBQueryError Removed
    • Template:Ops_Project
    • Category:On_Hold_projects
    • Category:Proposed_projects
    • Category:Completed_projects
  • Various user pages via {{UserProjects}} to automatically list what projects people are interested - I think we can kill this usage fairly easily without too much complaints.
  • Template:Nova_Project
    • Instances for this project - Perhaps could be partially replaced by DPL (Wouldn't show the summary of information) (T161662)
  • {{Days since incidents}} - This is cute, but not very important
  • An unknown amount of ad-hoc queries directly via [[Special:Ask]]

For Forms

  • Nova_Resource:Tools/Documentation: Uses Form:Tools Access Request - could be replaced with preload template with a slight decrease in usability, but still good enough.
  • {{Nova Project}} - Uses Form:Nova_Project_Documentation in an "add documentation" link. Could be replaced with preload template with minor decrease in usability.
  • Form:Server - unclear if people even use this form, or if they fill out {{Server}} by hand. I think this could be easily deleted and people can just fill in template with help of preload parameter/inputbox.
  • Stuff in [[category:Tools Access Requests]] will use Form:Tools Access Request/edit if you edit the page using the "edit with form" tab

#arraymap parser func

This comes from SemanticForms extension. Could probably easily be replaced with lua if really needed

  • {{tool}}
  • {{nova Project Documentation}}
  • {{nova Instance}}
  • {{Nova Project}} (T161662)
  • {{hashartemplate}}

Details

Reference
bz51642
Related Gerrit Patches:
operations/mediawiki-config : masterWikitech: Remove $smwgNamespacesWithSemanticLinks config
mediawiki/tools/release : masterStop branching Semantic stuff for wikitech
operations/mediawiki-config : masterUndeploy Semantic* from wikitech wikis
operations/mediawiki-config : masterRevert "Disable creation of new forms on wikitech"

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2015, 5:14 PM
Qgil added a subscriber: Qgil.Jul 28 2015, 9:11 AM
Florian raised the priority of this task from Medium to High.Feb 4 2016, 5:13 PM
Florian added a subscriber: Florian.

I know, increase of the Priority doesn't result in a faster resolve, if no-one is working on the task. But with a look at T123599: Old versions of Semantic* extensions break Wikitech (after removing long-living deprecated functions from core) I think, normal isn't the appropriate priority of this task anymore. Hopefully, we can get rid of these extensions in the near future.

Legoktm renamed this task from Get rid of SMW/SRF/SF to Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org.Dec 10 2016, 1:12 AM

Ok, so current uses of SMW+friends on Wikitech is:

For querying

  • Analytics/EventLogging: Calls {{#ask: [[Category:EventLogging/Incident documentation]] | format=ul | limit=50 | order=desc }}. This could easily be replaced with Extension:DynamicPageList_(Wikimedia)
  • Portal:Wikimedia_Labs: Calls SMW to compute some statistics. Does anybody care about them. If they do, maybe someone could write a bot.
  • Resource_query_examples: Just some examples. It can be deleted
  • Labs_Server_Admin_Log: Transcludes all the server admin logs for labs. Is this even useful to anyone?
  • Help:Tool_Labs/Web: {{#ask: [[Resource Type::instance]][[Project::tools]][[Instance Name::~*proxy*]] | ?Private IP |format=list |headers=hide }} - this could probably be replaced with a manually updated list, or converted to categories and use DynamicPageList.
  • Nova Resource:Tools/Documentation: {{#ask:[[Category:Tools Access Requests]][[Is Completed::false||no]]|format=broadtable|sort=Modification date|order=asc|headers=show|searchlabel=Outstanding Requests|limit=0|default=(No outstanding requests)}} and {{#ask:[[Category:Tools Access Requests]][[Is Completed::true||yes]]|format=broadtable|sort=Modification date|order=asc|headers=show|searchlabel=Completed Requests|limit=0|default=(No completed requests)}} - Could probably mostly be replaced with DPL, or even just plain categories.
  • Help:MediaWiki-Vagrant in Labs/Hosts: Doesn't even seem to work
  • Things using the custom bug tracker {{Ops Project}} template. Doesn't seem to be used since 2012-ish
    • Category:All_projects: DBQueryError Removed
    • Template:Ops_Project
    • Category:On_Hold_projects
    • Category:Proposed_projects
    • Category:Completed_projects
  • Various user pages via {{UserProjects}} to automatically list what projects people are interested - I think we can kill this usage fairly easily without too much complaints.
  • Template:Nova_Project
    • Including a snippet from SAL - Could be replaced with Lua. Or just transclude the whole thing. Or labelledSectionTransclude. Lua is probably easiest.
    • Instances for this project - Perhaps could be partially replaced by DPL (Wouldn't show the summary of information)
  • {{Days since incidents}} - This is cute, but not very important
  • An unknown amount of ad-hoc queries directly via [[Special:Ask]]

For Forms

  • Nova_Resource:Tools/Documentation: Uses Form:Tools Access Request - could be replaced with preload template with a slight decrease in usability, but still good enough.
  • {{Nova Project}} - Uses Form:Nova_Project_Documentation in an "add documentation" link. Could be replaced with preload template with minor decrease in usability.
  • Form:Server - unclear if people even use this form, or if they fill out {{Server}} by hand. I think this could be easily deleted and people can just fill in template with help of preload parameter/inputbox.
  • Stuff in [[category:Tools Access Requests]] will use Form:Tools Access Request/edit if you edit the page using the "edit with form" tab

#arraymap parser func

This comes from SemanticForms extension. Could probably easily be replaced with lua if really needed

  • {{tool}}
  • {{nova Project Documentation}}
  • {{nova Instance}}
  • {{Nova Project}}
  • {{hashartemplate}}
demon added a subscriber: demon.Mar 14 2017, 9:10 PM
  • Help:MediaWiki-Vagrant in Labs/Hosts: Doesn't even seem to work

Killed this.

Generally: let's make this a checklist in the ticket summary? Easier to edit/track :)

Bawolff updated the task description. (Show Details)Mar 14 2017, 9:17 PM
bd808 updated the task description. (Show Details)Mar 15 2017, 9:44 PM
demon updated the task description. (Show Details)Mar 17 2017, 4:41 PM
demon updated the task description. (Show Details)Mar 17 2017, 5:02 PM
demon added a comment.Mar 17 2017, 5:06 PM

We already banned the creation of new forms, I wonder if we should do something with AbuseFilter to stop new introductions of {{#ask}} and {{#arraymap}}?

bd808 updated the task description. (Show Details)Mar 26 2017, 7:28 PM
bd808 moved this task from Triage to In Progress on the Cloud-Services board.Mar 26 2017, 7:35 PM
bd808 updated the task description. (Show Details)Mar 29 2017, 12:10 AM
bd808 updated the task description. (Show Details)Mar 29 2017, 12:20 AM
Reedy updated the task description. (Show Details)Mar 29 2017, 12:28 AM
Reedy updated the task description. (Show Details)Mar 29 2017, 12:30 AM
Reedy updated the task description. (Show Details)Mar 29 2017, 12:33 AM
Reedy added a subscriber: Reedy.
Method
    editArticle
Found usages  (6 usages found)
    Method call  (6 usages found)
        MediaWiki  (6 usages found)
            extensions/OpenStackManager/maintenance  (1 usage found)
                updateProjectPages.php  (1 usage found)
                    OpenStackNovaUpdateProjectPages  (1 usage found)
                        execute  (1 usage found)
                            21$project->editArticle();
            extensions/OpenStackManager/nova  (2 usages found)
                OpenStackNovaProject.php  (2 usages found)
                    OpenStackNovaProject  (2 usages found)
                        addMember  (1 usage found)
                            401$this->editArticle();
                        deleteMember  (1 usage found)
                            342$this->editArticle();
            extensions/OpenStackManager/special  (3 usages found)
                SpecialNovaProject.php  (1 usage found)
                    SpecialNovaProject  (1 usage found)
                        tryCreateSubmit  (1 usage found)
                            414$project->editArticle();
                SpecialNovaRole.php  (2 usages found)
                    SpecialNovaRole  (2 usages found)
                        tryAddMemberSubmit  (1 usage found)
                            222$project->editArticle();
                        tryDeleteMemberSubmit  (1 usage found)
                            265$project->editArticle();
bd808 updated the task description. (Show Details)Mar 29 2017, 1:03 AM
Qgil removed a subscriber: Qgil.Apr 5 2017, 9:50 AM
bd808 updated the task description. (Show Details)Apr 6 2017, 10:14 PM
bd808 updated the task description. (Show Details)Apr 6 2017, 10:28 PM
bd808 updated the task description. (Show Details)Apr 7 2017, 12:52 AM
bd808 updated the task description. (Show Details)Apr 7 2017, 10:56 PM

At this point, the only useful thing using forms and/or SMW properties is the Tool Labs membership request system. I'm working on porting that whole process over to Striker. I just realized I actually need to make a task for that.

demon added a comment.Apr 7 2017, 11:43 PM

Hah, removing 'edit' from Form: means I can't even delete 😂

bd808 updated the task description. (Show Details)May 5 2017, 11:58 PM

Change 352721 had a related patch set uploaded (by BryanDavis; owner: BryanDavis):
[operations/mediawiki-config@master] Revert "Disable creation of new forms on wikitech"

https://gerrit.wikimedia.org/r/352721

Change 352721 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Disable creation of new forms on wikitech"

https://gerrit.wikimedia.org/r/352721

Mentioned in SAL (#wikimedia-operations) [2017-05-08T23:21:44Z] <bd808@tin> Synchronized wmf-config/wikitech.php: Revert "Disable creation of new forms on wikitech" (T53642) (duration: 01m 10s)

Change 352720 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/tools/release@master] Stop branching Semantic stuff for wikitech

https://gerrit.wikimedia.org/r/352720

Change 352722 had a related patch set uploaded (by Reedy; owner: Reedy):
[operations/mediawiki-config@master] Undeploy Semantic* from wikitech wikis

https://gerrit.wikimedia.org/r/352722

  • Deleted Form:Nova_Project_Documentation
  • Deleted Form:Server
  • Deleted Form:Tools_Access_Request/Edit
  • Deleted Form:Tools_Access_Request

No moar forms.

Tool Labs membership requests are now handled by https://toolsadmin.wikimedia.org/tools/membership/apply

bd808 updated the task description. (Show Details)May 9 2017, 12:54 AM
bd808 updated the task description. (Show Details)May 9 2017, 1:08 AM
bd808 updated the task description. (Show Details)May 9 2017, 1:11 AM
bd808 added a comment.EditedMay 9 2017, 1:15 AM

I hacked together a version of mwgrep that only looks at wikitech's pages and hits all namespaces at the same time. @Reedy helped me edit a lot of things that were creating SMw properties, using Semantic Forms, or querying Special:Ask. These matches that are left all seem to be false positives.

$ ./wikitech-grep 'Special:Ask|\#ask|FormEdit|\[\[[^]]+::|\#arraymap'
## Public wiki results
labswiki            Logmsgbot
labswiki            Nova Resource:Cvn/SAL/Archive 1
labswiki            Nova Resource:Metavidwiki/openmeetings v metavidwiki
labswiki            Server Admin Log
labswiki            Server Admin Log/Archive
labswiki            Server admin log/Archive 17
labswiki            Server admin log/Archive 3
labswiki            Server admin log/Archive 30
labswiki            Server admin log/Archive 31
labswiki            Talk:Main Page
labswiki            Talk:Server Admin Log/Transclude the last log entries only/reply
labswiki            User talk:Taha
labswiki            User talk:Tim Landscheidt/Archive

(total: 13, shown: 13)

Change 352722 merged by jenkins-bot:
[operations/mediawiki-config@master] Undeploy Semantic* from wikitech wikis

https://gerrit.wikimedia.org/r/352722

Change 352720 merged by jenkins-bot:
[mediawiki/tools/release@master] Stop branching Semantic stuff for wikitech

https://gerrit.wikimedia.org/r/352720

bd808 closed this task as Resolved.May 9 2017, 11:02 PM
bd808 claimed this task.

Thanks to everyone who pitched in to work on this. It was certainly a team effort.

Restricted Application added a project: User-bd808. · View Herald TranscriptMay 9 2017, 11:02 PM

Change 352980 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/mediawiki-config@master] Wikitech: Remove $smwgNamespacesWithSemanticLinks config

https://gerrit.wikimedia.org/r/352980

Change 352980 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikitech: Remove $smwgNamespacesWithSemanticLinks config

https://gerrit.wikimedia.org/r/352980

Paladox removed a subscriber: gerritbot.

Well done, guys! It took a lot of effort, but you've achieved "a slight decrease in usability".

Well done, guys! It took a lot of effort, but you've achieved "a slight decrease in usability".

I can understand this view. Before we forget, it would be nice to write down the things which will no longer be possible without SMW.

A simple "regression" I encountered is that the infobox in wikitechwiki pages no longer contains some information, and instead links to an external domain, so any full text search will be less meaningful.

Personally I would be quite interested in some way to populate the {{server}} infobox automatically (with things like those that ganglia, or maybe rackspaces, know) for pages like https://wikitech.wikimedia.org/wiki/Mchenry , which were quite useful in the past. Integrating external data into wiki pages is always a hot topic, let's keep a light on it.

Reedy added a comment.EditedMay 10 2017, 9:33 AM

It would be perfectly reasonable to create other seperate tasks for the broken/missing functionality

Baring in mind, the intention is to undeploy OpenStackManager too in the future, moving towards using seperate OpenStack tools... See T161553 for that

In those cases, replacing them might not be worth the intermediate effort, unless you can query horizon directly, say using​ Lua, and add an amount of future proofing going forward