Page MenuHomePhabricator

Install SMW on AffCom wiki
Closed, DeclinedPublic

Description

Hi Ops,
We would like to install SMW extensions on the AffCom wiki, available at:
https://chapcom.wikimedia.org
What is the process to do this?

  • Forwarded message ----------

Stephen, youl you pass this on to the right people in the ops department
please?
Just get the SMW Bundle, Semantic Forms and Semantic Result Formats -
which include Calendar - are already packaged into it.
The SMW Bundle can be easily installed using Composer
composer.json: add
{

"require": {
        "mediawiki/semantic-media-wiki": "~1.9",
},

}
run:
php composer.phar install
php maintenance/update.php
...and you are done!

/Manuel

Event Timeline

emailbot added a subscriber: emailbot.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 5 2015, 9:59 PM
chasemp triaged this task as Normal priority.Feb 5 2015, 10:02 PM
chasemp changed the visibility from "Public (No Login Required)" to "WMF-NDA (Project)".
chasemp changed the edit policy from "All Users" to "WMF-NDA (Project)".
chasemp set Security to None.
chasemp added a subscriber: chasemp.

Just a note, this was an email sent to ops-requests@rt I would imagine, that gets forwarded and creates an issue on phab with the Operations project. We do this in order to not lose these things completely but it seems like maybe these users are WMF and would benefit from being first class Phab users.

demon added a subscriber: demon.Feb 5 2015, 10:04 PM

Not an operations thing, just a general site config.

But generally speaking, the only wiki we've ever allowed SMW on was wikitech, and that's for different reasons.

Dzahn removed subscribers: demon, chasemp.Feb 5 2015, 10:05 PM
Greetings Stephen!

We got your ops-request and it has been logged in our new system Phabricator here https://phabricator.wikimedia.org/T88748

Do you have a Phabricator account?  It may be beneficial to sign up as most history and correspondence will be in this medium.

Thanks!

-- 
Chase Pettet
Operations Engineer
demon renamed this task from Fwd: [AffCom] "Brazil UG is no more" to Install SMW on AffCom wiki.Feb 5 2015, 10:08 PM
demon changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".
demon changed the edit policy from "WMF-NDA (Project)" to "All Users".

Pretty sure this is automatically declined per T10390

Reedy added a subscriber: Reedy.Feb 5 2015, 10:17 PM

FYI, installing SMW on the cluster really isn't "as easy as that". There's more to it. Plus security review for SMW and dependencies\. And we're running 1.8 for wikitech.

I'm not sure if the security review is more or less of an issue for a private wiki

Slaporte updated the task description. (Show Details)Feb 6 2015, 12:47 AM
Slaporte added a subscriber: 80686.
Slaporte added a subscriber: Slaporte.

What do we need to get SMW installed for a small private wiki (chapcom.wikimedia.org)?

Aklapper lowered the priority of this task from Normal to Low.Feb 16 2015, 10:11 AM
Aklapper added a subscriber: greg.

Wondering if @greg can provide some general input / thoughts on deploying SMW...

80686 added a comment.Mar 2 2015, 7:39 AM

Can we get some feedback on this please?
We want to start tracking stuff in the wiki and build some automated tools, which need the structured data in the wiki.

You'll need to convince people to get the rejection on T10390 reverted, at least. Then you'll need it to be properly security reviewed.

80686 added a comment.Mar 2 2015, 2:58 PM

I don't understand. We alread use SWM on wikitech wiki, why can't we just do the same for the affcom wiki then? Where is the difference?

labsconsole (now wikitech) was much more separate from production at the time it got SMW installed. chapcomwiki being private seems like another good reason to require it.

80686 added a comment.Mar 2 2015, 3:27 PM

Well, in the end we simply need the features and (since we lost quite some time since this discussion has started) we need them soon. We have a huge tech department, server admins and software developers, so we just need to get it done.

You cannot reasonably expect Wikimedia's developers to just drop everything and rush over here to start preparing an extension for deployment because you said so. You're a small (and private!) wiki, and haven't even provided your use case.

It's not even clear this would be allowed.

chasemp removed a subscriber: chasemp.Mar 2 2015, 4:37 PM
80686 added a comment.Mar 2 2015, 4:44 PM

Well, we are a volunteer body who serve the movement, as volunteers, and provide a service.
The techs are paid staff intended to provide their tech service as well. Of course I don't expect "drop everything", but so far we have made a resolution that we need to introduce certain workflows and the best and easiest way doing so is using SMW, as that already exists, is in use by the WMF and runs on top of our existing working platform. That was on January 29th, so far we have lost one month doing nothing, we don't even have an ETA and no real reason not to just do it. That is not an acceptable service level to me, especially as the whole install takes five minutes - so we wasted more time discussing than to just fix the actual issue.

The AffCom has reviewed its policies and needs to (re-)introduce certain workflows and management systems. We need to collect and maintain information of the orgs we deal with and trigger workflows based on deadlines or certain regular dates, eg. the renewal process for WUGs. Obviously using SMW would be crucial as it would give us the database features (forms, entities) to manage the information and the logic to implement the workflows needed.
I wonder why I need to excuse myself to do the (volunteer) work I have committed to and that has been approved by the committee to be needed to be done, while a simple request for help is being killed here.

greg updated the task description. (Show Details)Mar 2 2015, 4:54 PM
greg added a project: Wikimedia-Site-requests.

Hi Manuel,

Sorry for the delay and confusion here. I'll talk with some people here to determine if and how we can do this for AffCom. I can't make promises (there is a long standing decision not to use SMW on production wikis, and wikitech was only an exception, not a path creator) but I'll let you know asap.

80686 added a comment.Mar 2 2015, 5:02 PM

Thanks a lot Greg for your response. That certainly brings some brightness into this discussion :-)

I don't know the rationals behind such a general decision of not using SMW (I run around 60 instances of it), I am just convinced that using SMW more broadly would have huge benefits, especially for committees like the AffCom. Maybe this request can facilitate some general changes (to SMW or wherever the underlying problems are) so we can make that step of being able to allow SMW in the future.

I know that SMW is used in several public and private sites, ranging from austrian authorities to the NASA, so I wonder if they are all stupid not seing the issues you see or if they done something we don't know about, or if we have such a special infrastructure (it is special for sure, but that much of an issue?).

csteipp added a subscriber: csteipp.Mar 2 2015, 5:09 PM

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

What's the affcom wiki dbname? Is it private?

If we need to install smw, the extensions will need a fair amount of review and work.

greg added a comment.Mar 2 2015, 5:16 PM

Regarding the decision not to use SMW on WMF production wikis: See https://phabricator.wikimedia.org/T10390#132245

I can't stop you from bringing up that discussion again, but I personally don't see the decisions being any different this time, especially since Wikidata is addressing much of the use-cases on many wikis.

What this task is requesting is a special exemption from the rule of not installing SMW on WMF prod wikis. If you want to start the "should WMF allow SMW?" conversation, please do so on a different ticket or mailing list thread.

Krenair added a comment.EditedMar 2 2015, 5:17 PM

Of course I don't expect "drop everything", but so far we have made a resolution that we need to introduce certain workflows and the best and easiest way doing so is using SMW, as that already exists, is in use by the WMF and runs on top of our existing working platform. That was on January 29th, so far we have lost one month doing nothing, we don't even have an ETA and no real reason not to just do it. That is not an acceptable service level to me, especially as the whole install takes five minutes - so we wasted more time discussing than to just fix the actual issue.

This ticket is less than a month old. Do you know that compares to many other extension deployment requests?
Do not pretend this is a simple, mindless, <5 minute deployment that requires no thought.
The process described at the top of this ticket is not even how deployments work on the WMF production cluster anyway (update.php? composer?)

Additionally, I noticed that the "SMW Bundle" requested appears to include much more than what is already on wikitech - so yes, some sort of security review is definitely required.

What's the affcom wiki dbname? Is it private?

chapcomwiki, domain is chapcom.wikimedia.org - yes, it's private.

80686 added a comment.Mar 2 2015, 5:24 PM

Hmm, having Wikibase could be an alternative. It would need some help from Wikidata pros to achieve what we planned to do with SMW.

What we need:

  • forms with specific fields for all types of affiliates, to store / maintain information about all orgs
  • queries / list views creating an overview of exising orgs in the wiki - with seperate views based on status and dates - such as affiliated, applications, denied applications, open requests ordered by date (tasklist) etc.
  • some templates with a logic (parser functions?) that trigger events based on dates (eg. when WUG renewal date is reached)

I don't know if / how that is doable with Wikibase. Is there some way to create queries that create lists or even other result types?

What we need:

  • forms with specific fields for all types of affiliates, to store / maintain information about all orgs
  • queries / list views creating an overview of exising orgs in the wiki - with seperate views based on status and dates - such as affiliated, applications, denied applications, open requests ordered by date (tasklist) etc.
  • some templates with a logic (parser functions?) that trigger events based on dates (eg. when WUG renewal date is reached)

All of this should be doable with Scribunto/Lua (IMO) and maybe a gadget to make the editor a bit nicer.

80686 added a comment.Mar 2 2015, 5:41 PM

Ok, problem here: SMW is all simple MW syntax, there are a bunch of examples and experience around.

I have no clue how to use Lua or Scribunto. I am happy to learn it but I certainly need help. This should all be up running until the next AffCom meeting (May 14). With SMW no problem, already know what to do. With Wikibase / Lua / Scribunto: No idea. So is there anyone helping me with that?

Nemo_bis added a subscriber: Nemo_bis.EditedMar 2 2015, 5:59 PM

Lua and forms don't have much to do with each other. If the use case is simple, perhaps https://www.mediawiki.org/wiki/Extension:Cargo + SemanticForms is an option? Usage is similar for the end user and there's a much smaller codebase to review.

greg changed the task status from Open to Stalled.Mar 12 2015, 10:14 PM

@80686: I'm setting this request to "stalled" for now given the reality of what is likely to happen any time soon.

To deploy SMW on the production environment, it'd still need a security review and unfortunately there's a fair amount to review and our security team (aka Chris) is overloaded.

Even after that, there is still the policy question of allowing SMW on WMF production systems which I am unlikely to overturn lightly.

mwjames added a subscriber: mwjames.Apr 4 2015, 2:38 AM

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

I'm stunned by such statemens the WMF (or an employee on behalf of WMF) makes about SMW without citing the potential list of issues.

Given that the Municipal and Provincial Archives of Vienna (MA8) [0] hosts a SMW/Wiki and needed an external review with the result of "All of the used software components were security checked by the Information and Communications Technology (ICT) department of the City (MA 14)", I wonder about your assertion.

If you have a list of issues you see or like see to be addressed, the community would welcome it but making a statement without reference has the smell of public discreditation.

[0] http://semantic-mediawiki.org/wiki/Vienna_History_Wiki

I too would like some clarification. In particular, compared to what is the number of security issues so unusually high? Other MediaWiki extensions? Also, is this about relative or absolute amounts? And is this about SMW, or SMW and all extensions to it?

@greg, @csteipp, it seems clear that SMW isn't going to be installed on WMF-controlled sites. It is my understanding that SMW's reliance on the job queue and the performance hit that can cause on a large site is the big reason WMF has deemed it necessary to create other tools for use cases that SMW handles. (Of course, I'm willing to be corrected here since my understanding of all this is very poor.)

What is less clear is what actual security problems you've identified in SMW that have gone unresolved.

Given that orgs like CERT, NASA, and major corps are using SMW, it would help to identify the problems so that they can be fixed or their use of SMW can be re-evaluated and perhaps be replaced with the presumably more secure combination of Wikidata, Lua and JS,.

greg added a comment.Apr 6 2015, 9:52 PM

Since this is stalled and there is nothing for RelEng to do here, I'm removing our team's project.

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

I'm stunned by such statemens the WMF (or an employee on behalf of WMF) makes about SMW without citing the potential list of issues.
Given that the Municipal and Provincial Archives of Vienna (MA8) [0] hosts a SMW/Wiki and needed an external review with the result of "All of the used software components were security checked by the Information and Communications Technology (ICT) department of the City (MA 14)", I wonder about your assertion.
If you have a list of issues you see or like see to be addressed, the community would welcome it but making a statement without reference has the smell of public discreditation.

@mwjames, it was not by intention to discredit SMW generally. There have been 3 closed security issues in SMW. That's more than some extensions, and less than others. I'd be very interested to see the results of the security audit performed by your city. They have probably examined it more thoroughly than I have.

I made that statement after I had triaged several bug reports from a bounty program where all of the reports were issues in SMW. Most of the issues have ended up being because the admin was confused about how to upgrade to a version of SMW that fixed all the known security issues, so my comment should have been, "Anecdotally, I'm seeing more issues reported against SMW than the default extensions distributed with the tarball, when it's been in scope for a bug bounty program, but that may be due to other factors".

Given that orgs like CERT, NASA, and major corps are using SMW, it would help to identify the problems so that they can be fixed or their use of SMW can be re-evaluated and perhaps be replaced with the presumably more secure combination of Wikidata, Lua and JS,.

Mark, those organizations should do their own risk assessments. I'm not trying to say that SMW is worse than Wikidata+Lua+JS in combination. Those extensions have had more security issues fixed than SMW, although they have likely had more scrutiny as well. Each organization should run the minimum amount of code possible, and turn off any unessesary functions.

I am apposed to adding SMW to the existing Wikidata+Lua+etc on our cluster, because it increases our attack surface.

Aklapper added a subscriber: Qgil.Apr 7 2015, 11:58 AM

I am apposed to adding SMW to the existing Wikidata+Lua+etc on our cluster, because it increases our attack surface.

This is reasonable. Thank you for this clarification.

I made that statement after I had triaged several bug reports from a bounty program where all of the reports were issues in SMW.

Ugh... I think I saw a couple of those. Bug bounty programs seem like a good way for the offering org to be informed that it is time to update or reconfigure the software that they're using on their website, even if they aren't responsible for writing it.

Qgil added a comment.Apr 7 2015, 3:59 PM

This task started with a request for implementing a specific solution to a problem that has been vaguely defined.

The AffCom has reviewed its policies and needs to (re-)introduce certain workflows and management systems. We need to collect and maintain information of the orgs we deal with and trigger workflows based on deadlines or certain regular dates, eg. the renewal process for WUGs.

There are two basic questions:

  • What good tools can solve this problem?
  • Who will maintain the selected tool(s)?

Some options might be self-hosted, some hosted by third parties. Some options might have to be maintained by the Wikimedia Foundation, some options might allow to be maintained by the AffCom / other volunteers themselves (in Labs, somewhere else) or be maintained by third parties.

The discussion here goes around the Wikimedia Fundation teams not willing to take the responsibility of hosting SemanticMediaWiki in production, for the reasons explained. Are there other alternatives?

Lua and forms don't have much to do with each other. If the use case is simple, perhaps https://www.mediawiki.org/wiki/Extension:Cargo + SemanticForms is an option? Usage is similar for the end user and there's a much smaller codebase to review.

much smaller codebase

Line count is not everything. Cyclomatic complexity and so on are probably more relevant

Krenair closed this task as Declined.Jul 16 2015, 9:13 PM
Krenair claimed this task.

We want to get rid of Semantic stuff (T53642) from the cluster, not enable it further.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 16 2015, 9:13 PM