+1. We should also find a way to evangelise it. Perhaps by doing a tech talk or something similar would do and/or present it at the relevant conferences and venues (Hackathon, Wikimania at least).

+1 to this and tech talk, btw Sarah and Subbu are looking for more tech talks (see their recent email)

This RFC is proposed for approval. The last call for comments is open until April 24 2019. If no major issues are raised and remain unresolved until that time, this RFC will be approved and the draft as proposed will become official policy.

Note that while this task has only been created on April 10 2019, the draft has been under discussion on mediawiki.org since May 2018.

@daniel - per this comment, is there a documented definition of "official WMF tool" anywhere? Specifically, who makes that determination?

In T220657#5105472, @sbassett wrote:

@daniel - per this comment, is there a documented definition of "official WMF tool" anywhere? Specifically, who makes that determination?

There is no formal definition, but in practice, it means anything that is maintained by WMF staff, and/or is deployed to WMF production infrastructure (i.e. not Labs), and/or is published by the WMF for use by others. This is in contrast to software maintained by volunteers or 3rd party developers. We have no authority over what they do, nor any way to enforce any policy. WMF only controls what gets merged into WMF owned repos and run on WMF owned servers.

In T220657#5105495, @daniel wrote:

In T220657#5105472, @sbassett wrote:

@daniel - per this comment, is there a documented definition of "official WMF tool" anywhere? Specifically, who makes that determination?

There is no formal definition, but in practice, it means anything that is maintained by WMF staff, and/or is deployed to WMF production infrastructure (i.e. not Labs), and/or is published by the WMF for use by others. This is in contrast to software maintained by volunteers or 3rd party developers. We have no authority over what they do, nor any way to enforce any policy. WMF only controls what gets merged into WMF owned repos and run on WMF owned servers.

Non WMF staff can merge into a lot (most?) of WMF owned repos...

What about WMF staff doing "volunteer" activities? Do they count because they're staff? Where do we draw the line? I know for numerous members of staff, the lines between work and volunteer activities can be pretty blurred.

Issues I see in the current text:

• The introduction makes it sound like it would be possible to always comply to all principles the same time, just by trying hard enough. In reality such principles often conflict with each other. Our job is more to find a good balance. I wish the introduction would accept and mention this. I certainly don't want anybody to pick a single principle from the list, run around and make everybodies life hard who needs to balance all of them, and isn't able to fully comply to one because of this.
• "MUST be designed to avoid any need for clients to process wikitext." I find combining a "MUST" with an "avoid" confusing and dangerous. I feel this needs to be a "SHOULD". There are certainly situations where it is not possible to "avoid" wikitext entirely. What happens then? Does the "MUST" still allow users to enforce this rule and block patches?

In T220657#5107914, @thiemowmde wrote:

I feel this needs to be a "SHOULD". There are certainly situations where it is not possible to "avoid" wikitext entirely.

What situations are those? If something is expected to return wikitext, the API should have an option to parse it and return HTML. That's not hard to achieve.

It would be useful to add something about who is expected to know / enforce the policy. This is a pretty heavy technical document; throwing it at new contributors is a good way of making them not contributors anymore. IMO the point by which one is expected to read and understand the document is getting +2; reviewers should be expect to act as human interface to the policy and explain specific problems to patch authors without making them read four pages of dense technical jargon. That might be obvious but IMO worth spelling out.

In T220657#5109260, @Tgr wrote:

It would be useful to add something about who is expected to know / enforce the policy. This is a pretty heavy technical document; throwing it at new contributors is a good way of making them not contributors anymore. IMO the point by which one is expected to read and understand the document is getting +2; reviewers should be expect to act as human interface to the policy and explain specific problems to patch authors without making them read four pages of dense technical jargon. That might be obvious but IMO worth spelling out.

We had the same concern while discussing the document. We thought these shorthand names for each principle might help: https://www.mediawiki.org/w/index.php?title=Wikimedia_Engineering_Architecture_Principles&type=revision&diff=3188743&oldid=3188307. You're right that the human interfacing and not throwing a huge document at people is key to this policy. So hopefully being able to refer to specific points and link directly to them will work sort of like WP: policies work. The burden remains on reviewers, of course, so we should be careful to include some guidelines for reviewers when we announce these principles.

In T220657#5107324, @Reedy wrote:

Non WMF staff can merge into a lot (most?) of WMF owned repos...

I think we can expect that people who work on software that is deployed on the production cluster know the relevant policies and guidelines and follow them.

What about WMF staff doing "volunteer" activities? Do they count because they're staff? Where do we draw the line? I know for numerous members of staff, the lines between work and volunteer activities can be pretty blurred.

I suppose it's not so much who is staff and what is done on payed time, but rather what happens to software running in production.

In T220657#5107914, @thiemowmde wrote:

Issues I see in the current text:

• The introduction makes it sound like it would be possible to always comply to all principles the same time, just by trying hard enough. In reality such principles often conflict with each other. Our job is more to find a good balance. I wish the introduction would accept and mention this. I certainly don't want anybody to pick a single principle from the list, run around and make everybodies life hard who needs to balance all of them, and isn't able to fully comply to one because of this.
• "MUST be designed to avoid any need for clients to process wikitext." I find combining a "MUST" with an "avoid" confusing and dangerous. I feel this needs to be a "SHOULD". There are certainly situations where it is not possible to "avoid" wikitext entirely. What happens then? Does the "MUST" still allow users to enforce this rule and block patches?

Originally, all principles were using SHOULD, because of the difficulties you mentioned. But several people had the impression that this would make the policy useless and blurry, so we tightened it. I suppose this is another case for striking a balance. I suppose most of them are a REALLY REALLY SHOULD BUT NOT ABSOLUTELY MUST or something. The relevant discussion is here: https://www.mediawiki.org/wiki/Topic:Unl2nb3os3lemrfv

In T220657#5109260, @Tgr wrote:

It would be useful to add something about who is expected to know / enforce the policy. This is a pretty heavy technical document; throwing it at new contributors is a good way of making them not contributors anymore. IMO the point by which one is expected to read and understand the document is getting +2; reviewers should be expect to act as human interface to the policy and explain specific problems to patch authors without making them read four pages of dense technical jargon. That might be obvious but IMO worth spelling out.

This is a very good point, thank you. How about adding the following after the first sentence of the "Application" section:

...from detailed core review to high level RFCs. People with merge rights on software in production on WMF servers, as well as people responsible for technical decision making and planning for such systems, are expected to know and follow these principles.Any new component we design should be...

What do you think?

In T220657#5113082, @daniel wrote:

I suppose it's not so much who is staff and what is done on payed time, but rather what happens to software running in production.

Pedantically, this leads next to the question of what is "production". As an example: in the talk page discussion @daniel proposed that Quarry should be considered to fall under the policy "at least if it's an official WMF tool". Quarry is deployed inside of a Cloud-VPS project. That computing runtime environment is not generally considered "production" by the Foundation's SRE team or the cloud-services-team. To a certain extent I think this is just another example highlighting that making binary classifications of software projects is going to be tricky.

In T220657#5113119, @daniel wrote:

What do you think?

Looks good. Maybe I'd replace "know and follow" with "know and enforce".
Probably should be mentioned on the responsibilities section of the +2 page as well.

In T220657#5113283, @Tgr wrote:

Looks good. Maybe I'd replace "know and follow" with "know and enforce".
Probably should be mentioned on the responsibilities section of the +2 page as well.

Done, with "know and apply". https://www.mediawiki.org/w/index.php?title=Wikimedia_Engineering_Architecture_Principles&type=revision&diff=3195195&oldid=3195070&diffmode=source

In T220657#5113120, @bd808 wrote:

Pedantically, this leads next to the question of what is "production". As an example: in the talk page discussion @daniel proposed that Quarry should be considered to fall under the policy "at least if it's an official WMF tool". Quarry is deployed inside of a Cloud-VPS project. That computing runtime environment is not generally considered "production" by the Foundation's SRE team or the cloud-services-team. To a certain extent I think this is just another example highlighting that making binary classifications of software projects is going to be tricky.

Quarry is not in scope of the Architecture Principles by virtue of running in a production environment. But as far as I know (please correct me if I'm wrong), WMF staff "owns" the project, maintains the code, responds to bug reports. If this is the case, the policy applies. If this is not the case (and it's not running in a production environment), then the policy doesn't apply.

In T220657#5113328, @daniel wrote:

Quarry is not in scope of the Architecture Principles by virtue of running in a production environment. But as far as I know (please correct me if I'm wrong), WMF staff "owns" the project, maintains the code, responds to bug reports. If this is the case, the policy applies. If this is not the case (and it's not running in a production environment), then the policy doesn't apply.

The usefulness of this branch of discussion is moving in a debatable direction, but I will try one more time to make my point using Quarry as an example. The project was started by Yuvi as a sort of "20% time" experiment while under the employ of the Wikimedia Foundation. Since Yuvi's departure in 2017 the WMCS team has only been able to commit to keeping the application running in the most bare bones sense, primarily because we do not have enough people on the team to give Quarry and a few other projects started in the past (PAWS being a prime example) proper support. We have however been very, very lucky to find some volunteers (Yi Fei & Framawiki) who have been able to take over maintenance of Quarry. Today the application is being promoted by the Cloud Services team as a good option for folks who need to run basic SQL queries against the Wiki Replica servers, but it is primary development and support is coming from technical volunteers.

In T220657#5105495, @daniel wrote:

In T220657#5105472, @sbassett wrote:

@daniel - per this comment, is there a documented definition of "official WMF tool" anywhere? Specifically, who makes that determination?

There is no formal definition, but in practice, it means anything that is maintained by WMF staff, and/or is deployed to WMF production infrastructure (i.e. not Labs), and/or is published by the WMF for use by others. This is in contrast to software maintained by volunteers or 3rd party developers. We have no authority over what they do, nor any way to enforce any policy. WMF only controls what gets merged into WMF owned repos and run on WMF owned servers.

At each drilldown into this statement there has been a bit of a pivot. I think you are heading towards an I know it when I see it definition. That's ok in my opinion. Maybe the clarification is even as easy as using the RFC 2119 language to define the uncontested MUST applications (MediaWiki core, Wikimedia movement project wiki deployed extensions, Parsoid, RESTBase, etc) and a SHOULD for pretty much anything else?

There is no formal definition, but in practice, it means anything that is maintained by WMF staff

I would object to defining this by if the person is staff. Whether this applies to something should be the type of thing it is, not who is doing it.

We already have major components of MediaWiki that are de-facto maintained by volunteers.

Unless otherwise specified, the scope of policies proposed through and/or approved by TechCom, is expected to be the scope of the committee itself. (See TC Charter § Scope).

Specific documents may defined a narrower scope for some or all of its statements (See the Development policy for example). I do not think it is the intention here to define a new or narrower scope, so I suggest we just refer to the charter on this and leave it as a debate for another time whether that scope includes "software de-facto owned by WMF but deployed outside production and maintained primarily by volunteers".

What about extensions bundled with MediaWiki in the release tarballs, but not deployed on Wikimedia servers? Usually we require the same standards for those, as that's what users will expect.

In T220657#5113690, @Tgr wrote:

What about extensions bundled with MediaWiki in the release tarballs, but not deployed on Wikimedia servers? Usually we require the same standards for those, as that's what users will expect.

Yes, I agree that the same standards should apply.

In T220657#5113616, @Krinkle wrote:

Unless otherwise specified, the scope of policies proposed through and/or approved by TechCom, is expected to be the scope of the committee itself. (See TC Charter § Scope).

That's a very good approach, thank you!

The big question is now whether the draft should be changed to be more clear about its scope. Currently, it just says "Wikimedia engineering endeavors". Is this enough, or does it need to change? If there is a need to be more specific, I propose the wording "all software maintained by the WMF, deployed by the WMF, or distributed by the WMF". In other words, anything the WMF is responsible for.

@bd808 Sorry, only now saw the second part of your post.

In T220657#5113517, @bd808 wrote:

At each drilldown into this statement there has been a bit of a pivot.

Heh :) I woudn't call it a pivot, but I did realize that saying "WMF staff" isn't very useful, I'm really looking for a way that says "controlled by WMF as an org", but failing to express that in a good way.

I think you are heading towards an I know it when I see it definition. That's ok in my opinion. Maybe the clarification is even as easy as using the RFC 2119 language to define the uncontested MUST applications (MediaWiki core, Wikimedia movement project wiki deployed extensions, Parsoid, RESTBase, etc) and a SHOULD for pretty much anything else?

That might work, but such lists tend to become outdated quickly. What do you think of the "all software maintained by the WMF, deployed by the WMF, or distributed by the WMF" phrasing proposed above?

And: do we really need any clarification beyond "Wikimedia engineering endeavors"?

In T220657#5119927, @daniel wrote:

In T220657#5113517, @bd808 wrote:

I think you are heading towards an I know it when I see it definition. That's ok in my opinion. Maybe the clarification is even as easy as using the RFC 2119 language to define the uncontested MUST applications (MediaWiki core, Wikimedia movement project wiki deployed extensions, Parsoid, RESTBase, etc) and a SHOULD for pretty much anything else?

That might work, but such lists tend to become outdated quickly. What do you think of the "all software maintained by the WMF, deployed by the WMF, or distributed by the WMF" phrasing proposed above?

I'm going to continue to push back on "Wikimedia Foundation" being a part of the definition. The reason I'm going to continue to push back on this point is that the Wikimedia Foundation is just one of many organizations involved in the production of software to support the needs of the Wikimedia movement. It also in my opinion downplays the importance of "un-affilliated" volunteer technical contributors.

As I understand it, there would be a completely different process other than a Technical Committee RFC if the policy was only meant for the internal needs of the Wikimedia Foundation's Audiences and Technology departments. By framing this discussion as an RFC rather than a shared Definition of Done or other internal policy you are attempting to establish guidelines for the Wikimedia movement's technical contributors across a range of FOSS software projects.

I see the attempts at carving out a scope at all as trying to be sensitive to the Quality expectations that can reasonably be placed on software and services which are related to but not completely central to the experience of interacting with the Wikipedia and the sister projects. This is awesome, and I applaud you and everyone involved for being aware of that.

And: do we really need any clarification beyond "Wikimedia engineering endeavors"?

For me (and I readily admit that this may not be a widely held belief today), "Wikimedia engineering endeavors" would encompass nearly all Wikimedia technical community contributions including bots, webservices, gadgets, lua modules, and libraries in addition to MediaWiki core, MediaWiki extensions, 'services' (which is another poorly defined term of art in use in this discussion), etc. That puts me right back to asking for more clarity in scoping.

@Krinkle proposed in T220657#5113616 that the scope of this policy be aligned with the declared scope of the Technical Committee itself. That is currently "[...] all the official software that serves Wikimedia users. That includes MediaWiki and extensions, services, and other software running on the production cluster, along with official mobile apps.". I'm comfortable with that as the scope of this policy's 'official' reach and also with encouraging projects that do not fall entirely (or at all) inside that scope to consider and attempt to implement as much of the policy as is reasonably possible for each particular project. This would in my reading however put some things that have been mentioned as possibly in scope into the category of volunteer rather than mandatory compliance. Specifically, it returns importance to my question from T220657#5113120, what is "production"?

I know this is pedantic and tedious. I also know that Wikimedians as a collective group tend to take rules very seriously, and that disagreement in interpretation of rules is a common source of conflict both on and off wiki. That's in part why I said I would be satisfied by an I know it when I see it definition. I believe that being explicit rather than implicit that there are fuzzy edges to be debated may reduce friction in the future between otherwise well intentioned parties who may come into conflict over whether or not a particular software project is compelled or merely encouraged to comply with the policy.

This Last Call closes next Wednesday. Is the scoping the only open concern? Are there no reservations or questions about the individual principles that haven't been discussed you? Do we have agreement that there are principles that we can and will follow in out day to day work?

As to your comment, @bd808, I see your point, and I'd like to know what wording you would suggest to resolve the issue. On the other hand, I'd like us to climb out of that rabbit hole, and make sure that there are not other things to discuss before this last call closes. So perhaps let's give other discussions some space first.

In T220657#5121865, @daniel wrote:

This Last Call closes next Wednesday. Is the scoping the only open concern? Are there no reservations or questions about the individual principles that haven't been discussed you? Do we have agreement that there are principles that we can and will follow in out day to day work?

I left a comment on the talk page, but I do not believe we can adopt the section "To provide a web application that can be freely used to collaboratively collect and share knowledge" since it codifies a product/business assumption.

What situations are those? If something is expected to return wikitext, the API should have an option to parse it and return HTML. That's not hard to achieve.

I'm sorry, but I'm not ok with putting me in a corner where I have to defeat my need to be allowed to work with wikitext. That's pretty much exactly how I don't want these principles to be used.

APIs MUST be designed to avoid any need for clients to process wikitext.

Can we please rephrase this? Suggestion:

APIs MUST not require clients to parse wikitext, unless this is the purpose of the API.

In T220657#5134234, @thiemowmde wrote:

I'm sorry, but I'm not ok with putting me in a corner where I have to defeat my need to be allowed to work with wikitext. That's pretty much exactly how I don't want these principles to be used.

API/NOWIKI

APIs MUST be designed to avoid any need for clients to process wikitext.

The principle does not in way, shape or form explicitly forbid APIs to output wikitext. Rather, the idea is that we are aware of the fact that there is a much bigger need for non-wikitext, more-widely-adopted formats (HTML being just one of them) and that the APIs should take that into account. In practice, this would mean that the default output format would not be wikitext, but should the client require it, they can specify it.

Can we please rephrase this? Suggestion:

API/NOWIKI

APIs MUST not require clients to parse wikitext, unless this is the purpose of the API.

I wouldn't endorse this phrasing, sorry. The purpose of the API is to provide or convey a specific piece of information, not a format.

In T220657#5123431, @dbarratt wrote:

I left a comment on the talk page, but I do not believe we can adopt the section "To provide a web application that can be freely used to collaboratively collect and share knowledge" since it codifies a product/business assumption.

I'm confused by that statement. The architecture principles are driven by the product/business goals and requirements. They apply these goals and requirements to the technical realm, so of course they "codify" them. Why would that be a problem?

Or do you mean that this point assumes a product goal or priority that doesn't have consensus? The requirement that MediaWiki shall be usable by third parties has been established for as long as MediaWiki has existed. As such, it makes sens to codify it in the principles. If there is consensus to change this, the architecture principles will change. Can you point me to a decision that says that it is no longer WMF's goal to "provide a web application that can be freely used to collaboratively collect and share knowledge"?

In any case: The architecture principles follow product goals and requirements, they do not dictate them. If there is any principle that is counter to the product goals and priorities as defined by the mid term planning process, please explain that conflict, so we can adjust the principles accordingly.

In T220657#5134234, @thiemowmde wrote:

I'm sorry, but I'm not ok with putting me in a corner where I have to defeat my need to be allowed to work with wikitext. That's pretty much exactly how I don't want these principles to be used.

If I understand correctly, you want to still be able to process wikitext. Nothing in the principles is intended to prevent that - they just say that you should not need to process wikitext, all information should be available in a form different from wikitext. In other words, wikitext should not be part of the API (though it can be loaded and stored via the API). The reason for this is that there is no formal specification for wikitext, and it's effectively impossible to correctly process outside of MediaWiki.

You are right that there is currently nothing that guarantees that wikitext will be available via the API - I see no good way to phrase this, since we may in the future have wikis that don't use wikitext but some other markup, or only structured content, and wikitext will then not be available via the API of those wikis.

In uyh72uhhsytptwkh, @daniel wrote:

Releasing MediaWiki as a software product for use by third parties has been part of WMF's "business plan" for as long as it has existed.
Re-assessing the value of this for our mission would certainly be good, but as long as there is no decision to drop the goal of releasing MediaWiki for third parties, it is a goal, and it should be present in the principles.

When the product goals and requirements change, the architecture principles change. The principles apply the product goals to the technical realm. Such such, they indeed codify product goals. But they do not dictate them.

I'm not arguing against releasing MediaWiki as a software product for use by third parties. I'm asking who are those third parties. I think releasing the software under a free license is a completely valuable business strategy that I strongly support. However, that does not mean that the software is for everyone. MediaWiki is a product, and every product needs to solve a problem for someone. I would like to know what that problem is and who has that problem. I tried looking to see if we had done any research on this, but I couldn't find anything, so if I'm missing this please point me in the right direction. :)

To summarize, I think we have not done the work to identify who the customer of MediaWiki is. And by customer, I do not mean user. I mean the person who has a problem, and is experiencing so much pain they are willing to open their wallet in order to relieve that pain. That does not mean we would actually charge them money, it just means, they are willing to pay X amount of money for a solution to their problem.

To put it another way, If we had a company that made blank CDs, and we decided to give them away for free and we asked the people later what problems they had with the CDs, they might say "It doesn't fly far enough" or "my drink slips off of it" because the users we gave the CDs too didn't have the problem the product intended to solve and therefor was using it for a different purpose entirely.

If MediaWiki is the solution what is the problem that it solves, and for whom does it solve it for?

In T220657#5135741, @daniel wrote:

I'm confused by that statement. The architecture principles are driven by the product/business goals and requirements. They apply these goals and requirements to the technical realm, so of course they "codify" them. Why would that be a problem?

Or do you mean that this point assumes a product goal or priority that doesn't have consensus?

Exactly. I think that it codifies who the customer (see above) is, when there hasn't been enough research (as far as I know) to answer that question.

The requirement that MediaWiki shall be usable by third parties has been established for as long as MediaWiki has existed. As such, it makes sense to codify it in the principles.

I totally agree that this is a good, and valuable business strategy.

If there is consensus to change this, the architecture principles will change. Can you point me to a decision that says that it is no longer WMF's goal to "provide a web application that can be freely used to collaboratively collect and share knowledge"?

We absolutely should do this. But that statement is not a problem it's a solution. So what is the problem and who has this problem?

In any case: The architecture principles follow product goals and requirements, they do not dictate them. If there is any principle that is counter to the product goals and priorities as defined by the mid term planning process, please explain that conflict, so we can adjust the principles accordingly.

Let me step though this in a little more detail and hopefully that will bring some clarity:

the basic MediaWiki stack MUST be easy to deploy on standard hosting platforms.

If we don't know the problem that MediaWiki solves, and for whom it solves it for, how could we then say that it must work on "standard hosting environments"? Does our customer use standard hosting environments? Do they even need hosting? If they do need hosting, why don't we provide that (for a fee) or partner with someone else to provide it?

small MediaWiki instances MUST function in low-budget hosting environments.

Again, if we don't know who our customer is, how can we say that we must have the software work on low-budget hosting environments? How do we know they use low-budget hosting environments? If they do, is this something we could provide (for a fee) or partner with someone to provide? Wouldn't that be cheaper than forcing the software to work that way?

it SHOULD be possible to install and upgrade MediaWiki without much technical knowledge.

Does our customer not have a lot of technical knowledge? Do they even need to install it? If so, is that something we could provide (for a fee) or partner with someone else to provide?

it MUST be possible to upgrade MediaWiki without the risk of losing content or disrupting operation.

Does our customer even need to upgrade MediaWiki? Is this something we could do for them?

Again, I think the goal "to provide a web application that can be freely used to collaboratively collect and share knowledge" is a good goal, it's just that that is a solution to a problem and I'm not sure (and maybe this is just my own ignorance) what that problem is and who has that problem. Without knowing that (through research) I'm not sure we can say what MediaWiki needs to look like in order to solve that problem. I don't even know if MediaWiki solves the entire problem. Is that really all our customer needs? Do they need training? Do they need support? Do they need hosting? Do they need custom development? Do they need wiki stewardship? What exactly do they need in order to solve their problem?

In T220657#5136052, @dbarratt wrote:

MediaWiki is a product, and every product needs to solve a problem for someone. I would like to know what that problem is and who has that problem. I tried looking to see if we had done any research on this, but I couldn't find anything, so if I'm missing this please point me in the right direction. :)

Flexible CMS systems like MediaWiki tend to solve a wide range of different problems for different people. If you are looking for more information on what those problems are / who those people are, you could look at the MediaWiki user survey, or the testimonials page, or the EMWCon presentations, or Yaron's great podcast series.

I would certainly agree we as an organization haven't done as much research as we should have, but there isn't anything TechCom can do in a few weeks to fix that multi-year knowledge debt, and the current draft does a good job of extrapolating from what we do know, so I don't think your objection includes anything actionable.

To summarize, I think we have not done the work to identify who the customer of MediaWiki is. And by customer, I do not mean user. I mean the person who has a problem, and is experiencing so much pain they are willing to open their wallet in order to relieve that pain. That does not mean we would actually charge them money, it just means, they are willing to pay X amount of money for a solution to their problem.

Most readers of Wikipedia would probably not be willing to pay for access; that does not disqualify them from being part of the userbase. What fraction of your userbase would be willing to pay for your service/product and how they differ from the "average" user are good questions to ask, but do not make that user segment automatically more important (much less the only important one). There are many other considerations (is providing a free state-of-the-art knowledge management and sharing tool to third parties in line with our mission? our strategic direction of becoming the infrastructure of free knowledge? does it result in more users, developers, experts, partnerships? does it result in some of the work that the WMF would like to see done being done by other organizations? etc).

Also talking about customers is confusing as neither the WMF nor the wider Wikimedia movement provides any paid MediaWiki services. Maybe we should (it's certainly an idea that has been thrown around a lot), but you can't write a policy on top of maybes.

If we don't know the problem that MediaWiki solves, and for whom it solves it for, how could we then say that it must work on "standard hosting environments"? Does our customer use standard hosting environments? Do they even need hosting? If they do need hosting, why don't we provide that (for a fee) or partner with someone else to provide it?

MediaWiki hosting providers exist (even some pretty decent free ones, like Miraheze) but are not widely popular, so yeah, many users use standard hosting and are unwilling to switch away from it.

Again, if we don't know who our customer is, how can we say that we must have the software work on low-budget hosting environments? How do we know they use low-budget hosting environments? If they do, is this something we could provide (for a fee) or partner with someone to provide? Wouldn't that be cheaper than forcing the software to work that way?
Does our customer not have a lot of technical knowledge? Do they even need to install it? If so, is that something we could provide (for a fee) or partner with someone else to provide?
Does our customer even need to upgrade MediaWiki? Is this something we could do for them?

Radical proposals for what the WMF or some partner might be doing in the future are not valid basis for current policy. Currently we don't do any of these, and while there are others who do, most MediaWiki installs are self-managed, so they do need to deal with installation, upgrade and operation in a low-budget environment. The reasonable thing for the policy to do is to codify all that (which is not the same as setting it in stone).

Is that really all our customer needs? Do they need training? Do they need support? Do they need hosting? Do they need custom development? Do they need wiki stewardship? What exactly do they need in order to solve their problem?

There certainly are people who need all that; probably a lot of them. Seems well outside the scope of an architecture policy, though (other than making sure that standard development best practices like documentation are followed, which is something we need anyway).

I'd like to see MediaWiki move towards an architecture that makes operating hosting services easy (there are a number of benefits to that, some of them directly useful to the Wikimedia movement, some not), but even that's not really an architecture-principle-level issue, I think.

The purpose of the API is to provide or convey a specific piece of information, not a format.

This is pretty much the opposite of what you just wrote before:

there is a much bigger need for non-wikitext […] the default output format would not be wikitext […]

How can you say this is not about a format, when everything you say is about one very specific format?

wikitext should not be part of the API […]

So the principles are in fact banning wikitext?

they just say that you should not need to process wikitext, all information should be available in a form different from wikitext.

I'm not sure if it makes sense if people constantly try to explain how the API/NOWIKI sentence I have a problem with was meant. The problem is that the sentence, as it is currently phrased, fails to explain itself. This failure is well demonstrated by the fact that several people tried to explain it already, all using different words to express surprisingly different things in very different ways. It seems everybody does have it's own idea what the sentence means. This is a problem and must be fixed.

May I ask again what the problem with my suggestion is?

APIs MUST not require clients to parse wikitext, unless this is the purpose of the API.

Or can we just ditch the sentence?

In T220657#5137058, @thiemowmde wrote:

How can you say this is not about a format, when everything you say is about one very specific format?

We discuss wikitext explicitly because we historically have the problem that clients needed to process wikitext in order to access certain information, and we want to fix this problem.

The purpose of the API is to communicate information, in a way that is as agnostic to format and encoding as possible, and easy to decode and process on the client. Wikitext is the opposite of that, so it not be the only way to get a certain bit of information.

So the principles are in fact banning wikitext?

No. Wikitext is supported as payload. But the wikitext syntax should not be part of any API spec. ALl information should be accessible without parsing wikitext.

I'm not sure if it makes sense if people constantly try to explain how the API/NOWIKI sentence I have a problem with was meant. The problem is that the sentence, as it is currently phrased, fails to explain itself. This failure is well demonstrated by the fact that several people tried to explain it already, all using different words to express surprisingly different things in very different ways. It seems everybody does have it's own idea what the sentence means. This is a problem and must be fixed.

It seems to me all the people who have tried to explain it are saying the same thing, with different words. The same thing that is already in the original sentence. It seems like that sentence means something to you that it doesn't mean to other people. I'm trying to understand why that is.

May I ask again what the problem with my suggestion is?

APIs MUST not require clients to parse wikitext, unless this is the purpose of the API.

Or can we just ditch the sentence?

The problem with this proposal is that it suggests that there can or should be an API that has the purpose of requiring the client to parse wikitext. That doesn't make sense to me. We can and will have an API that provides wikitext to the client, but the API code should not expect or require the client to be able to parse that wikitext. That of course doesn't mean that the client isn't allowed to parse it.

We could use only the first part of your proposal: "APIs MUST not require clients to parse wikitext". I'd be OK with that, though it seems less clear to me than what is in the draft. It's less clear because it does not directly convey the idea that the API should be designed in the way so that clients have no need to parse wikitext.

The sentence should not be removed. It codifies an important requirement: all information encoded in the wikitext should be available to the client without the need for the client to parse wikitext. The design of the API should assume that clients are unable to parse wikitext. That doesn't forbid clients to parse wikitext, nor does it keep the API from returning wikitext.

Taking a step back, trying to see your point, I can come up with this: it's impossible to predict all kinds of information that a client may want to extract from wikitext, so it is impossible to design an API that removes the need for clients to process wikitext in all cases. I can't think of any useful example, but if I want to contrive one, it it would be a client that specifically analyzes the use of certain features of wikitext. Maybe it collects statistics on how often an unbalanced '' (double-apostrophe) occurs or something.

This case would be OK according to the current wording - the API can still be designed to avoid the need to process wikitext, even though it can't foresee all use cases, and still provides access to the wikitext. With the wording you suggest, we would have to introduce an extra API that servers wikitext for the purpose of being processed by the client, instead of just having the regular APIs that return wikitext anyway, for import/export or just because people want to edit it by hand.

There is no intent to remove the ability to load wikitext, and nothing in API/NOWIKI says that it should be removed. We'll need it on the client for manual editing, if nothing else. Which means that clients always can parse it, if they want to.

This last call was due to close yesterday. However, yesterday's TechCom meeting didn't have sufficient attendance to reach quorum, so the members present (@mobrovac, @Krinkle and myself) decided to extend the last call period until May 1st. This extension also avoids cutting short the ongoing discussions.

After reviewing the discussion so far, we identified three main themes:

1. Scope (mainly raised by @bd808): There are grey areas for which it is unclear whether the architecture principles apply. This reflects that fact that there are grey areas for which the WMF's commitments and responsibilities are unclear. This is unfortunate, but addressing this is outside the scope of the principles. The principles apply to a project in as far as the WMF takes control of and responsibility for that project, or as far as other maintainers choose to apply them.
2. Perpetuating potentially problematic product policy (raised mainly by @dbarratt), such as providing 3rd parties with a web application that runs on shared hosting. The architecture principles codify the established product goals and priorities; revising these may be appropriate, but is out of scope here. The lead paragraph of the draft makes it clear that the architecture principles are intended to evolve along with the product strategy.
3. Allowing clients to parse wikitext (raised mainly be @thiemowmde). The draft says that the API must be designed in a way that avoids the need for clients to parse wikitext. This doesn't mean it can't return wikitext, nor does it say that clients can't parse wikitext.

After discussing the above points (during yesterday's meeting as well as the week before), TechCom found no improvements to be made the draft. The last call continues until May 1st with the text as proposed.

I think I have to go crazy right now. By saying that wikitext is not allowed to be part of any API any more you are banning APIs like export and action=edit that either return or expect wikitext. It even bans the action=parse API whose entire purpose is to accept wikitext. Neither of these APIs does make any sense any more when we require clients to not have any knowledge about wikitext any more.

Why is such a fundamental political decision hidden in a document labeled "architectural principles", as if it would just be a minor technical detail?

The problem with this proposal is that it suggests that there can or should be an API that has the purpose of requiring the client to parse wikitext. That doesn't make sense to me.

I really don't get that. How does a raw wikitext export "not make sense"?

the API code should not expect or require the client to be able to parse that wikitext.

What is "API code"?

What's the problem with having a raw wikitext export?

Do we need to add optional parse parameters to every API that happens to return wikitext just to conform to this new requirement? Why?

In T220657#5137411, @thiemowmde wrote:

I think I have to go crazy right now. By saying that wikitext is not allowed to be part of any API any more

That is not what it says. What I said earlier is that the wikitext syntax is not "part of the API" in the sense that clients do not have to know anything about that syntax in order to use the API, or in order to access the information present on the wiki. The API is supposed to offer alternative ways to access that information. This is already the case for almost everything.

you are banning APIs like export and action=edit that either return or expect wikitext.

The "ban" is on relying on the client's ability to parse wikitext. These APIs return and except wikitext, and they should continue to do that. No problem. They do not require the client to know the wikitext syntax.

It even bans the action=parse API whose entire purpose is to accept wikitext.

No, quite to the contrary. That API offers the server's ability to parse wikitext as an API, so the client has no need to process wikitext. That's exactly the idea behind this principle: wikitext parsing should be done on the server side, it should not be necessary on the client side.

Neither of these APIs does make any sense any more when we require clients to not have any knowledge about wikitext any more.

Why? It seems to me, these APIs can be used without any problem without the client knowing anything about the syntax of wikitext. Use the API to load wikitext into an editor, have people edit it, save it back via the API. No need for the client to know the syntax at all. Can the editor still know (some of the) wikitext syntax to do syntax highlighting? Sure! But the API doesn't require the editor to know wikitext syntax.

Let me try an re-phrase the principle. How about this?

APIs MUST be designed to avoid any need for clients to have any knowledge of wikitext syntax. The "single source of truth" for parsing and rendering wikitext is on the server side.

Does that sound better to you? To me it seems to need a lot more words to say the exact same thing, and it's not very elegant. But if it serves to removes a source of confusion, I'd be fine with it.

These APIs […] do not require the client to know the wikitext syntax.

Sure they do. What's the point of delivering wikitext to a client that can't do anything with it? What's the point of an edit API that accepts wikitext if the client does not know anything about the format? What the client receives and submits is binary garbage then. This is not only pointless, but dangerous, as such a client will most certainly destroy wiki pages.

Use the API to load wikitext into an editor, have people edit it, save it back via the API. No need for the client to know the syntax at all.

It appears we are not talking about the same thing when we use the word "client". The moment the client in your example presents the wikitext in an editor, this client does contain knowledge about the syntax.

• Any kind of WYSIWYG editor is only possibly if it knows at least key elements of the syntax.
• Syntax highlighting is only possible if at least the most relevant syntax features are known.
• Even if it's just a textarea with a few convenient insert buttons (e.g. a button that inserts [[]]), it means this client contains knowledge about these syntax elements.
• Even if it's just a help page that explains the syntax, this help page is an integral part of the client. Even if this help page is on mediawiki.org.

APIs MUST be designed to avoid any need for clients to have any knowledge of wikitext syntax. The "single source of truth" for parsing and rendering wikitext is on the server side.

Demanding that clients should not know anything about the syntax actually makes it worse. The last sentence is fine. Why not just use that?

In T220657#5139742, @thiemowmde wrote:

These APIs […] do not require the client to know the wikitext syntax.

Sure they do. What's the point of delivering wikitext to a client that can't do anything with it? What's the point of an edit API that accepts wikitext if the client does not know anything about the format? What the client receives and submits is binary garbage then.

There are plenty of uses for an "opaque", content-agnostic CRUD API: import/export for instance, and manual editing.

This is not only pointless, but dangerous, as such a client will most certainly destroy wiki pages.

So, the problem is the need for clients to construct wikitext, or to modify it, as classic "wiki bots" often do? Something like "find the references section, and in that section, add templae Foo with such and such parameters"?

In this context, the intent of this principles is indeed to make it possible to do this kind of thing without touching wikitext - by performing that manipulation on an annotated HTML representation of the document, for example. This is much safer and easier, since there are standard libraries for reliably manipulating an HTML DOM, while wikitext manipulation is inherently brittle.

It is however not the intent to ban such direct manipulation of the wikitext. The principle just calls for APIs to provide alternatives ("avoid the need").

The idea behind this is that automated editing should move away from direct manipulation of wikitext, and rely on structured data (JSON) or semi-structured data (HTML DOM) instead. That doesn't mean wikitext manipulation is to be forbidden. But it does mean that other data formats are preferred.

I don't expect bots to shift away from messing with wikitext any time soon. We still need to create good alternatives, that's what this principles is about. The point here is to enable them to shift away from wikitext, not to force them.

It appears we are not talking about the same thing when we use the word "client". The moment the client in your example presents the wikitext in an editor, this client does contain knowledge about the syntax.

• Any kind of WYSIWYG editor is only possibly if it knows at least key elements of the syntax.

That's incorrect. VE is entirely based on annotated HTML. All wikitext processing happens on the server. VE would work the same even if we didn't have wikitext on the backend at all.

• Syntax highlighting is only possible if at least the most relevant syntax features are known.
• Even if it's just a textarea with a few convenient insert buttons (e.g. a button that inserts [[]]), it means this client contains knowledge about these syntax elements.

That is true, but unrelated to the API. Of course a wikitext editor will contain knowledge of the wikitext syntax. But it doesn't need that knowledge to interact with the API. It needs it to help the user.

• Even if it's just a help page that explains the syntax, this help page is an integral part of the client. Even if this help page is on mediawiki.org.

That's not "processing wikitext". But I agree that the new proposed wording would be problematic in that respect.

APIs MUST be designed to avoid any need for clients to have any knowledge of wikitext syntax. The "single source of truth" for parsing and rendering wikitext is on the server side.

Demanding that clients should not know anything about the syntax actually makes it worse. The last sentence is fine. Why not just use that?

Because it's not sufficient. The principles is intended to guide the design of APIs. The requirement for the design is that all information encoded in wikitext needs to be available to the client without the need to parse wikitext. It's not just about the single source of truth, it's about how content can be accessed.

But I agree that the phrase "knowledge of wikitext syntax" is problematic. The original wording is better.

There are plenty of uses for an "opaque", content-agnostic CRUD API […]

A straight ex/import is the only one I can think of.

the problem is the need for clients to construct wikitext, or to modify it […]

No. Just consuming it as well, e.g. for statistical analysis. Not everybody is able and willing to traverse possibly millions of DOM trees.

The principle just calls for APIs to provide alternatives ("avoid the need").

This is not the same. There is nothing wrong with "providing alternatives", while having a requirement to "avoid" something is effectively a ban.

That's incorrect. VE […]

I did not mentioned VisualEditor. I'm talking about any kind of syntax-agnostic editor.

But it doesn't need that knowledge to interact with the API.

Sure it does. Otherwise it will pass binary garbage to an API that expects wikitext.

That's not "processing wikitext".

It's certainly "processing of wikitext" when wikitext is presented in a textarea and users can fiddle with it.

The requirement for the design is that all information encoded in wikitext needs to be available to the client without the need to parse wikitext.

This is impossible. There is an almost endless amount of Parsoid tickets because it still messes with meta-information present in the wikitext (mostly whitespace, e.g. indention of template parameters). There are certainly "information encoded in wikitext" that are not available in any other format. But even if we ignore syntax styling and focus on the content, what does it mean when we have to make "all information encoded in wikitext" available in a format that doesn't require parsing? How can we even know what information can be extracted by parsing wikitext? What if, for example, I want to scan a wiki for dates, names, or places that are not marked with a template or link? I need to parse it myself no matter what.

Why do we need to make live painful for people who just want to provide and/or consume the raw data as it is in the database?

There is a parse API. Just pass wikitext to that API if you don't want to parse it yourself. What is the issue the new principle tries to solve? I mean, do we actually have a problem with people introducing tons of new APIs that don't have any alternative? If not, why do we need to establish a principle like that?

I just realized that the API/STRUCTURED principle fully covers the need to "provide alternatives" already. API/NOWIKI does not add anything constructive what's not already said in API/STRUCTURED. API/NOWIKI is the only destructive principle. It's the only one that says NO.

Can we please remove it and move on?

@thiemowmde we are turning in circles. I can only offer to try to resolve this when I'm at the WMDE office on Monday.

Does anyone else see the problem Thiemo sees? Do you read the API/NOWIKI principles to prohibit clients from processing wikitext, or to ban APIs that return or accept wikitext?

In T220657#5140781, @daniel wrote:

Do you read the API/NOWIKI principles to prohibit clients from processing wikitext, or to ban APIs that return or accept wikitext?

To me, API/NOWIKI prompts the following thought process: if I am creating a new API (regardless of its purpose/function), I have to make sure that formats other than wikitext are supported by it as well. That does not imply I can't provide wikitext, though.

Stating we would turn in circles is certainly a way to dodge uncomfortable questions.

That does not imply I can't provide wikitext […]

It's literally called "no wiki(text)". It would already help to not name it like this, but API/WIKITEXT, for example. Also, it does not say "provide" but "process", without explaining what this means.

Maybe it would help to list some real-life (or made-up but plausible) examples of what we think this principle requires / forbids. (IMO that would be useful as a companion document for all the other principles as well, although I realize it would be a fair amount of work.)

The obvious one I suppose is page editing: circa 2012 you could use the action=parse API to get the HTML version of the page, so API/STRUCTURED was fulfilled; but to make simple edits like changing an infobox parameter on all pages using an infobox, your client had to be able to parse wikitext. Which is to say, the edit API was most definitely not designed to avoid any needs for clients to support wikitext. With the introduction of Parsoid, that has been fixed.

Other than ensuring all the API endpoints that return wikitext can also return HTML and all the endpoints that accept wikitext can also accept Parsoid HTML, could this principle cover anything else?

In T220657#5145540, @Tgr wrote:

Maybe it would help to list some real-life (or made-up but plausible) examples of what we think this principle requires / forbids. (IMO that would be useful as a companion document for all the other principles as well, although I realize it would be a fair amount of work.)

I'd welcome this, but I'll not commit to writing such a document. Perhaps it can be created iteratively, be recording the outcome of discussions about relevant (edge)cases.

The obvious one I suppose is page editing: circa 2012 you could use the action=parse API to get the HTML version of the page, so API/STRUCTURED was fulfilled; but to make simple edits like changing an infobox parameter on all pages using an infobox, your client had to be able to parse wikitext. Which is to say, the edit API was most definitely not designed to avoid any needs for clients to support wikitext. With the introduction of Parsoid, that has been fixed.

It would be fixed with Parsoid if we treated the annotated HTML that Parsoid processes and accepts to be a stable public interface. To my knowledge, this is currently not the case. This principle would however commit us to making Parsoid output a stable public API.

Other than ensuring all the API endpoints that return wikitext can also return HTML and all the endpoints that accept wikitext can also accept Parsoid HTML, could this principle cover anything else?

Yes. For example, it means that a client should be able to get a list of the links added and removed by an edit. This could be done as HTML, but that would be cumbersome. Having a direct JSON representation of this list would be more straight forward. We already have this kind of structured access to lists of links (and images, and categories, etc) for the current version of the page, but we are lacking it for old revisions, and for diffs. To get these, the client still needs to process wikitext; this principles means we'd need to fix that.

I met with @thiemowmde at WMDE today and we came up with a proposal for an alternative phrasing. The key difference is to find a positive wording, instead of a negative one. The proposed new version of API/NOWIKI is:

APIs must be designed to enable clients to process content without the need to parse or manipulate wikitext.

in addition, we propose to change the handle to API/PARSE, to indicate that the parsing should be left to the API (that is, the wiki's server side code is the single source of truth for converting between wikitext and HTML), rather than preventing the use of wikitext.

The question is now: can we agree on the new phasing (and the new handle)? And is it similar enough to the old phrasing that the last call can continue, and does not need to be restarted or extended?

In T220657#5147282, @daniel wrote:

I met with @thiemowmde at WMDE today and we came up with a proposal for an alternative phrasing. The key difference is to find a positive wording, instead of a negative one. The proposed new version of API/NOWIKI is:

APIs must be designed to enable clients to process content without the need to parse or manipulate wikitext.

s/wikitext/wikitext directly/ but otherwise I think the two are equivalent.

in addition, we propose to change the handle to API/PARSE, to indicate that the parsing should be left to the API (that is, the wiki's server side code is the single source of truth for converting between wikitext and HTML), rather than preventing the use of wikitext.

API/DELIVERY or API/FORMAT would be more appropriate here, perhaps, but I don't have strong opinions on this. It's a matter of POV, I guess, and I prefer to look at it from the client's side since it's about APIs.

The question is now: can we agree on the new phasing (and the new handle)? And is it similar enough to the old phrasing that the last call can continue, and does not need to be restarted or extended?

I don't think we need to restart/extend here, unless an objection is raised in the next ~24h. Regarding the last call itself, though, we need a TC quorum to validate it, but we have cancelled the May 1st meeting. Do it over email?

In T220657#5146202, @daniel wrote:

It would be fixed with Parsoid if we treated the annotated HTML that Parsoid processes and accepts to be a stable public interface. To my knowledge, this is currently not the case. This principle would however commit us to making Parsoid output a stable public API.

There is a versioned API and a versioned spec with content negotiation. What else would be needed?

For example, it means that a client should be able to get a list of the links added and removed by an edit. This could be done as HTML, but that would be cumbersome. Having a direct JSON representation of this list would be more straight forward. We already have this kind of structured access to lists of links (and images, and categories, etc) for the current version of the page, but we are lacking it for old revisions, and for diffs. To get these, the client still needs to process wikitext; this principles means we'd need to fix that.

That begs the question: what makes something a valid use case for the purposes of this principle? You can use the wikitext or HTML to get a list of links in old revisions, but there was never an effort to make an API that supports that use case. Scraping infobox content is one of the most common uses of the API (per the distribution of StackOverflow API questions); does this mean the principle requires us to build an API for it? (Not that it would be a bad idea...) If someone starts using wikitext for, say, building structured views of discussion pages, are we required to build an API for that too? How would all these new requirements be resourced and prioritized?

It seems more reasonable to intepret it as saying, if you build a new API for some specific use case, it must provide a "pre-processed" representation. But that seems to be of little practical impact - no one would build a wikitext API anyway, we already have one. So I struggle to see how this principle would be used in practice.

In T220657#5149174, @Tgr wrote:

That begs the question: what makes something a valid use case for the purposes of this principle? You can use the wikitext or HTML to get a list of links in old revisions, but there was never an effort to make an API that supports that use case. Scraping infobox content is one of the most common uses of the API (per the distribution of StackOverflow API questions); does this mean the principle requires us to build an API for it? (Not that it would be a bad idea...) If someone starts using wikitext for, say, building structured views of discussion pages, are we required to build an API for that too? How would all these new requirements be resourced and prioritized?

It seems more reasonable to intepret it as saying, if you build a new API for some specific use case, it must provide a "pre-processed" representation. But that seems to be of little practical impact - no one would build a wikitext API anyway, we already have one. So I struggle to see how this principle would be used in practice.

You are raising a good point here. I'd say that if we *expect* clients to be able to process infoboxes, we *must* build an API for them to do so without parsing wikitext. If we *expect* clients (our own as well as third parties') to be able to provide a structured view of discussion pages, we *must* build an API for them to do so without parsing wikitext.

I think we *should* build such APIs for all the cases you mentioned.

In T220657#5149143, @mobrovac wrote:

I don't think we need to restart/extend here, unless an objection is raised in the next ~24h. Regarding the last call itself, though, we need a TC quorum to validate it, but we have cancelled the May 1st meeting. Do it over email?

That sounds good to me. I'll start an email thread tomorrow.