Page MenuHomePhabricator

New Public Wiki for the API Portal
Open, Stalled, HighPublic

Description

CPT is intending to launch a new Wiki that will act as documentation for the API Gateway.

We would like the wiki to be available as a public wikimedia.org subdomain:

api.wikimedia.org

Details:


Pre-install automatic checklist:

The creation is blocked until these part are all done.


Post install automatic checklist:


Step by step commands:
On deploy1001:
cd /srv/mediawiki-staging/
git fetch
git log -p HEAD..@{u}
git rebase
On mwmaint1002:
scap pull
mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikimedia apiportalwiki api.wikimedia.org
On deploy1001:
scap sync-file dblists "Creating apiportalwiki (T246945)"
scap sync-wikiversions "Creating apiportalwiki (T246945)"
scap sync-file multiversion/MWMultiVersion.php "Creating apiportalwiki (T246945)"
scap sync-file wmf-config/InitialiseSettings.php "Creating apiportalwiki (T246945)"
scap sync-file static/images/project-logos/ "Creating apiportalwiki (T246945)"
scap update-interwiki-cache

End of automatic output

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
brennen moved this task from Backlog to Watching on the User-brennen board.
Urbanecm moved this task from Backlog to Radar on the User-Urbanecm board.Mar 4 2020, 11:14 PM
JJMC89 added a subscriber: JJMC89.Mar 5 2020, 12:24 AM

My understanding is that @eprodromou has already cleared this with Technical Engagement? He's out this week, but I will check with him. @Bmueller, do you have a recollection of this?

Regarding the subdomain, the request was intended to be developer.wikimedia.org, which aligns with industry practice (as would developers.wikimedia.org). Those seem to be available, correct? Perhaps we should consider potential confusion between those and dev.wikimedia.org, however.

bd808 added a comment.Mar 5 2020, 5:52 PM

Regarding the subdomain, the request was intended to be developer.wikimedia.org, which aligns with industry practice (as would developers.wikimedia.org). Those seem to be available, correct? Perhaps we should consider potential confusion between those and dev.wikimedia.org, however.

T179463: Create a single application to provision and manage developer (LDAP) accounts has hoped to use 'developer.wikimedia.org' as a hostname, but that cookie is only lightly licked. Broadly though this needs public discussion among many parties.

Hi @CCicalese_WMF, no, @eprodromou has not discussed this with Technical Engagement yet. My understanding was that the plan was not to duplicate documentation or replacing wikitech or MediaWiki.org, but providing a static/simple dynamic site that directs people into existing documentation, and also focuses on APIs only. Could you provide more context?

My understanding is that @eprodromou has already cleared this with Technical Engagement? He's out this week, but I will check with him. @Bmueller, do you have a recollection of this?

I now understand that this did not happen. I will have discussions around this and comment back here in order to figure out next steps on this task. Thank you for identifying this issue.

WDoranWMF added a subscriber: cicalese.EditedMar 6 2020, 8:46 PM

@greg @Bmueller Thanks for inputting on this, apologies for the delay was out sick yesterday. I'll follow up with @eprodromou and @CCicalese_WMF so we can figure out next steps.

Ah disregard, I see @apaskulin is already on it!

apaskulin updated the task description. (Show Details)Mar 11 2020, 3:47 PM
EBjune added a subscriber: EBjune.Mar 11 2020, 5:11 PM
CCicalese_WMF renamed this task from New Public Wiki for the Developer Portal, dev.wikimedia.org to New Public Wiki for the API Portal.Mar 18 2020, 3:28 PM
CCicalese_WMF moved this task from Developer Portal to API Gateway on the CPT Initiatives board.
Urbanecm lowered the priority of this task from High to Low.Apr 20 2020, 3:10 PM
This comment was removed by eprodromou.
This comment was removed by eprodromou.
apaskulin updated the task description. (Show Details)May 19 2020, 3:16 PM
apaskulin changed the task status from Stalled to Open.May 20 2020, 12:37 AM

I've reviewed the updated project plan with the Developer Advocacy team and incorporated their feedback, so this task should be ready to move forward.

As the project develops, I'll be sending out updates via the stakeholder communications group and continuing to monitor the project talk page, so feel free to reach out with additional feedback!

bd808 rescinded a token.May 20 2020, 12:38 AM
Ladsgroup updated the task description. (Show Details)May 20 2020, 1:34 AM

I've reviewed the updated project plan with the Developer Advocacy team and incorporated their feedback, so this task should be ready to move forward.

As the project develops, I'll be sending out updates via the stakeholder communications group and continuing to monitor the project talk page, so feel free to reach out with additional feedback!

Thanks. So I added the form we use for other project creations (an example: T253029) so my bot would be able to keep track of parts needed and the parts that are done. It seems there are some missing bits, let me know if I can help with determining any of it.

Another question: You said the wiki you want will be public. By public, you mean connected to SUL and basically everyone can edit it or you want it to be a fishbowl wiki (it's public but only small group of people can edit, an obvious example is foundation.wikimedia.org) which won't be connected to SUL.

Sorry for asking lots of questions. Hope we can move forward with this soon.

Ladsgroup updated the task description. (Show Details)May 20 2020, 1:44 PM

Another question: You said the wiki you want will be public. By public, you mean connected to SUL and basically everyone can edit it or you want it to be a fishbowl wiki (it's public but only small group of people can edit, an obvious example is foundation.wikimedia.org) which won't be connected to SUL.

We would like to manage the permissions as described in T249834, which is a variation on a pure fishbowl wiki. Logged-in users will be able to use a set of Special pages to manage their app credentials; they won't be able to edit content pages, but will be able to edit talk pages.

Thanks for your help!

Reedy added a comment.May 20 2020, 2:51 PM

Another question: You said the wiki you want will be public. By public, you mean connected to SUL and basically everyone can edit it or you want it to be a fishbowl wiki (it's public but only small group of people can edit, an obvious example is foundation.wikimedia.org) which won't be connected to SUL.

We would like to manage the permissions as described in T249834, which is a variation on a pure fishbowl wiki. Logged-in users will be able to use a set of Special pages to manage their app credentials; they won't be able to edit content pages, but will be able to edit talk pages.

This feels a lot like us using MW as a CMS when we tell other people MW is a wiki not a CMS, and if they want to do this, they should potentially be using something else...

This feels a lot like us using MW as a CMS when we tell other people MW is a wiki not a CMS, and if they want to do this, they should potentially be using something else...

The notion that MediaWiki cannot be used as a CMS is outdated. There are numerous examples of MediaWiki being used very effectively as a CMS. For example, all of Splunk's user-facing documentation is managed in MediaWiki maintained by a team of tech writers. The same features, including maintaining edit history, attribution, and others) that make MediaWiki excellent for its primary purpose of supporting encyclopedic content in the Wikipedias also make a powerful platform for certain CMS applications.

Another question: You said the wiki you want will be public. By public, you mean connected to SUL and basically everyone can edit it or you want it to be a fishbowl wiki (it's public but only small group of people can edit, an obvious example is foundation.wikimedia.org) which won't be connected to SUL.

We would like to manage the permissions as described in T249834, which is a variation on a pure fishbowl wiki. Logged-in users will be able to use a set of Special pages to manage their app credentials; they won't be able to edit content pages, but will be able to edit talk pages.

Thanks for your help!

Sorry, I'm a little bit confused. Do you want it to be connected to SUL? (in that case, it's not a fishbowl wiki)

Sorry for the confusion. We do want it to be connected to SUL, but we want to configure access as specified in T249834.

Dzahn added a subscriber: Dzahn.May 21 2020, 11:57 AM

potential confusion between those and dev.wikimedia.org

dev.wikimedia.org creation has been an open ticket since a couple years. T67074

Among the problems it was trying to solve was "Where do I find out about the existence of: doc.wikimedia.org wikitech.wikimedia.org etc". Adding yet another separate site may make this worse rather than better?

not to duplicate documentation or replacing wikitech or MediaWiki.org, but providing a static/simple dynamic site that directs people into existing documentation, and also focuses on APIs only. Could you provide more context?

https://doc.wikimedia.org already is the site for documentation and automatically generated. Couldn't it be used for the API documentation?

Over the years we have seen a trend of special case wikis being created and they have to be supported basically forever. Introducing new wikimedia.org subdomains is also something that should not be taken too lightly. Keep in mind once introduced they will have to stay redirects forever. Is it really making it easier for people to find documentation if it its split between more different sites?

WDoranWMF raised the priority of this task from Low to High.May 27 2020, 6:55 PM

hey @Ladsgroup thanks again for helping out with this. Does @CCicalese_WMF answer unblock us for this?

Sorry for the confusion. We do want it to be connected to SUL, but we want to configure access as specified in T249834.

I have strong worries about this model. I don't believe we've ever done that before, and I'd be concerned about how stable that would be in practice.

@Jdforrester-WMF what are the concerns/likely problems and how can we determine if they are blockers?

@Jdforrester-WMF what are the concerns/likely problems and how can we determine if they are blockers?

The entire assumption that you can have an SUL wiki with locked down groups relies on large areas of our code to work the way it's hoped it will, and you can rely on it for access control.

That's a pretty heroïc assumption; we've repeatedly said over many years that people should not ever do this, and instead put their MW install behind a VPN, or at least have an entirely isolated set of user accounts and groups (like we do for fishbowl wikis).

Of course, the general area of permissions/rights has got better over time as we've paid down technical debt, so maybe it'll work. But it seems to be just asking for complications.

hey @Ladsgroup thanks again for helping out with this. Does @CCicalese_WMF answer unblock us for this?

Also, there's several TBD points in the task description I added that are needed to make the initial configuration patch is not answered yet. Please look at a similar wiki creation ticket (an example: T253029)

Change 599273 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/dns@master] Add api.wikimedia.org and api.m.wikimedia.org DNS entries

https://gerrit.wikimedia.org/r/599273

@Ladsgroup Thanks, I have passed these to @apaskulin and @eprodromou to get answered.

apaskulin updated the task description. (Show Details)May 28 2020, 3:32 PM

@Ladsgroup should be up to date now, thanks @apaskulin

Change 599751 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Add api.wikimedia.org

https://gerrit.wikimedia.org/r/599751

I need the DNS patch and the puppet patch be merged and deployed before I can continue. SRE team should take a look.

Dzahn removed a subscriber: Dzahn.May 29 2020, 9:23 AM
Ladsgroup updated the task description. (Show Details)May 30 2020, 2:05 PM
Maintenance_bot updated the task description. (Show Details)
Ladsgroup added a comment.EditedMay 30 2020, 2:28 PM

okay, to recap:

  • The dns and puppet patches are blocker for me
  • The comments made by @Jdforrester-WMF hasn't been answered yet. I think this is really important as this can't be easily changed later. Either a fishbowl wiki or a public wiki connected to SUL. We might get away with a public wiki (everyone can create account) without being connected to SUL but as I said, these are really hard to fix later.
  • Nitpick: "API Portal" doesn't look right as project name, Do you mean "Wikimedia API Portal"? It's not that important as it can be changed later.

I'm sorry for being so naggy and asking lots of questions, but creating a wiki is a really complex task. Look at our playbook: https://wikitech.wikimedia.org/wiki/Add_a_wiki

Ladsgroup updated the task description. (Show Details)May 30 2020, 2:46 PM

okay, to recap:

  • The dns and puppet patches are blocker for me

Ok, we will follow up on this once we've unblocked the other issues below.

  • The comments made by @Jdforrester-WMF hasn't been answered yet. I think this is really important as this can't be easily changed later. Either a fishbowl wiki or a public wiki connected to SUL. We might get away with a public wiki (everyone can create account) without being connected to SUL but as I said, these are really hard to fix later.

@CCicalese_WMF has suggested we setup a test wiki in beta so we can fully test out the permission model we want to use. @Jdforrester-WMF would this be reasonable as an approach?

  • Nitpick: "API Portal" doesn't look right as project name, Do you mean "Wikimedia API Portal"? It's not that important as it can be changed later.

@apaskulin Could you answer for this?

I'm sorry for being so naggy and asking lots of questions, but creating a wiki is a really complex task. Look at our playbook: https://wikitech.wikimedia.org/wiki/Add_a_wiki

No worries, thanks for your help.

apaskulin updated the task description. (Show Details)Jun 1 2020, 3:42 PM

Thanks, @Ladsgroup! I've updated the project name to Wikimedia API Portal.

Reedy added a comment.Jun 2 2020, 6:27 PM

James just pointed out that while "api" is not currently a language code, it's certainly possible it might be in future. Certainly any two or three characters potentially fall under this

[19:21:12] <James_F> apa is the code for the Apache macrolanguage cluster.
[19:21:25] <James_F> So it'd be plausible for one of the Apache languages to be granted api in the future.

As such, for the dbname we shouldn't just use apiwiki (to save pain later) as the dbname.

At point of creation, dbname doesn't matter at all, as we can map it to the hostname

This obviously affects T246945 too

As such, wikimediaapiwiki or something is suggested. As long as it's something obvious (ie not labswiki == wikitech), it doesn't matter too much, so definitely not much thought needs to be put into this

Reedy added a comment.Jun 2 2020, 6:30 PM

I would also suggest "apiwikimedia" is possibly not used, because the wikimedia prefix is used for chapter wikis as is... see metawiki, commonswiki etc

We are fine with your choice of dbname. You could also consider 'apiportal', but we are OK with whatever you think is best.

Reedy added a comment.Jun 2 2020, 9:51 PM

apiportalwiki works! :)

Another thing to think about is probably any stock/default extensions we want to turn off on the wiki

Reedy added a comment.Jun 2 2020, 10:13 PM

Another thing to think about is probably any stock/default extensions we want to turn off on the wiki

And relatedly... Yes, the developed skin is going to be the default.. But do we want to allow users to use the other Wikimedia deployed skins? Or don't enable any of them?

Change 601924 had a related patch set uploaded (by Reedy; owner: Reedy):
[operations/puppet@production] Make api.wikimedia redirect to mobile site

https://gerrit.wikimedia.org/r/601924

Reedy added a comment.Wed, Jun 3, 1:14 AM

^ What's the expection of mobile browsing of this site?

And relatedly... Yes, the developed skin is going to be the default.. But do we want to allow users to use the other Wikimedia deployed skins? Or don't enable any of them?

No, we don't want to allow users to use any other skins.

What's the expection of mobile browsing of this site?

We expect the site to work on mobile devices, but we don't need the site to redirect to m.api.wikimedia.org. Let me know if this answers your question.

@Ladsgroup Are there blockers on this that CPT should resolve?

Change 608702 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikimediaMessages@master] Add messages for apiportalwiki

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/ /608702

@Ladsgroup Are there blockers on this that CPT should resolve?

Hey, Yes. The SRE patches (one in DNS and one in puppet has not been merged)

Change 608702 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@master] Add messages for apiportalwiki

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/ /608702

@Ladsgroup Who should be a reviewer for the DNS patch?

WDoranWMF added subscribers: ema, BBlack.EditedTue, Jun 30, 8:07 PM

@BBlack @ema Is it possible for us to get review for 601924

Reedy updated the task description. (Show Details)Tue, Jun 30, 9:02 PM

One thing that hasn't been outlined in this thread is that ultimately the public endpoint for api.wikimedia.org will not function identically to that of other standard wikis. api.wikimedia.org will be served by the Envoy instances that make up the API gateway cluster, and Envoy itself will serve the documentation wiki content to users in addition to calls to the APIs themselves. We'll still need Apache and Mediawiki configuration for the API portal wiki on the appservers but Envoy will be in the path for requests for the wiki content.

To that end, the final DNS configuration at least will not be the same as on other wikis. Currently we do not have an Envoy setup that is ready to go, so in theory normal configuration could continue as a stopgap if time requires it. But given that we have a wiki on beta for the time being, I think we can pause this ticket until we have a viable Envoy configuration in Kubernetes at which point we can make the appropriate LVS & DNS changes

WDoranWMF changed the task status from Open to Stalled.Thu, Jul 2, 1:02 PM