Page MenuHomePhabricator

New Public Wiki for the API Portal
Closed, ResolvedPublic

Description

CPT is intending to launch a new Wiki that will act as documentation for the API Gateway.

We would like the wiki to be available as a public wikimedia.org subdomain:

api.wikimedia.org

Details:

  • Language code: en
  • Site URL: api.wikimedia.org
  • Project name: Wikimedia API Portal
  • Project name (English): Wikimedia API Portal
  • Project namespace: API Portal
  • Project talk namespace: Discuss API Portal
  • Project logo: https://commons.wikimedia.org/wiki/File:Wikimedia-logo_black.svg
  • Timezone: UTC
  • Local file uploads: no
  • Database name: apiportalwiki
  • Extra details: group0.
  • Shard: s5

Pre-install automatic checklist:

The Wiki is ready to be created.


Post install automatic checklist:


Step by step commands:
On deploy1001:
cd /srv/mediawiki-staging/
git fetch
git log -p HEAD..@{u}
git rebase
On mwmaint1002:
scap pull
mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=muswiki en wikimedia apiportalwiki api.wikimedia.org
On deploy1001:
scap sync-file wmf-config/db-eqiad.php "Creating apiportalwiki (T246945)"
scap sync-file wmf-config/db-codfw.php "Creating apiportalwiki (T246945)"
scap sync-file dblists "Creating apiportalwiki (T246945)"
scap sync-wikiversions "Creating apiportalwiki (T246945)"
scap sync-file multiversion/MWMultiVersion.php "Creating apiportalwiki (T246945)"
scap sync-file static/images/project-logos/ "Creating apiportalwiki (T246945)"
scap sync-file wmf-config/InitialiseSettings.php "Creating apiportalwiki (T246945)"
scap update-interwiki-cache

End of automatic output

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@Jdforrester-WMF what are the concerns/likely problems and how can we determine if they are blockers?

The entire assumption that you can have an SUL wiki with locked down groups relies on large areas of our code to work the way it's hoped it will, and you can rely on it for access control.

That's a pretty heroïc assumption; we've repeatedly said over many years that people should not ever do this, and instead put their MW install behind a VPN, or at least have an entirely isolated set of user accounts and groups (like we do for fishbowl wikis).

Of course, the general area of permissions/rights has got better over time as we've paid down technical debt, so maybe it'll work. But it seems to be just asking for complications.

hey @Ladsgroup thanks again for helping out with this. Does @CCicalese_WMF answer unblock us for this?

Also, there's several TBD points in the task description I added that are needed to make the initial configuration patch is not answered yet. Please look at a similar wiki creation ticket (an example: T253029)

Change 599273 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/dns@master] Add api.wikimedia.org and api.m.wikimedia.org DNS entries

https://gerrit.wikimedia.org/r/599273

@Ladsgroup Thanks, I have passed these to @apaskulin and @eprodromou to get answered.

apaskulin updated the task description. (Show Details)May 28 2020, 3:32 PM

@Ladsgroup should be up to date now, thanks @apaskulin

Change 599751 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Add api.wikimedia.org

https://gerrit.wikimedia.org/r/599751

I need the DNS patch and the puppet patch be merged and deployed before I can continue. SRE team should take a look.

Dzahn removed a subscriber: Dzahn.May 29 2020, 9:23 AM
Ladsgroup updated the task description. (Show Details)May 30 2020, 2:05 PM
Maintenance_bot updated the task description. (Show Details)
Ladsgroup added a comment.EditedMay 30 2020, 2:28 PM

okay, to recap:

  • The dns and puppet patches are blocker for me
  • The comments made by @Jdforrester-WMF hasn't been answered yet. I think this is really important as this can't be easily changed later. Either a fishbowl wiki or a public wiki connected to SUL. We might get away with a public wiki (everyone can create account) without being connected to SUL but as I said, these are really hard to fix later.
  • Nitpick: "API Portal" doesn't look right as project name, Do you mean "Wikimedia API Portal"? It's not that important as it can be changed later.

I'm sorry for being so naggy and asking lots of questions, but creating a wiki is a really complex task. Look at our playbook: https://wikitech.wikimedia.org/wiki/Add_a_wiki

Ladsgroup updated the task description. (Show Details)May 30 2020, 2:46 PM

okay, to recap:

  • The dns and puppet patches are blocker for me

Ok, we will follow up on this once we've unblocked the other issues below.

  • The comments made by @Jdforrester-WMF hasn't been answered yet. I think this is really important as this can't be easily changed later. Either a fishbowl wiki or a public wiki connected to SUL. We might get away with a public wiki (everyone can create account) without being connected to SUL but as I said, these are really hard to fix later.

@CCicalese_WMF has suggested we setup a test wiki in beta so we can fully test out the permission model we want to use. @Jdforrester-WMF would this be reasonable as an approach?

  • Nitpick: "API Portal" doesn't look right as project name, Do you mean "Wikimedia API Portal"? It's not that important as it can be changed later.

@apaskulin Could you answer for this?

I'm sorry for being so naggy and asking lots of questions, but creating a wiki is a really complex task. Look at our playbook: https://wikitech.wikimedia.org/wiki/Add_a_wiki

No worries, thanks for your help.

apaskulin updated the task description. (Show Details)Jun 1 2020, 3:42 PM

Thanks, @Ladsgroup! I've updated the project name to Wikimedia API Portal.

Reedy added a comment.Jun 2 2020, 6:27 PM

James just pointed out that while "api" is not currently a language code, it's certainly possible it might be in future. Certainly any two or three characters potentially fall under this

[19:21:12] <James_F> apa is the code for the Apache macrolanguage cluster.
[19:21:25] <James_F> So it'd be plausible for one of the Apache languages to be granted api in the future.

As such, for the dbname we shouldn't just use apiwiki (to save pain later) as the dbname.

At point of creation, dbname doesn't matter at all, as we can map it to the hostname

This obviously affects T246945 too

As such, wikimediaapiwiki or something is suggested. As long as it's something obvious (ie not labswiki == wikitech), it doesn't matter too much, so definitely not much thought needs to be put into this

Reedy added a comment.Jun 2 2020, 6:30 PM

I would also suggest "apiwikimedia" is possibly not used, because the wikimedia prefix is used for chapter wikis as is... see metawiki, commonswiki etc

We are fine with your choice of dbname. You could also consider 'apiportal', but we are OK with whatever you think is best.

Reedy added a comment.Jun 2 2020, 9:51 PM

apiportalwiki works! :)

Another thing to think about is probably any stock/default extensions we want to turn off on the wiki

Reedy added a comment.Jun 2 2020, 10:13 PM

Another thing to think about is probably any stock/default extensions we want to turn off on the wiki

And relatedly... Yes, the developed skin is going to be the default.. But do we want to allow users to use the other Wikimedia deployed skins? Or don't enable any of them?

Change 601924 had a related patch set uploaded (by Reedy; owner: Reedy):
[operations/puppet@production] Make api.wikimedia redirect to mobile site

https://gerrit.wikimedia.org/r/601924

Reedy added a comment.Jun 3 2020, 1:14 AM

^ What's the expection of mobile browsing of this site?

And relatedly... Yes, the developed skin is going to be the default.. But do we want to allow users to use the other Wikimedia deployed skins? Or don't enable any of them?

No, we don't want to allow users to use any other skins.

What's the expection of mobile browsing of this site?

We expect the site to work on mobile devices, but we don't need the site to redirect to m.api.wikimedia.org. Let me know if this answers your question.

@Ladsgroup Are there blockers on this that CPT should resolve?

Change 608702 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikimediaMessages@master] Add messages for apiportalwiki

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/ /608702

@Ladsgroup Are there blockers on this that CPT should resolve?

Hey, Yes. The SRE patches (one in DNS and one in puppet has not been merged)

Change 608702 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@master] Add messages for apiportalwiki

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/ /608702

@Ladsgroup Who should be a reviewer for the DNS patch?

WDoranWMF added subscribers: ema, BBlack.EditedJun 30 2020, 8:07 PM

@BBlack @ema Is it possible for us to get review for 601924

Reedy updated the task description. (Show Details)Jun 30 2020, 9:02 PM

One thing that hasn't been outlined in this thread is that ultimately the public endpoint for api.wikimedia.org will not function identically to that of other standard wikis. api.wikimedia.org will be served by the Envoy instances that make up the API gateway cluster, and Envoy itself will serve the documentation wiki content to users in addition to calls to the APIs themselves. We'll still need Apache and Mediawiki configuration for the API portal wiki on the appservers but Envoy will be in the path for requests for the wiki content.

To that end, the final DNS configuration at least will not be the same as on other wikis. Currently we do not have an Envoy setup that is ready to go, so in theory normal configuration could continue as a stopgap if time requires it. But given that we have a wiki on beta for the time being, I think we can pause this ticket until we have a viable Envoy configuration in Kubernetes at which point we can make the appropriate LVS & DNS changes

WDoranWMF changed the task status from Open to Stalled.Jul 2 2020, 1:02 PM
Urbanecm updated the task description. (Show Details)Jul 14 2020, 2:40 PM
Reedy added a comment.Jul 14 2020, 3:02 PM

@Urbanecm Why did you change the dbname? It's not going to be apiwikimedia

@Reedy Hi, that was actually an automated change using https://github.com/Ladsgroup/Phabricator-maintenance-bot/blob/master/new_wikis_handler.py, while I was testing my changes to the bot. I'm sorry for that.

@Reedy Hi, that was actually an automated change using https://github.com/Ladsgroup/Phabricator-maintenance-bot/blob/master/new_wikis_handler.py, while I was testing my changes to the bot. I'm sorry for that.

Yeah, the wiki's db name like almost every other aspect of it (not connecting to app servers, etc.) is special. Maybe we can sorta have a section in the standard form to override the default db name there.

Change 599273 merged by Hnowlan:
[operations/dns@master] Add api.wikimedia.org and api.m.wikimedia.org DNS entries

https://gerrit.wikimedia.org/r/599273

hnowlan updated the task description. (Show Details)Aug 24 2020, 2:03 PM
Ladsgroup updated the task description. (Show Details)Aug 24 2020, 5:30 PM
Ladsgroup changed the task status from Stalled to Open.Aug 24 2020, 5:38 PM
Ladsgroup updated the task description. (Show Details)
Urbanecm updated the task description. (Show Details)Aug 27 2020, 4:59 PM

Change 622853 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[operations/mediawiki-config@master] Initial configuration for apiportalwiki

https://gerrit.wikimedia.org/r/622853

Urbanecm updated the task description. (Show Details)Aug 27 2020, 5:11 PM
Urbanecm updated the task description. (Show Details)Aug 27 2020, 5:11 PM
Urbanecm updated the task description. (Show Details)Aug 27 2020, 5:13 PM

Change 622853 merged by jenkins-bot:
[operations/mediawiki-config@master] Initial configuration for apiportalwiki

https://gerrit.wikimedia.org/r/622853

Change 622867 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add apiportal to case of special mappings

https://gerrit.wikimedia.org/r/622867

Change 622867 merged by jenkins-bot:
[operations/mediawiki-config@master] Add apiportal to case of special mappings

https://gerrit.wikimedia.org/r/622867

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:11:29Z] <urbanecm@deploy1001> Synchronized wmf-config/db-eqiad.php: Creating apiportalwiki (T246945) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:12:58Z] <urbanecm@deploy1001> Synchronized wmf-config/db-codfw.php: Creating apiportalwiki (T246945) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:14:18Z] <urbanecm@deploy1001> Synchronized multiversion/MWMultiVersion.php: Creating apiportalwiki (T246945) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:15:41Z] <urbanecm@deploy1001> Synchronized dblists: Creating apiportalwiki (T246945) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:16:57Z] <urbanecm@deploy1001> rebuilt and synchronized wikiversions files: Creating apiportalwiki (T246945)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:19:13Z] <urbanecm@deploy1001> Synchronized static/images/project-logos/: Creating apiportalwiki (T246945) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-08-27T19:20:29Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Creating apiportalwiki (T246945) (duration: 01m 03s)

Change 623325 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/mediawiki-config@master] Add title for apiportalwiki

https://gerrit.wikimedia.org/r/623325

Change 623325 merged by jenkins-bot:
[operations/mediawiki-config@master] Add title for apiportalwiki

https://gerrit.wikimedia.org/r/623325

Mentioned in SAL (#wikimedia-operations) [2020-09-02T11:36:18Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 796b4fa8d561986a20ad5c9671b696809fa09b67: Add title for apiportalwiki (T246945) (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2020-09-02T15:32:23Z] <hnowlan> Temporarily disabling apache for configuration change T246945

Change 599751 merged by Hnowlan:
[operations/puppet@production] mediawiki: Add api.wikimedia.org

https://gerrit.wikimedia.org/r/599751

Change 623833 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] Add apache config for api.wikimedia.org

https://gerrit.wikimedia.org/r/623833

Change 623833 merged by Hnowlan:
[operations/puppet@production] Add apache config for api.wikimedia.org

https://gerrit.wikimedia.org/r/623833

Change 624037 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] mediawiki: place api-portal entry above wikimedia catchall

https://gerrit.wikimedia.org/r/624037

Change 624037 merged by Hnowlan:
[operations/puppet@production] mediawiki: place api-portal entry above wikimedia catchall

https://gerrit.wikimedia.org/r/624037

hnowlan updated the task description. (Show Details)Sep 3 2020, 2:23 PM

It seems the wiki is live now. Shall we close this ticket and make ones for followups?

Urbanecm closed this task as Resolved.Sep 7 2020, 4:23 PM
Urbanecm claimed this task.

Agreed

Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Sep 7 2020, 5:15 PM

Change 635657 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Add api-portal wiki in prod.

https://gerrit.wikimedia.org/r/635657

Change 635657 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Add api-portal wiki in prod.

https://gerrit.wikimedia.org/r/635657

Change 636431 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/deployment-charts@master] Expose RESTBase for api-portal wiki.

https://gerrit.wikimedia.org/r/636431

Change 636431 merged by jenkins-bot:
[operations/deployment-charts@master] Expose RESTBase for api-portal wiki.

https://gerrit.wikimedia.org/r/636431