Make Wikitech an SUL wiki
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Andrew
	Mar 30 2017, 9:04 PM

Description

We're moving lots of Labs-specific functions off of Wikitech. Once that's done, it should be possible to merge wikitech with the rest of the wikiverse.

See https://meta.wikimedia.org/wiki/Community_Tech/Tool_Labs_support/Tool_Labs_vision for some of the reasoning leading up to this task.

Remove dependency on Semantic MediaWiki
Remove dependency on ~~OpenStackManager~~ OpenStack APIs (Keystone, Nova, etc)
T237773: Move Wikitech onto the production MW cluster (including database)
Replace wikitech as source of two-factor auth protection for LDAP accounts
Replace wikitech as source of LDAP account creation T196171: Developer account creation without OpenStackManager/T179463: Create a single application to provision and manage developer (LDAP) accounts
Connect active LDAP accounts with SUL accounts
Create SUL accounts for inactive LDAP accounts
Unify wikitech local accounts with SUL accounts [the final(?) SUL unification]

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T189531 All Wikimedia developer services should use single sign-on
Open		None	T161859 Make Wikitech an SUL wiki
Open		None	T161553 Remove OpenStackManager from Wikitech
Resolved		Andrew	T150091 Support project creation without OpenStackManager
Resolved		Andrew	T159021 Wikitech error when adding users to projects
Resolved		Andrew	T162097 Create a Horizon panel for managing per-project sudo policies
Resolved		Andrew	T194191 Allow projectadmins to change project roles in Horizon
Resolved		MarcoAurelio	T194670 Copy maintenance/attachLdapUser.php to another extension
Resolved		Andrew	T194794 Add new users to Bastion group whenever they join a project
In Progress		None	T196171 Developer account creation without OpenStackManager
Open		None	T179463 Create a single application to provision and manage developer (LDAP) accounts
Duplicate		None	T189639 Separate LDAP account creation bits from Striker to create a new identity management platform
Open		None	T319405 Create an IDM for Wikimedia developer accounts
Resolved		SLyngshede-WMF	T319407 IDM milestone 1 "Initial development work"
Resolved		None	T319409 Pick a name for the IDM
Resolved		SLyngshede-WMF	T319410 Initial Django project setup
Resolved		SLyngshede-WMF	T319415 Evaluate Striker codebase
Resolved		Marostegui	T320426 Figure out where/how to store IDM internal data
Resolved		SLyngshede-WMF	T320427 Design and implement async LDAP operations
Resolved		SLyngshede-WMF	T320428 Initial IDM puppetisation
Resolved		SLyngshede-WMF	T320430 Create an initial IDM/LDAP image for tests and CI
Resolved		SLyngshede-WMF	T320431 IDM: Central logging on all changes
Open		None	T320603 IDM milestone 2 "Initial limited deployment"
Resolved		SLyngshede-WMF	T320604 Decide on model for serving idm.wikimedia.org
Resolved		None	T320605 Figure out an HA setup for the IDM
Resolved		SLyngshede-WMF	T320794 Extend LDAP to allow storing all necessary attributes
Resolved		SLyngshede-WMF	T320795 Implement a staging setup for the IDM
Resolved		SLyngshede-WMF	T320797 Initial production deployment of the IDM
Resolved		SLyngshede-WMF	T320799 IDM integration into CAS SSO
Resolved		SLyngshede-WMF	T320807 Implement OAuth account validation for linking an account to a wiki account
Resolved		SLyngshede-WMF	T320808 Implement email address validation workflow
In Progress		SLyngshede-WMF	T148048 Store Wikimedia unified account name (SUL) in LDAP directory
Resolved		SLyngshede-WMF	T335091 Determine which sender address to use for email notification
Resolved		SLyngshede-WMF	T340721 Build Debian packages for Bookworm
Resolved		SLyngshede-WMF	T340722 Update IDM servers to Bookworm
Open		SLyngshede-WMF	T320801 Build-out for self service
Resolved		SLyngshede-WMF	T320802 Create a mockup and involve designers
Open	BUG REPORT	None	T338824 Bitu is not responsive
Open		None	T320805 Define the core attribute list managed in the IDM with all stakeholders
Resolved		SLyngshede-WMF	T320806 Consider reusing some wiki data sources for signup/restrictions
Resolved		SLyngshede-WMF	T320809 Figure out a captcha option for IDM
Open		None	T335478 Automatic detection of inactive LDAP account
Open		None	T335485 Automatically deploy documentation to docs.wikimedia.org
Resolved		SLyngshede-WMF	T340636 Implement "Forgot my username" feature
Resolved		SLyngshede-WMF	T340637 Implement "update email" functionality
Open		SLyngshede-WMF	T340717 Implement "source" field on user objects
Open		SLyngshede-WMF	T345168 Live validation of usernames
Resolved		SLyngshede-WMF	T345226 Switch developer account creation to Bitu
Open		None	T338477 `Nova Resource:` namespace should be declared in wmf-config, not in Extension:OpenStackManager
Resolved		• bd808	T53642 Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org
Resolved		yuvipanda	T103995 Build a simple tool to query which instances have which roles / puppet variables
Resolved		• bd808	T161662 Replace SMW data on project instances with a tool using the OpenStack APIs
Resolved		• bd808	T162508 Implement Tool Labs membership application and processing in Striker
Resolved		• bd808	T164787 Keystone permissions exception when trying to approve Tool Labs membership request via Striker
Open		None	T163478 Allow viewing/searching LDAP account creations including date
Resolved		None	T153821 wikitech.wikimedia.org missing from pageviews API
Open		None	T237773 Move Wikitech onto the production MW cluster
Resolved		Marostegui	T167973 Move database for wikitech (labswiki) to a main cluster section
Resolved		Andrew	T188029 Move labswiki database to m5
Resolved		Marostegui	T282074 Audit labswiki grants
Resolved		Andrew	T282209 Enable connection from labweb1001/labweb1002 to s6
Declined		Andrew	T282573 Move labweb/cloudweb hosts to private IPs
Resolved		Andrew	T284106 Replicate & sanitize wikitech data
Resolved		Andrew	T284108 Bring labswiki database tables up to date
Resolved		Ladsgroup	T269348 wikitech database has almost all of its varbinary fields wrong
Duplicate		None	T284736 Create views on labswiki on clouddb* hosts
Resolved		Bstorm	T287442 Create views for labswiki (wikitech)
Declined		None	T237889 Install php-ldap on all MW appservers
Resolved		taavi	T237890 Remove and empty useless user groups
Resolved		Jdforrester-WMF	T196466 Remove 'shell user' right on wikitech
Declined		None	T246405 On wikitech, make sysops unable to edit site JS, and then retire contentadmin
Open		None	T292707 Migrate Wikitech to Kubernetes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Does this encompass T98813: Move wikitech (silver) to HHVM too?

• bd808 mentioned this in T153821: wikitech.wikimedia.org missing from pageviews API.May 8 2017, 3:19 PM

Milimetric added a subtask: T153821: wikitech.wikimedia.org missing from pageviews API.May 9 2017, 2:06 PM

• bd808 closed subtask T53642: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org as Resolved.May 9 2017, 11:02 PM

TerraCodes added a subscriber: TerraCodes.May 9 2017, 11:08 PM

In T161859#3236750, @Jdforrester-WMF wrote:

Does this encompass T98813: Move wikitech (silver) to HHVM too?

Yes, but indirectly I think. Once wikitech is a SUL wiki it should just live in the main wiki cluster rather than on silver. We can actually even do that before full SUL unification. The remaining blocker to moving the wiki to the normal hosting pool is T161553: Remove OpenStackManager from Wikitech.

jcrespo mentioned this in T167973: Move database for wikitech (labswiki) to a main cluster section.Jun 15 2017, 2:50 PM

• bd808 mentioned this in T168470: Setup wikitech, horizon, and striker on new labweb hardware.Jun 20 2017, 9:01 PM

• bd808 updated the task description. (Show Details)Jun 20 2017, 10:19 PM

• bd808 added a subtask: T148048: Store Wikimedia unified account name (SUL) in LDAP directory.

demon mentioned this in T168692: Blocking an account on wikitech should disable LDAP logins.Jul 10 2017, 6:01 PM

Mainframe98 added a subscriber: Mainframe98.Jul 17 2017, 9:05 PM

Would/Is T171570 a blocker for this one?

In T161859#3469379, @MarcoAurelio wrote:

Would/Is T171570 a blocker for this one?

T171570: Rename database 'labswiki' to 'wikitechwiki' seems like a large potential for breaking things with a very small overall impact on the movement. I personally don't think it is worth the effort and risk.

Wouldn't this also break gerrit?

In T161859#3470502, @Paladox wrote:

Wouldn't this also break gerrit?

gerrit uses LDAP account, so as long as LDAP is sane gerrit is okay.

Yep but would new accounts work?

In T161859#3470511, @Paladox wrote:

Yep but would new accounts work?

See the outline at the top which clearly includes Replace wikitech as source of LDAP account creation.

Oh i see. Thanks.

Liuxinyu970226 added a subscriber: Liuxinyu970226.Jul 25 2017, 11:33 PM

BethNaught added a subscriber: BethNaught.Jul 30 2017, 9:25 AM

He7d3r added a subscriber: He7d3r.Aug 30 2017, 7:02 PM

• bd808 mentioned this in T110987: Determine whether wikitech should really depend on production search cluster.Sep 7 2017, 12:31 AM

nshahquinn added a subscriber: nshahquinn.Sep 11 2017, 6:14 PM

• bd808 mentioned this in T179461: Use the term "developer account" for Wikimedia LDAP accounts.Nov 1 2017, 3:13 AM

• bd808 mentioned this in T179463: Create a single application to provision and manage developer (LDAP) accounts.Nov 1 2017, 4:26 AM

• bd808 mentioned this in T103874: Update Vagrant role for Extension:OpenStackManager.Feb 1 2018, 4:52 AM

Quiddity added a subscriber: Quiddity.Feb 23 2018, 9:09 PM

Is there also a "make everything SUL" plan? More specifically, a plan to replace passwords with some kind of SUL-based remote login on gerrit, horizon, logstash etc?

What will happen to accounts already created and SUL accounts? Would we be able to get them merged/migrated/etc?

@Tgr, I don't think we would want to do this. The threat model for horizon/gerrit/etc is quite different from on-wiki account access so I'd prefer that we keep the two separate account types.

That said, a simpler/unified login service for all developer account types might be nice.

In T161859#3997722, @MarcoAurelio wrote:

What will happen to accounts already created and SUL accounts? Would we be able to get them merged/migrated/etc?

My intent is for all existing LDAP accounts (that are still in use) to be associated with a SUL account. This is the "Connect active LDAP accounts with SUL accounts" step in the very high level plan. It is possible today to make this association using https://toolsadmin.wikimedia.org/. The "Replace wikitech as source of LDAP account creation" step would be a good time to introduce this idea more broadly and to start a campaign to get active users to make the association.

In T161859#3997670, @Tgr wrote:

Is there also a "make everything SUL" plan? More specifically, a plan to replace passwords with some kind of SUL-based remote login on gerrit, horizon, logstash etc?

That decision is largely beyond the scope of the Cloud Services team. The projects we own that could do this are Horizon and Striker (toolsadmin). There is a plan for Striker to use SUL via OAuth as the authentication mechanism. I think it would be possible for Horizon as well, but one thing we would probably want to be able to do for Horizon is to require that the SUL account be using 2FA protection.

Framawiki added a subscriber: Framawiki.Feb 24 2018, 4:28 PM

• Mholloway added a subscriber: • Mholloway.Feb 28 2018, 5:30 PM