This has been done successfully in the past when it had dedicated resourcing, turning parts of MediaWiki into PHP libraries. Unfortunately funding of that team/project halted (due to the last WMF reorg, I think?). There are probably still documents or phabricator tasks laying around about which other MediaWiki components had been identified back then as possible areas to split out of the monolith. It would also be a good idea to look at how the parts that did get turned into libraries and components are doing in terms of popularity/community/support.

If I am not mistaken, the project umbrella is Librarization .

What would be very great is to market those components and advocate their usages to others. Possibly with dedicated mini websites with documentation, tutorials, use cases etc. We do borrow components from third party projects such as Symfony or Guzzle, I would love our components to be reused by more people and us acting as good upstream. The scope could be larger and cover dedicated software we wrote, notably from SRE: Cumin or docker-pkg come to mind.

So I am wondering whether this session could be made to have a larger scope than just MediaWiki, along the line of "how our community could be a recognized open source actor". Maybe I am too ambitious, nonetheless I am interested in this session.

@Mooeypoo @WMDE-leszek - Would either or both of you be interested in leading this session?

We still don't have an owner identified for this session. If we can't find one by Friday, the session proposal will be closed.

What would be very great is to market those components and advocate their usages to others.

+1.

So I am wondering whether this session could be made to have a larger scope than just MediaWiki, along the line of "how our community could be a recognized open source actor".

+1

Advocates for our packages and software that can be used by others would be great. Publicity is needed, better up to date and modern documentation is needed. There are probably many "norms" that other open source vendors use that we could probably list such as mini sites etc, but I wonder how any of that would fit into the wmf budget or get priority?

Looking again at the task description:

Question: What parts of MediaWiki are ripe for componentization?
Question: What 3rd party components or libraries could we integrate into MediaWiki to make it easier to maintain?

That requires knowledge of MediaWiki internal and nowadays I lack that knowledge and can't really lead a session on that basis. For the 3rd party libraries that probably means knowing a few things about other components from popular framework (be it Laravel or Symfony).

I don't feel qualified for the technical details, but maybe I can do the session lead so that the outcome of the session would be some kind of business plan / grand plan. Potentially identifying rough areas of improvement ( Maintenance.php *shrug*), shareholders that would definitely have a say (MediaWiki team, security..).

Question: How can we advertise our components to the broader open source community?

I could come up with some baits / ideas as to how we could potentially better expose the software we produce. Which really turns out to:

• setting up an infrastructure to host projects
• identify maintainers
• have clear paths of communications (be it irc/slack/mailinglist/forum/wiki talk page whatever)
• create a dynamic (changelogs prepared and announced), present to conference / record presentation videos
• work with communication (?) for the story telling

Etc.

@Addshore wrote:

I wonder how any of that would fit into the wmf budget or get priority?

My theory is that if we can make a "business case" for it, we should be able to get a budget / resource allocation. One can start small and fail fast to reduce the spending: a first iteration could focus on just a single lib / software only involving a couple or so people for a few weeks. Then evaluate whether the outcome is any positive (from user feedbacks, number of installs or whatever).

I am willing to lead the session, but that would not go into going into the details of which part of MediaWiki to convert. Though that is definitely to be taken in account into the discussion, for a one hour session we can't really get our head into the details.

Question: What parts of MediaWiki are ripe for componentization?
Question: What 3rd party components or libraries could we integrate into MediaWiki to make it easier to maintain?

One way to think about it would be to do a thought experiment about which components of core could be replaced with third party components (e.g. Symfony console for maintenance scripts). Then see what's left over, and talk about those pieces as components that could be shared out.

I think it's important to talk about both directions though, as I think we're more likely to have components that we can share with the broader FLOSS world if we make more ready use of existing FLOSS components in our code.

What would be very great is to market those components and advocate their usages to others. Possibly with dedicated mini websites with documentation, tutorials, use cases etc.

Agreed. Was just looking at packagist (https://packagist.org/?query=wikimedia), and it seems like we already have a few components/libraries with potential to be publicized more.

In T234654#5551028, @Gilles wrote:

This has been done successfully in the past when it had dedicated resourcing, turning parts of MediaWiki into PHP libraries.

I'd argue it has been done, but not successfully. Resources were put into creating the libraries, but not into promoting them, and consequently they are not used much, if at all. This is part of the wider theme of how the lack of product thinking within Technology (ie. the job is not done when you have written the code, the job is done when the impact has been reached) hurts some of our software efforts.

Some of the topics we might want to discuss:

• vision (see my talk page comment on the project page - sometimes librarization is claimed to be about improving architecture and code quality, and sometimes about good open source citizenship, and those are very different goals)
• identifying components to librarize - a lot of good planning happened in the past, I'm not sure much value could be added here
• technical plans - how to integrate librarization with dependency injection, the hook system, the i18n system, how to handle complex dependency trees etc. There has been progress on some of these recently (such as making ObjectFactory work with services, or the creation of MessageFormatterFactory) but I'm sure there are still things to discuss
• how should we promote libraries?
• what metrics should we have for libraries? (awareness, promotion, participation...)
• how resources and responsibilities would be allocated for the above steps

+1 to @Tgr reply, I am definitely endorsing that view. Looks like we already have a few key points which we probably want to reflect in this session description?

@hashar, @Tgr, @kostajh - Feel free to edit the description to incorporate any of these ideas if you feel they are worth discussion. At this point, though, the main thing is that we need someone to put their name under "Session Leader(s)", otherwise the session proposal will be closed on Friday. It sounds like hashar might be tentatively interested, but is looking for someone to help on the technical side of things. Anyone interested in co-leading with hashar?

I just wanted to mention this is something that the Platform Engineering is working on as part of our decoupling work. Not sure if that helps in finding a session leader. @daniel would be excellent but I don't think has the time.

Using standard tools, libraries, packages, etc is in my experience often cited as an important ingredient of the developer productivity. I've also heard from numerous WMDE developers that they'd like to be contributing more to non-Wikimedia open source software.

One way to think about it would be to do a thought experiment about which components of core could be replaced with third party components (e.g. Symfony console for maintenance scripts). Then see what's left over, and talk about those pieces as components that could be shared out.

That is exactly my thinking when I hear about componentization/librarization of the software that have been created as part of MediaWiki and co. Thanks for putting it nicely into words @kostajh!

Given the broad scope, focusing on one aspect of the librarization, as @Tgr put them seems like a good idea. In my eyes contributing to the wider open source community and establishing using standard libraries are important aspects of the Developer Productivity. While we might be starting with giving out/back libraries created for MediaWiki to the outside developer world, I'd agree that it would likely/hopefully also create more opportunities to also use non-MediaWiki/Wikimedia-created libraries in our software.

The fact that splitting out libraries that are beneficial for the general public can be also improving the MediaWiki (above mentioned decoupling work), and hence boosting the developer productivity again, is of course good to keep in mind and a nice bonus!
To me the opening ourselves up for the broader open source community suits the conference theme, and the "Standarization" track better.

Given that this topic has received a significant amount of interest (comments, tokens), I'd be happy to help to make it happen, if folks here were agreeing to focus on the "sharing with the open source community" aspect. Given I've already committed to help leading several other sessions, I wouldn't realistically be able to offer more than being a co-lead (and likely more of a 25% lead). So someone else joining me leading this would be needed. What do others say?
Sadly, given my limited expertise, I couldn't really help if the session was leaning more to improving MW architecture direction.

It might also be a good moment for @daniel to step in saying that CPT have this topic covered and we shouldn't be shaking things too much in this area now, as this might causing more problems than benefits for their work.

@hashar - Doesn't look like anyone else is interested in leading this session. If you're interested in leading it by yourself, please add your name under "Session Leader(s)" in the description, otherwise, I'm going to have to close it as declined.

I am not a good facilitator but if no one else is interested I can try co-leading. IMO it's best to focus on the non-technical aspects (documentation, promotion, metrics), for the technical ones handling them on Phab / in RfCs has worked reasonably well in the past.

@Tgr If you don't mind, I will be happy to try helping with co-leading, e.g. with the facilitating (there will be dedicated people helping with facilitation, BTW).

BTW, focusing on the non-technical aspects sounds like a really sensible move! Figuring out technical details could be parked to more ad-hoc unconference session, and for the further work in usual places.

@WMDE-leszek thank you, that would be great!

What I would see as the expected output of the conference is 1) a shared vision of why componentization and sharing is important (do we all agree that other communities using Wikimedia libraries/tools is a valuable thing to have?), 2) a kind of "product plan" for the whole lifecycle of a component (engineering work, documentation, promotion, tracking etc), 3) a "cost estimate" (if we started doing things properly, how much extra work would it be, and for whom?). That could could be used to suggest the new, expanded process to teams dealing with componentization, and make an ask for resources.

That means it would be great to have participation from the Product side, and also from teams doing regular componentization work (CPT?).

(Programming note)

This session was accepted and will be scheduled.

Notes to the session leader

• Please continue to scope this session and post the session's goals and main questions into the task description.
  • If your topic is too big for one session, work with your Program Committee contact to break it down even further.
  • Session descriptions need to be completely finalized by November 1, 2019.
• Please build your session collaboratively!
  • You should consider breakout groups with report-backs, using posters / post-its to visualize thoughts and themes, or any other collaborative meeting method you like.
  • If you need to have any large group discussions they must be planned out, specific, and focused.
  • A brief summary of your session format will need to go in the associated Phabricator task.
  • Some ideas from the old WMF Team Practices Group.
• If you have any pre-session suggested reading or any specific ideas that you would like your attendees to think about in advance of your session, please state that explicitly in your session’s task.
  • Please put this at the top of your Phabricator task under the label “Pre-reading for all Participants.”

Notes to those interested in attending this session

(or those wanting to engage before the event because they are not attending)

• If the session leader is asking for feedback, please engage!
• Please do any pre-session reading that the leader would like you to do.

I updated the description in the spirit of T234654#5593800. If you think that's not the right topic to focus on please don't hesitate to say so.

Librarization spreads code across various repositories and increases overhead:

• Less visibility
• Complicates contributing (lot of effort to bump library for WMF production)
• Makes compatibility tracking a nightmare (core updates some library, breaks some extension down the line)
• Automation and standard tooling and processes far from what we have for extensions/skins

In context of jquery.i18n and jquery.uls, we did not get much benefits. Some people used those libraries, but not that much was contributed back. There was and still is increased overhead of first updating libraries in GitHub, then rolling into UniversalLanguageSelector extension. There was also overhead of people asking how to use the library and requesting additional features. Now the technology of those libraries is obsolete and they are in need of rewriting or replacement.

For internal benefits, I see that it can help to create code that is better encapsulated and is small enough for a person/team to be a maintainer off (as opposed to the big vague sections like "Editing" of MediaWiki core). There is usually a tradeoff between good integration and using features available in core (i18n, logging, etc.) that are not librarized.

It would be interesting the discuss whether the bottleneck issue is "lack of outreach" or "lack of tooling/processes" or "lack of resources/culture to build independent libraries".

Complicates contributing (lot of effort to bump library for WMF production)
Makes compatibility tracking a nightmare (core updates some library, breaks some extension down the line)

These are probably only the case as we split code out into new repos etc.
I don't believe we use a mono repo approach anywhere just releasing libraries from there?
I haven't personally thought about if that would work well for us at all, but it's probably worth a thought.

In T234654#5631616, @Nikerabbit wrote:

Librarization spreads code across various repositories and increases overhead:

I agree those are important concerns. I'm a bit worried whether we can squeeze them into the session (60 minutes is not a long time and we are trying to focus on product concerns, not technical concerns). The current proposal is along the lines of "if we do librarization anyway, we should do it well" and doesn't really argue either way on whether we should do it.

It would be interesting the discuss whether the bottleneck issue is "lack of outreach" or "lack of tooling/processes" or "lack of resources/culture to build independent libraries".

Lack of outreach can be a bottleneck for usage by third parties, but not really for librarization itself; lack of tooling would be a bottleneck for doing it in the first place (unless I misunderstand what you mean by it). Lack of culture could be a bottleneck on either level I guess; either by causing us to not librarize or by causing us to make libraries which are not truly independent enough to be usable standalone; but I'm not aware of any indication for the latter. So the other two issues seem less relevant to me for the topic of sharing with the open source community.

4 questions asked during the session
Question 1:

Question 2:

Question 3:

Question 4:

Snapshot of the etherpad notes from the session

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Session Name / Topic
Componentization and sharing with the open source community
Session Leader: Gergő + Leszek; Facilitator: Birgit + Deb; Scribe: Will
https://phabricator.wikimedia.org/T234654

Notes:

For our librarisation work we are not reaching out and documenting our efforts. To promise people that we will do this is committing resources to do it. What are the requirements in supporting people in actually using our libraries?

Try to put together sort of action plan, what are benefits that can convince product owners to spend resources on this, and what are resources needed?

 While we don't expect to complete in this session, we are hoping to start a seed for a project we can continue and complete offline

We want to plant the seed that it is a shared responsibility, and we want to do this incrementally. We will ask you as a collective intelligence for questions

1: What are the benefits of making components easily usable by the broader open source community? To what extent can we quantify them? Are there any disadvantages?

2: How can we make the components usable by the OSS community?

3: How can we makes these usable outside the community?

4: What components and tools are the most useful and unique to promote broadly

Process

1 Self reflection -1 min

2 Generate ideaas in pairs 2 mins

3 Share and develop ideas - 4 mins

4 All: "What is the idea that stood out in your conversation" - 5 mins

 

[Splitting up room into groups; to discuss questions in phases: self reflection; then generate two or three ideas in pairs; then combine to bigger groups and share / develop ideas; then the "all people" phase]

Try to stick to 2-3 maximum ideas

Is this method clear?

No objections

When you break out please immediately select the person to report back

Question/Statement: It might be good to put the questions on the screen

Shareback for questions:

1: What are the benefits of making components easily usable by the broader open source community? To what extent can we quantify them? Are there any disadvantages?

There were 3 questions wrapped into one, benefits, quantification and disadavantages

Spent a great deal of time talking about benefits

Attraction to 3rd party developers

Could pull in knowledge, especially given isolation of knowledge, other domain experts, more points of view

Could build a brand and draw in candidates ("attractive employer")

Could force to reuse to avoid redundancies

Discrete dependencies could make the component or lib replaceable in the future, if a better component exists

more clear boundaries, isolate domains from each other

We'll be implementing standards, not MVPs

Disadvantages

There could be costs if the lib is not used or picked up (e.g. overengineered implementation as a library with features that will not be used)

Could be seen as adding to the maintenance burden

Potential dependency if there are many libs depending on diff version of a 3rd party

Contributors could increase bug count (might not mean it is failing but just that bugs are found at a higher rate)

The time for bugs to be fixed could be decreased

2: How can we make the components usable/discoverable by the OSS community?

First step, is if we keep it for ourselves no one can use it, so package and release it to package managers (be part of the ecosystem)

We need to make good software, README, docs, tutrorials, usecases - need to understand the purpose clearly

have templates/guidelines on what our minimal README / docs / tutorials should look like for our components

what is are appropriate dependencies for a component ?

fix the feedback cycle, so people know where to go and the software can improve

You need to be able to sell it - how do you sell it? We are the best!

Be discoverable/SEO (via google and package manager)

You need to highlight the problems it solves (document unique selling point) 

One page show case

Videos with quick tutorials or demos

Once you have the above, you are ready to reach out

Stackoverflow....

Promote at conferences

Deployment on the International Space Station!

Proper advocacy/marketing for components

Need a project manager

Triage backlog, set priorities

Need developers, developers, developers

Need users.

3: How can we monitor usage of our components outside Wikimedia? (Should we?) What resources are require

If we figuring out if this is useful, we feel the resources will be useful

Stars in github, proxy of usage

Download numbers

Heavily skewed by CI

Opt in api wikipiary method, volun-tolds you

Ping backs

Ethically controversial

Opt in/out

Focused on Opt in

Question of should we?

Only if we really want to build a community and expend resources on this focus

INFO: Dependecy tab as github as a means to quantify usage

4: What/Which software components inside MediaWiki/Wikimedia software landscape are unique and the most valuable to be promoted in the broader open source community?

4 s/ws, not necessarily the four most important

Each of us had a sense of 1 or 2:

Banana, pure js client for message localization system (i18n)

Universal language selector

UI for wikidata query service (data visualization)

OGV.js, pure js webm reader, to make sure webm is read and displayed properly everywhere

 Question: I understood the question as  what are the possible candidate "components" to be extracted as a library from MediaWiki/Wikimedia (i.e. not existing libraries/packages yet)

Group understood it as elements that could already be componentised

 Thanks all for your contributions, Next we will package this into documentation, which will be v 0.1

If you are interested in making this happen, there will be updates on the phab task with details so that it can be turned into a document for decision makers

Given the question given to the group 4, i.e. "What/Which software components inside MediaWiki/Wikimedia software landscape are unique and the most valuable to be promoted in the broader open source community?" was interpreted differently by the group ("which is existing - and, somewhat at least, extracted - components would be the best to promote to broader open source community"), hence the question was rephrased and published in the conference space, asking participants to drop and vote on ideas (just as an exercise, to gather ideas and views of participants).

Results are captured in the picture: