Page MenuHomePhabricator
Paste P6622

IRC discussion on Evolving the MediaWiki Architecture (Summit 2018 preparation)
ActivePublic
Actions

Authored by daniel on Jan 19 2018, 9:37 PM.
Condensed log of Wednesday's discussion on IRC:
22:10:14 <coreyfloyd> On architecture as it relates to 3rd party: I think that those 2 are very intertwined. The implication is that 3rd parties can only be supported by shipping a installable PHP app in a LAMP stack. This limits our architecture decisions
22:10:56 <tgr> DanielK_WMDE: the proposed session structure seems very bottom-up (identify current / short-term problems, try to compose them into a larger strategy)
22:11:34 <tgr> I wonder if having top-down discussions can fit into the structure, or best not to expect those to happen
22:11:42 <mdholloway> +1 to coreyfloyd; the future of a lot of the node.js work that we're doing over in Reading Infrastructure now seems very uncertain at the moment
22:12:11 <_joe_> DanielK_WMDE: it's not clear to me how we link the discussion about the architecture relates to the strategy process - that would seem to make sense as a top-down approach
22:12:53 <_joe_> start from the strategy direction, which is not very deeply defined though, can be challenging
22:13:00 <Niharika> brion: DanielK_WMDE: Okay. So is there a possibility that the session outcomes might affect the annual plan?
22:13:11 <tgr> DanielK_WMDE: I mean driven by long-term strategy, not specific features
22:13:34 <mdholloway> and that has to do with the idea that we may not want to impose installing node.js on third parties, among other things
22:13:39 <TimStarling> marvin is not even aiming to support wiktionary let alone 3rd parties
22:13:59 <DanielK_WMDE> coreyfloyd, mdholloway: yes, i agree that "how long do we need to support LAMP" is an essential question. and one that cannot be asnered by engineers. it's a product decision.
22:14:08 <tgr> e.g. dbarratt's proposal of switching to a model where most code is maintained by third parties is top down
22:14:54 <tgr> I wrote a proposal https://www.mediawiki.org/wiki/User:Tgr_(WMF)/Turn_MediaWiki_core_into_an_embeddable_library which is also top-down and wondering if the devsummit is the right place to try to discuss it
22:18:32 <DanielK_WMDE> tgr: the dev summit would be a good place to pitch this as a possible strategy, and see if other find it worth exploring. there will unfortunately not be enough time to explore the idea then and there.
22:15:55 <_joe_> so with no product strategy, nt even a high level one, how do we link to it at the tech level?
22:16:53 <coreyfloyd> _joe_ good question - I have given this feedback to Toby/Victoria to let them know it is a blocker for long term technical planning
22:17:58 <_joe_> to be more specific - I think some of the strategy directions can affect our architecture - if we want to store efficiently things other than text, we are in for some necessary evolutions on multiple levels
22:21:51 <DanielK_WMDE> _joe_: re efficiently storing things other than text: one outcome could be to say just that: "*if* the product strategy is to go for multimedia aggressively, we'll need to look into X, Y, and Z".
22:16:04 * brion would eventually love to raise things like "storage integrity" and "security in depth". we still have a "monolithic kernel" design in terms of database access, and it worries me. but is this too big-scope?
22:22:46 <brion> i think we should seriously consider long-term things like "should password hashes be in a big mysql database" and "should someone who gets a privilege escalation on an app server have full access to private data?"
22:16:48 <TimStarling> "timeless vs marvin" is the main question of frontend architecture for me at this
22:18:18 <TimStarling> marvin is a frontend written in typescript, a RESTBase consumer
22:18:33 <TimStarling> imagined as convergence between mobile apps and web
22:20:07 <TimStarling> https://www.mediawiki.org/wiki/Marvin
22:17:03 <DanielK_WMDE> _joe_: I think the strategy process should be informed from the technical side. In particular, needs for scaling and sustaining should be communicated, as well as constraints on feature development
22:19:57 <_joe_> So I think some big questions about our architecture are: do we want a REST API? Should it be part of MediaWiki itself or stay external? How do we move away from the cache-but-also-storage-but-also-router-and-almost-ESB that restbase currently is
22:20:47 <tgr> DanielK_WMDE: to put it more generally, I think we should have a tech vision driving longterm planning, not just a product vision (not that we have a product vision right now...) and it would be cool if the devsummit (the MW architecture session more specifically) could be used to come up with it
22:21:57 <tgr> this year's dev summit is intended to inform strategic planning and so far I don't see that happening, it's more like the usual "let's make a tech roadmap for the next 3 years"
22:20:54 <apergos> I hope we talk about the future of microservices; what types of services are a good fit? what layers should they talk to? how do we enforce that?
22:22:28 <coreyfloyd> _joe_: FWIW on the router question, one of the things that came out of last weeks ATWG meeting is that we need routing in MediaWiki, but also need to route non-MW traffic (like analytics)
22:23:54 <tgr> _joe_: strategic planning is, IMO, deciding where we want to arrive first and on the route to get there only after that
22:24:37 <_joe_> tgr: where depends a lot on what we want to do; I can name you 3 or 4 issues I see as critical for being able to grow in some directions
22:24:45 <tgr> _joe_: if we could come to consensus on those issues, that would already be more productive than all past dev summits, sure
22:24:52 <_joe_> like - how to be able to scale inter-wiki relationships
22:27:22 <DanielK_WMDE> _joe_: i had a long conversation about inter-wiki change propagation with Marko the other day.
22:28:06 <_joe_> DanielK_WMDE: I think that's one of the challenges we have to tackle in order to sustain our *current* feature set for the next few years
22:26:29 <tgr> DanielK_WMDE: what I am trying to say is that the current structure proposed in the task seems to be focused on issues but not ideas (I might be misunderstanding it of course)
22:29:05 <DanielK_WMDE> tgr: i suppose you are right - and you are welcome to change that. My mind is pretty split between "issues we need to address to scale and sustain" and "what'S our technical vision" - but the later depends a lot of where we want to go with products. And that's still quite unclear.
22:29:16 <coreyfloyd> _joe_: DanielK_WMDE I also think we need to split our concerns into short term (scaling current ops) and long term (strategy)
22:29:28 <_joe_> coreyfloyd: I think 5-year plans tend to fall on their back pretty fast, but sure, we could identify some directions that could influence our decisions on such a timeline
22:30:14 <DanielK_WMDE> _joe_: we are not trying to make a 5 year plan. the idea is to identify the questions we need to think about to get a good idea of what we need to do to get to the place we want to be in in 5 years..
22:30:33 <cscott> fwiw, i think our original "1 year plan" in parsing has ended up taking 5 yrs or so
22:30:38 <brion> strategy -> keep at 5 years. actual work plans will change every year. :D
22:32:03 <cscott> making big changes on the wikis can be hard, there's a lot of momentum going in one direction
22:25:19 <DanielK_WMDE> tgr: that (a tech vision) is indeed the idea. the summit is intended to be the start of a process that will deliver that. right now, we have a ton of ideqas and issues. we need to structure them somehow so we can find a vision, and a strategy.
22:25:52 <_joe_> for the future, I'd see things like "how can we ingest more than 50 multimedia files at once"? for instance. Or what brion said, a security approach for the 2020s
22:26:40 <coreyfloyd> I wonder because how far ahead should we plan? What is long term
22:27:16 <_joe_> coreyfloyd: long-term is anything that will develop over more than a year, or will not be needed before a year, I'd say
22:28:19 <coreyfloyd> _joe_ I feel like especially from the product side of the house has a 1 year horizon on features planning… and technology changes fast. So if we could create a tech strategy that lasted 5 years would that be a win?
22:28:39 <apergos> 5 years is a long dang time, 15 years is unimaginable
22:29:46 <apergos> we could talk about what would be needed to support a massive increase in editors (example: 10-fold growth in wikidata)
22:07:38 <Dysklyver> Over the last few years object storage systems such as Amazons S3 service have become very popular, also many people like static storage systems like git (github), currently there appears to be no way to export mediwiki databases to these services, despite the fact that doing so could allow easy backup and/or static html sites to use data stored in this way.
22:30:54 <Dysklyver> Well I think there should be a move away from tons of serverside processes and complex storage, towards cheaper static storage and client side processes, perhaps using the tech used by other major websites as an example, but certainly less reliance on so much hard-to-setup, costly to run server architecture. However my view is based mainly on the view of MW as a general software for website, less what wikipedia itself is needing,
22:30:54 <Dysklyver> although I would imagine that cutting costs by cutting complexity would be a favorable outcome, especially when scaling.
22:33:48 <Dysklyver> google has done a significant amount of work towards that aim, I believe it is feasible to cover all the main customization of MW within a lightweight framework and what is effectively a web-app shell
22:32:12 <DanielK_WMDE> Dysklyver: the question is how you can suppoort the gazillion complex use cases that Wikimedia needs with such an architecture.
22:32:35 <coreyfloyd> DanielK_WMDE: I think that begs the question “do we want to reduce our use cases?"
22:33:38 <coreyfloyd> I think that is something we can push back on. We can tell product that we need to scale back the features of the software in order to move forward
22:30:57 <TimStarling> if it is revolutionary in nature, e.g. rewrite everything in typescript, then it helps to re-evaluate after you do prototypes
22:31:16 <TimStarling> if the idea is to keep doing what we're doing, then we already have the prototype
22:32:41 <cscott> TimStarling: i could get behind a 1-year plan of 5 or so prototypes
22:32:55 <cscott> each focusd on a different "key need" (or approach)
22:33:38 <cscott> TimStarling: the danger being, of course, that we product-ize the prototypes w/o deprecating the things they were supposed to replace
22:33:42 <brion> but that does give the opportunity to think about commonalities in those cases (for instance, do things need additional types of data storage -> leading to MCR etc)
22:33:52 <TimStarling> we've done prototypes of node.js backend services and I'm not very impressed by them
22:34:15 <TimStarling> so if the plan is "let's do more things like RESTBase" then I would have concerns about that
22:34:39 <brion> i'd love for mediawiki-as-a-big-php-app to reduce itself into a core-with-an-api and do the ui separately, but doing it well is a tricky thing
22:34:55 <brion> and i love the idea of storage becoming pluggable, but that's ........... hard
22:35:25 <coreyfloyd> brion: I think thats the feeling of many in audiences - to have a core and an API, then let clients build UIs
22:34:43 <DanielK_WMDE> tgr: do you think it is useful/sensible to try and develop a "top dow" tech vision for the platform before we know what products we want to build?
22:34:55 <DanielK_WMDE> tgr: if yes, on what parameters would that be based?
22:34:58 <tgr> DanielK_WMDE: so here's an example (cscott's, I think): the whole Wikimedia universe should be a single system internally. That's well beyond 5 years, doesn't depend that heavily on product plans, it can be broken down into meaningful short-term steps which won't be a complete waste of time even if the goal is abandoned after 2 years
22:35:20 <cscott> it's cross-cutting because it affects how we do cross-wiki and cross-language collaboration
22:37:05 <DanielK_WMDE> cscott: ah, ok, this is about dissolving the boundaries between sites/projects/communities, not so much about system architecture.
22:37:54 <DanielK_WMDE> cscott: but that only makes sense if we anticipate that on the product level, we want/need that integration.
22:38:19 <cscott> DanielK_WMDE: maybe. or maybe it makes sense in terms of simplifying our db config & etc purely from the tech/ops side as well.
22:39:01 <cscott> coreyfloyd, tgr https://en.wikipedia.org/wiki/User:Cscott/2030_Vision
22:36:28 <apergos> we could talk about the eternal parser issue (mw is the one true parser, should we put more resources again into changing that, or give up for good?)...
22:37:13 <cscott> mw isn't really the one true parser, the one true parser is a platonic ideal no one has yet implemented. mw has known bugs, which are different from parsoid's known bugs, and the One True Parser is in between them.
22:39:02 <apergos> cscott: mw has some edge cases that are exploited; that's why it gets to keep its status as authoritative, as far as I understand it
22:38:01 <Dysklyver> A simple way of looking at it is that all MW has to do is let people read and write files, and browse what looks like a website, everything else is UI customization and access control.
22:38:49 <DanielK_WMDE> Dysklyver: you are missing THE key feature that makesy wikipedia work: crowd-sourced curation and quality assurance
22:41:56 <Dysklyver> from a technical view its just read/write on files, and logging, viewing of such. it's grown very complex, thats true, but fundamentally there are not that many needed functions to the software, it would be an idea to create some kind of roadmap as to what function are really needed before deciding exactly how to make the framework more lightweight.
22:42:41 <DanielK_WMDE> Dysklyver: yes, that'S a pretty good point: we not only need a product strategy for new stuff.
22:42:48 <DanielK_WMDE> we need a product strategy for getting rid of old stuff
22:38:22 <tgr> DanielK_WMDE: I think the summit is not a good place for coming up with one true vision (not inclusive + not enough time) but maybe as a place to figure out which vision proposals are interesting, and start some structured conversation about them?
22:40:39 <tgr> DanielK_WMDE: I agree it's not going to be completely independent from product strategy. I worry that strategies put together by product people will have blind spots (just like those put together by tech people, of course) and having both kinds could be complimentary
22:40:51 <DanielK_WMDE> tgr: yes, and asking pointed questions to product people. "so, *do* we need to support LAMP?", "do we need to see edits on wikidata on the local watchlist?" "Is supporting multi-lingual content for anonymous users important?"
22:41:48 <DanielK_WMDE> tgr: yes, absolutely. we need iterations between product and tech. the more the better. and the summit should at least ask for, if not establish, a process for this. should surface needs for sustaining, and questions for exploration/development. shoudl offer possibilities and opportunitites
22:42:24 <coreyfloyd> tgr: yeah… they need help to understand the gaps in product planning. They don’t know about the constraints of the stack and what affects are
22:43:09 <coreyfloyd> DanielK_WMDE: tgr yes, the only way forward will be iteration with audiences
22:45:29 <DanielK_WMDE> cscott: i think it is also useful to be able to say "hey, this is really expensive, maybe doing 90% of that for 10% the cost would be enough"? I think we should ask that question. Not make the decision.
22:45:12 <tgr> DanielK_WMDE: having the "cost" side for some kind of cost/benefit evaluation would be very useful, although for the most important issues it's not achievable in a week IMO
22:46:07 <DanielK_WMDE> tgr: yea, all we can do is to ask the right questions, and try to establish procecsses.
22:48:09 <DanielK_WMDE> coreyfloyd: sure, that's how it should be: requirements/wishes from the one side, cost/constraints from the other. iterate, agree, rinse, repeat.
22:48:39 <DanielK_WMDE> coreyfloyd: i think we don't do that enough on the large scale. I hope we can start doing that for the strategy process.
22:49:10 <coreyfloyd> DanielK_WMDE: yeah I think this is a large gap we are identifying… engineers who operate close to product seem to have this feedback loop, but it is non-existent at a high level
22:48:39 <apergos> suppose in 10 years most of our editors come from phones/tablets, or most of our edits (number, not necessarily amount of content), what would that mean? is that likely? do we want to support that? what would we have to deprioritize?
22:49:07 <brion> i think it's 100% for sure that most of our edits will come from phones in 10 years
22:49:48 <cscott> brion: google docs doesn't work well on phones. i think folks still use big machines & big screens when they want to actually tackle a big piece of editing.
22:50:20 <DanielK_WMDE> brion: ...but not by typing.
22:52:13 <cscott> it's just as likely we'll have this mobile vision where your phone is actually your primary processor, and it takes over whatever TV screen and/or keyboard device you happen to be near
22:52:25 <cscott> so there's not a sharp division between 'phone editors' and 'desktop editors'
22:52:27 <_joe_> I think we should not try to forecast tech trends ourselves, that's pretty much not what tech should think about. Flexibility, on the other hand, is something we should be concerned about
22:49:30 <tgr> cscott: for example, one possible tech vision is a distributed-ownership microservice architecture where people can take over the fringe things that they love, and the "official" Wikimedia site interfaces them and recovers gracefully if they do a poor job and the service is not keeping up
22:50:11 <tgr> toolforge is sort if edging towards that directions, but it could be taken much farther
22:50:36 <cscott> tgr: in my ideal world we'd have gradual on-ramps and off-ramps toward and away from "WMF maintanence of <feature X>".
22:50:50 <cscott> tgr: with intermediate steps between "we take care of everything" and "we turn the service off"
22:54:49 <tgr> cscott: "I can write an experimental feature and release it to beta testers without risking site security and reliability" is a nice long-term tech vision too
22:54:10 <coreyfloyd> DanielK_WMDE _joe_ So covering mobile plus flexibility: is it good to focus on APIs? Maybe focus on abstracting the front of MW from the core?
22:54:45 <cscott> making core smaller is the way i think about it
22:54:46 <coreyfloyd> _joe_: also that forces us to figure out “what is core to MW"
22:55:07 <bd808> coreyfloyd: bring this team back to life and staff it -- https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_API_Team
22:55:24 <bd808> "Accelerate user interface innovation/evolution and increase efficiency of editor automation by making all business logic for MediaWiki available via remotely callable APIs."
22:55:49 <coreyfloyd> DanielK_WMDE: yeah, and if we go in with a goal of just separating the front and back end with APIs and supporting the same functionality that would be a huge step forward… but yes the layers need designed
22:59:37 <cscott> tgr: anyone who wants to make a *fully-functional* new UX for mediawiki needs to grapple with Special:
22:59:53 <brion> yeah special: is the special hell of mediawiki :D
23:00:01 <brion> basically everything in there needs to be abstracted sanely
23:00:19 <DanielK_WMDE> yea, refactoring special pages to share logic with APIs isn't *that* hard. it's quite doable, but it needs time.

Event Timeline

daniel created this paste.Jan 19 2018, 9:37 PM
daniel mentioned this in T183313: Wikimedia Developer Summit 2018 Topic: Evolving the MediaWiki Architecture.
Quiddity subscribed.Jan 19 2018, 9:52 PM
• Jan_Dittrich subscribed.EditedJan 20 2018, 1:25 PM

strategic planning is, IMO, deciding where we want to arrive first and on the route to get there only after that

+1
Goal and the path to it will influence each other. But better a slightly changing clear goal than an unchanging vague one. This way of thinking might also help to deal with the product/dev-strategy divide: A clear code/tech vision will allow product to either jump on it and/or voice their concerns of it more clearly.

TomT0m subscribed.Jan 20 2018, 8:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits