Page MenuHomePhabricator

Theme: Deploying and Hosting
Closed, ResolvedPublic

Description

Theme program committee

@gregGreg GrossmeierWMF Release Engineering Team Manager. Program Committee Chair
@WMDE-leszekLeszek ManickiWMDE Engineering Manager

Theme proposal

This track will focus on delivering new ideas and improvements (features and bug fixes, code and configuration changes) to production in a simple and confident way. We will touch upon, but not limit ourselves to, topics of continuous delivery and deployment, staging and testing environments, talk about Docker and Kubernetes and many other themes. We will also cover areas related to installing and updating MediaWiki and related software.

Session proposals

See sub-tasks

Related Objects

Event Timeline

The session proposals will be added to this task tomorrow and an email will go out to techconf attendees asking for feedback on theme and session proposals before they are finalized.

Anything missing (topics, problems, workshops, reports, case studies, etc) from the area of Deployment and Hosting that you'd like to see at the Technical Conference 2019? Please comment here, reach out to me or @greg!

so.. some of this theme seems rather WMF focused.. Have we considered taking lessons and input from 3rd party MediaWiki hosters within this theme ? Maybe even have a separate session to learn about what they are doing ? Or at least validating the WMF ideas against their usecases ?

I don't know if this goes here, but I'd like to address one issue I thing we have to solve. Like twenty years ago I had a 3 1/4 diskette with a program to see how the planets where in the sky live, or what was the position of the moons of Jupiter right now. I could test with different times, places in the world (etc). Like ten years ago this could be a cool new feature for Wikimedia, something we could add to articles. Now is something we should have added ten years ago. But there is not a simple way to do it and every time I have tried to ask about that I find partial answers, mostly about security problems with applets.

The question here is that this is something we need, is something it was developped for knowledge centered webs like two decades ago but we still can't deploy. Is there any way we could discuss what would the best deploying system for something like this?

every time I have tried to ask about that I find partial answers, mostly about security problems with applets

That sounds like a very specific use case. Is there a dedicated task about this with more specific information? That task should also cover if "applet" implies "Java applet" and describe the wanted functionality in a way which allows others to potentially propose safer implementations, plus include links to previous discussions.

That sounds like a very specific use case. Is there a dedicated task about this with more specific information? That task should also cover if "applet" implies "Java applet" and describe the wanted functionality in a way which allows others to potentially propose safer implementations, plus include links to previous discussions.

This kind of answers are what I get normally, and I think that this is the real discussion I would like to have: it is very difficult for someone who doesn't know the exact path for something to happen, to express everything as if the path was clear.

The example was to illustrate an idea. It is not only about astronomy, it is about interactivity, text to speech, text annotations, or about some other ideas communities may have but can't deploy because processes are unknown. I don't know what "applets" should include, or what shouldn't.

@Theklan: Please describe in a separate task which specific problem you would like to see solved. Currently we do not really know. Thanks a lot!

I don't think that we can cover "How to choose the best suited technical implementation (MediaWiki extensions, gadgets, "applets", etc) to provide functionality XYZ" in this task / session - That is a decision which has to be made before discussing potential deployment of some existing code base on Wikimedia servers.
So there might be a different understanding of "deploying" here. :)

I'm not talking about "How to choose the best suited technical implementation (MediaWiki extensions, gadgets, "applets", etc) to provide functionality XYZ", but about "What is the path for things to be done". Imagine some kyrgyz university professor who wants to insert some interactive applets done in python at kyrgyz Wikipedia. There's not a Kyrgyz user group or even someone speaking the language at the WMF (as far as I know). What would the path be for this person?

Maybe this is completely out of the Conference topic, but if not here, then where?

Are 3rd party hosting issues in scope for this track? For example, I'd be interested in a discussion around the MediaWiki upgrade process (core, but probably mainly extensions), from the point of view of developers and of sysadmins. Mostly, we assume that people will either track a branch of a Git repo (trunk or a release branch) or download manually from ExtensionDistributor — which works great, but we also have version numbers (in Special:Version via extension.json, sometimes Git tags, and extensions' pages on MediaWiki.org) that look like they'd be useful info when upgrading, but actually don't really come into it.

Some questions that might be answered in such a discussion:

  • How can upgrading, and knowing what to upgrade, be made easier for sysadmins?
  • What purposes do version numbers of extensions serve?
  • If extension.json specifies a given MediaWiki version minimum requirement, can CI run tests on that and all higher versions (for every patch, rather than merging to master and then having subsequent tests run when backporting)?
  • What would it take to add 'please upgrade this extension' messages to Special:Version?
  • When releasing an extension, what are developers options? How do these differ norms in other places (i.e. how to we best explain things to new devs here)?
  • What can be added to Manual:Developing extensions#Publishing to help developers learn what to do?

@Theklan: That indeed sounds unrelated to deploying code and unrelated to the topic of this task.

Am I missing something or is Wikimedia Cloud Services really not at all represented in the conference track that's ostensibly about hosting and in a conference that's ostensibly about developer productivity? This seems like a step back in the trend of finally recognizing tool developers as a key element of supporting power users, community managers, partnership work and innovation on our wikis, who make heroic efforts despite having to put up with seriously sub-par developer productivity tooling (including deployment, hosting, logging/monitoring). There is a (draft) movement strategy recommendation largely focused on the topic, too.

Logging in general (including in production) is also something I'm surprised not to see. Admittedly I would not be able to contribute much to such a session so this is just kibitzing but good logging feels like an important element of developer productivity and something we could do better at.

@Theklan that seems more in the scope of the themes T234087: Theme: Local Development and Onboarding and T234091: Theme: People and Processes. Deploying and hosting is about anything after the code (including an integration) has already been written from my point of view.

Thanks @Tgr for the input! Shame we were only able to give so little time for the comments. I am not sure how much adjustment we'll still managed to squeeze to the sessions.

Am I missing something or is Wikimedia Cloud Services really not at all represented in the conference track that's ostensibly about hosting and in a conference that's ostensibly about developer productivity? This seems like a step back in the trend of finally recognizing tool developers as a key element of supporting power users, community managers, partnership work and innovation on our wikis, who make heroic efforts despite having to put up with seriously sub-par developer productivity tooling (including deployment, hosting, logging/monitoring). There is a (draft) movement strategy recommendation largely focused on the topic, too.

It is indeed a bit unfortunate that (not on-wiki) tooling received a little attention. If you were to name a problem area it would have been the most important and beneficial to cover at the Technical Conference, what would that be? Should be try digging what are the problems people are currently facing when it comes to tooling (you've mentioned "deployment, hosting, logging/monitoring")? Is any of aspects of the tooling particularly essential here?
The movement strategy recommendation is an interesting read. Would you see Tech Conf as a venue to seek for possible input, endorsement, etc from the audience (might be me shooting in the dark, but in case that was the reasonable use of conference, it could possibly also be linked to "Standardization Decisions" and "People and processes" themes)?

Logging in general (including in production) is also something I'm surprised not to see. (...) good logging feels like an important element of developer productivity and something we could do better at.

Good point indeed. I agree that logging contribute hugely to the developer productivity. I admit I failed myself to pinpoint a specific problem/topic related to logging that could be covered at the Tech Conf. At WMDE we have been recently revisiting the topic of better logging of client-side errors (in front-end code). I then discovered T217142 which, while still being marked as WIP, seemed somewhat on-track, hence I figured it might be worth to prioritize other topics. Lame excuse, I admit it. Do you think it is still worth bring this topic, to make it more specific, ideas presented in T217142 to the attention of Tech Conf participants?
Are there any other logging-related topics that come to your mind?

Thanks @TheDJ. I'm a bit lost about the topics here, but I think that this issue may be discussed at the Technical Conference.

It is indeed a bit unfortunate that (not on-wiki) tooling received a little attention. If you were to name a problem area it would have been the most important and beneficial to cover at the Technical Conference, what would that be? Should be try digging what are the problems people are currently facing when it comes to tooling (you've mentioned "deployment, hosting, logging/monitoring")? Is any of aspects of the tooling particularly essential here?

I suppose these are resourcing problems more than planning problems. Is TechConf useful for addressing those? IIRC the original promise was that it would be (as opposed to the DevSummit, this would be a conference with decisionmakers present), not sure how it played out in practice though.

The movement strategy recommendation is an interesting read. Would you see Tech Conf as a venue to seek for possible input, endorsement, etc from the audience (might be me shooting in the dark, but in case that was the reasonable use of conference, it could possibly also be linked to "Standardization Decisions" and "People and processes" themes)?

The original plan was to hand in the final recommendations (gained by merging the current nine thematic groups of recommendations into a unified single set) to various organizations for approval by Nov 1 (just in time for TechConf to consider them). Unfortunately that didn't work out and now they are going to be several months late. Asking endorsements for drafts would probably be a bad idea. Input would definitely be welcome. Last TechConf, TechCom used some nice approaches to gathering feedback on the engineering principles, maybe something like that could be tried...

I admit I failed myself to pinpoint a specific problem/topic related to logging that could be covered at the Tech Conf.

I think our server-side logging is fragile and there are a number of minor but quite annoying issues with both the logging code, Logstash and Kibana. The recent ops-l email is a nice illustration, but also Kibana's disallowing of identical event field names with different types, the whole SPI thing in MediaWiki where everyone just uses Monolog anyway but due to the flexibility of the config layer it's quite hard to set up (and the vagrant version is slightly broken too)... I'm not very active in logging code these days, but in the past I collected particularly annoying issues to T157850: Interacting with Wikimedia logs should be a pleasant experience.

Admittedly this is not very specific, I'm not sure if the root cause is lack of resourcing/ownership or maybe just ELK is not a very good fit for us / we outgrew it. What I wanted to say is that I think logging is currently a bit of a barrier to productivity (not a huge one, like e.g. code review, but it's there), both for experienced developers debugging production and for new developers setting up their local environments.

At WMDE we have been recently revisiting the topic of better logging of client-side errors (in front-end code). I then discovered T217142 which, while still being marked as WIP, seemed somewhat on-track, hence I figured it might be worth to prioritize other topics.

For client-side error logging, we have a plan and at least some resources. Client side logging in general is largely nonexistent though.

Notes from Etherpad:

Deploying and Hosting Track

Lead: Leszek M

Scribe: Brennen

Format

Summary of outcomes from LM to each room, followup questions from LM and general discussion.  Discussion is sorted under the rough session headings where it was practical, otherwise split out by room.

Summary of outcomes presented at the each station:

Deployment Pipeline

  • Collected requirements and needs from the audience, things they'd require to move their work to the Pipeline (Must Haves and Nice to Haves)
  • "Must Haves" Prioritized in the group discussion:
    • Integration/end-to-end testing
    • Ability to test multiple repos together
    • Ability to test and deploy MediaWiki and extensions using the Pipeline
  • Identify which parts of the Add a wiki process are related to the deployment pipeline

Testing and Staging environments

  • Collected use cases and requirements from the audience, grouped as Musts and Needs
  • There will be the Test Environments WG formed in upcoming weeks to continue with the topic
  • Identified the need for collaboration with Local Dev Environments work

Self-service Stateless Microservices

  • Collected input on the idea, clarified some potential use cases (whether there is an interest for a proposed solution to be used to serve those use cases)
  • Giuseppe to go back to the drawing board and define the idea more specifically for the next iteration of feedback/review

Termbox

  • WMDE and WMF SRE/RelEng to work on creating a checklist for deploying a service to WMF production (using Deployment Pipeline)

Release strategies using containers

  • Adam will collect answers to questions (What are the most basic use cases and requirements to be considered; How should configuration work?; How should extensions and skins be included?; What other services should be included?) and thoughts prioritized by participants, and document those on the page on mediawiki.org
  • WMDE will be working on moving their current Wikibase docker image to the Deployment Pipeline

Overall Summary of Points with Related Questions

  • - Continuous Delivery / Deployment in Wikimedia
      • *Include Wikimedia extensions*
      • *What parts of creating a new wiki process could be improved by...*
      • *Future of the Deployment Pipeline*
      • *Present throughout conference*
      • *MediaWiki core and extensions should be part of the pipeline*
        • QUESTION: Florian: Is this a Wikimedia-specific use case?
          • Addshore: I think lots of things that were discussed in my session and the release strategies one - might take WMF a few years to get to the point where there's an updated used WMF-flavored Docker image; WMDE / we will be trying to consolidate the images, using the build pipeline to create them, trying to get to the point where it's as close to the WMF as possible.
    • Quo Vadis Beta Cluster?
      • NK: I think could have used some more background on beta cluster, and/or some overview on what are existing testing environments.
        • DA: Better pre-reading?
        • NK: I don't know who read the pre-reading...
    • Self-service stateless microservices for APIs
      • Giuseppe
      • *Some clarification on requirements*
      • *Some use cases dismissed*
      • *Author will go back to drawing board and try to clarify concept*
    • Release strategies for MediaWiki and other elements of Wikimedia platform (containerize it)
      • *What have we learned when deploying a standaline server side rendering service for the new mobile wikidata termbox*
      • *Adam Shoreland*
        • *How configuration should be incorporated*
        • *To document*
        • *WMDE working on moving Wikibase docker images to deployment pipeline*
      • Timo: I couldn't attend the release strategies session - might have been in scope how we might release MW in a production-ready format in a container.
        • Do we support shared hosting?
        • Do we support not-shared hosting?
        • Dockerhub image - who?
          • BD: David Barratt (sp?)
      • Steven: Appreciated  hearing about the Termbox solution
    • Testing
      • test environments work group will be formed after conference
      • collaborate with local development group
      • Elena: testing environment testing - interesting to hear expectations for testing environment - disagree with most of it, but interesting to hear
        • Production data etc.
        • LM: Need different kinds of testing environments - what is required for what
        • Elena: One environment that does everything and...
        • Joe: ...you end up with beta cluster.
        • LM: We are not going to create beta cluster 2.0 - idea was just to figure out what kind of needs people have.
    • Unconference session on CI solutions
      • Well attended
      • Some heated conversation
      • Addshore: MW vs. WMF - good ideas about having base image, saying what extra things you want to be added to it
      • Riccardo: The CI discussion could have been part of the main track.  Was important.

 Social Room General

  • QUESTION: Amir: Creation of new wikis
    • Before this conference I thought it had to do with deployment pipeline, apparently not...
    • Some followup tasks out of this
    • Nice to see that people understand the problem and care about it, now it's left to get some resources and address it
    • Creation of new wikis:  Sucks.
    • Florian: Should just be a configuration step which doesn't need any deployment at all.
      • Amir: "just"
      • Addshore: Easily solvable
      • Amir: "easily"
    • Tyler: These questions got tangled up because deployment pipeline will change the things that the addwiki (?) script has to touch

Room 4 General

  • What were your takeaways?
    • BD: A lot of different use cases that are very different - there's some core overlap, but there are a lot of things that just spread out very widely, different org structs, etc. - that scares me a little bit about people who are trying to talk about the one solution.
    • J: Different use cases...  Some more nuance than in the past.
    • Riccardo: In particular continous delivery / beta cluster / unconf on CI - all interconnected - we're discussing them mostly separate, but having in the back of our head that they're interconnected - I don't know how to solve this problem.
    • BD: In terms of this track, everybody came into every room wanting to talk about greenfield stuff instead of fixing what we have...
      • Good for discussion, not very good for next steps.
      • Do you know you're talking about throwing away 50000 hours worth of work?
      • Timo: I agree exactly with what Bryan just said.  Frontend sanitization...  A very key part of it was we can all imagine interesting front end ??? - what was missing?  Not focused on the struggle so much as the paradise.
        • DA: To push back, I think that was covered on the pre-reading - I do think there's a lot of nuance and consideration, but you have to talk about the new thing.
        • LM: Need to talk about what you want
        • Timo: Not so much that we want to keep what we have - still very important to acknowledge that the core expertise of this conference is in what our needs are and what is unique to us.

Magnolia General

  • What were your takeaways?
    • Piotr:  I knew there are a couple of Docker images; I didn't know that there were so many
      • Joe: Main takeaway:  We should create a (more) official base image
      • Brennen: Docker SIG could be a thing...
        • [much as I'm not sure this will actually work, but we do need to coordinate on these questions]
  •  LM: Anything missing in the program?
    • Steven: I wish there was a hands-on session where it's like... What do we need to do to get you up to speed?
      • [Some discussion about this.]
      • Z: Are you looking for a local dev environment?
        • Steven: I just want to know what the options are in a concrete way.
        • [Sounds like there's some appetite for more concrete tutorial / hands on / learning sessions] 
    •  Sam: One thing I missed was lower-end of MW - plans to phase out low-end shared hosting type stuff...  That's gonna have to happen?
      • Certain things will become possible if we phase that out...
      • LM: There's no clarity on that.
  • - Steven: A lot of talk about docs - plz improve - etc. - but there was a lot of "we're moving to containers" but not a lot of education about that, it would be nice to communicate that to people.
    • AK: We agree that we won't have beta 2.0 but we still want something that does everything for us...

Oak General

  • What were your takeaways?
  • LM: Anything missing in the program?
    • [silence]
    • Toby: Incremental releases.  Allowing our users to select some features rather than others.  Customization for non-logged-in-users.
    • Deploy and hosting outside of WMF
      • What does versioning and release schedule mean for 3rd parties?
      • Toby: Or Wikibase cloud
        • Mention of development env track
      • QUESTION: While providing containers / images - will be provide an environment that says PHP this and that / MySQL this and that - or will we say Docker is the requirement?
        • LM: Came up in other rooms how containers play with shared hosting environments.
        • MS: I don't understand what problem you're trying to solve.
        • : Challenge is that this limits innovation.  If the requirement was just containerization, you lose compatibility with webhosts from 1995.
  • LM: Any final thoughts?
    • TPT: something between labs and hosting - [I think this is in reference to Giuseppe's session on stateless microservices?]
    • Enthusiasm for Adam's container work

Azalea General

  • What were your takeaways?
    • JF: I'm not sure about the readout about creating new wikis - I consider that to be basically pretty much solved at this point.
      • Once we've got Parsoid PHP done, 80% of the complexity goes away.
      • The only thing not configured automatically is DNS
      • Once the complexity is a problem, it goes away once we switch to containers.
      • Appreciate people are frustrated right now
  • LM: Anything missing in the program?
    • Tgr: I was hoping for some discussion of the shared hosting situation.  We seem determined to move to isomorphic rendering - JS run on the server side - fairly large part of the MW user base is PHP only.
      • Moriel: We're not moving forward without having a solution to that.
      • Tgr: Was surprised we didn't discuss, that's all.
      • LM: On related note - while we talk a lot about containers, "non-shared hosting" - while we talked about that, there was no clear statement about shared hosting, etc.  Are containers a replacement for the LAMP stack?
    • JF: A lot of our conversations were very long term - we talked about the deployment pipeline, but that's not happening for a year for MW at least
      • Containerization for release cycles etc
      • We didn't say "this is a problem right now, what's 3-6 solution"
      • Maybe that's because all our problems are fine...
      • Other people may not feel how I feel [i.e. worse] about deployments right now
      • I'm kind of surprised that all of our discussion is so long range
      • Moriel: That's funny - I didn't understand that it was so long range
        • [some back and forth in the room]
      • JF: Maybe everything's fine...
      • Moriel: ...I actually got the sense that everything is not fine.  I don't htink anyone expects us to be done next week, but the feeling I got was... This is what we're moving towards.
      • JF: I think these things we're working on are important, just...
      • LM: Balance between long-term and more immediate things not very satisfactory?  Just very little immediate outcome?
      • JF: Maybe
      • LM: I think intentional to focus on long term, not by accident, but I take of that
  •  LM: Any final thoughts?
  • LM: Mention of heated CI session
    • Again, long term
    • Some strategy goal things considered