@Fjalapeno What do you think about this. Has it been considered somewhere else?

Why do you need them?
You only really need parser styles here (ie. styles that are added by templates e.g. JS for Math).

Most of the styles here will be for the skin and not useful to your app.
Happy to talk about this in a hangout.

That is a good question indeed.

There are several type of styles to consider:

• site.styles e.g. MediaWiki:Common.css
• styles provided by extensions/core e.g. ext.cite.style or mediawiki.page.gallery.style - these are loaded only if the page is needed, but in Marvin you may want to load them all so they are available on subsequent page loads.
• TemplateStyles (which should be bundled in the content HTML)
• skin chrome styles (probably not applicable here)
  • mediawiki.skinning.interface is a default skin stylesheet that can you ignore or keep to make your life easier
  • mediawiki.skinning.content.parsoid to support figure element in Parsoid output. Minerva actually supports Parsoid styles to avoid needing this)

Trade offs:

• Using these saves you time
• Not using them helps you control the experience more at the cost of having to write out the styles yourself.

Thanks for the summary @Jdlrobson.

It seems like it would be interesting to have returned from the API:

• Names for modules:
  • styles provided by extensions/core e.g. ext.cite.style or mediawiki.page.gallery.style
  • content styles (mediawiki.skinning.interface & mediawiki.skinning.content.parsoid)
• Embedded styles (template styles)

With just the names of the modules, the client would be in charge of figuring out what it has pre-bundled, what is missing, and what it is not interested in (maybe some clients want skinning.interface, maybe they don't), and request it from the load.php url as needed.

@Jhernandez this makes sense… we have started thinking about CSS, but we haven't dug deep into how we should make this happen yet.

API consumers need to know which styles are required for a page, and where to fetch them from, in order to properly style the content of the page if they choose to apply them.

In mobile web styles are gathered by mediawiki & ResourceLoader, and output into the page. This approach wouldn't work with mobile web as an API driven skin, consuming parsoid content.

Usually CSS is just inlined within the HTML or style sheets are referenced within the HTML. I was assuming this is what we would do, but maybe that won't work.

Can you clarify why this doesn't work with an API driven web app? (I assumed this would work with the native apps just fine, so maybe this will be a problem with them, too)

In T173821#3579363, @Jhernandez wrote:

Thanks for the summary @Jdlrobson.

It seems like it would be interesting to have returned from the API:

• Names for modules:
  • styles provided by extensions/core e.g. ext.cite.style or mediawiki.page.gallery.style
  • content styles (mediawiki.skinning.interface & mediawiki.skinning.content.parsoid)
• Embedded styles (template styles)

With just the names of the modules, the client would be in charge of figuring out what it has pre-bundled, what is missing, and what it is not interested in (maybe some clients want skinning.interface, maybe they don't), and request it from the load.php url as needed.

This seems to one one reason to put it into an API rather than as HTML tags. But do we really want to burden clients with making complex CSS decisions based on modules? Or is this already happening? ¯\_(ツ)_/¯

If it is happening and we need to do it, is it something we should put behind the PCS (as you asked above) so it is centralized rather than leaving to the clients?

Adding reference to this work from @Jdlrobson:

https://www.mediawiki.org/wiki/Reading/Web/Projects/Performance/Inlining_CSS

In T173821#3580757, @Fjalapeno wrote:

Can you clarify why this doesn't work with an API driven web app? (I assumed this would work with the native apps just fine, so maybe this will be a problem with them, too)

Let me try to explain my point.

In production, style URLs are one URL bundling together a bunch of different modules, which as you said gets loaded as one link tag in the HTML. These URLs are like:

• //en.wikipedia.org/w/load.php?modules=mediawiki.legacy.commonPrint,shared|mediawiki.skinning.content.parsoid|mediawiki.skinning.interface|skins.vector.styles|site.styles|ext.cite.style|mediawiki.page.gallery.styles&only=styles&skin=vector

The apps, bundle a certain number of stylesheets at build time. Let's imagine, the styles the apps bundle are:

• mediawiki.legacy.commonPrint,shared|mediawiki.skinning.content.parsoid|mediawiki.skinning.interface|skins.vector.styles|site.styles|ext.cite.style

Because they are very common. When the user visits an article X that needs extra styles, there are then 2 options. Ignore the need for those styles and show the article with the styles you've got (the current apps strategy), or load the needed styles to show the article as it was intended (the mobile web approach). I'm not sure what we'll need on marvin.

When choosing to load the needed styles, imagine that X has a gallery, so there are extra styles needed: mediawiki.page.gallery.styles. If the client includes the full styles URL as it comes in the HTML, then it will be requesting all of the CSS, when it already has most of it from the bundled ones on the app, or previously visited pages (mediawiki.legacy.commonPrint,shared|mediawiki.skinning.content.parsoid|mediawiki.skinning.interface|skins.vector.styles|site.styles|ext.cite.style should be available instantly).

So ideally, the client app/skin should be able to check the styles that the content needs, see if they are available already, and choose to fetch the ones it doesn't have.

The problem with having the tag in the HTML, is that as soon as you put it in the browser/webview, the browser will go fetch it all, because it is a single unique URL.

Having individual URLs per module in the HTML is not an option for production because of the amount of HTTP requests would be terrible for performance.

So as I see it, a good option could be to identify the two sets of content specific styles (styles provided by extensions like ext.cite.style or mediawiki.page.gallery.style and embedded styles (template styles)) out of parsoid output, and provide it as meta data for the clients to act on.

This seems to one one reason to put it into an API rather than as HTML tags. But do we really want to burden clients with making complex CSS decisions based on modules? Or is this already happening? ¯\_(ツ)_/¯

AFAIK, the apps avoid doing any decisions by preloading CSS and accepting that some articles that need special styles won't be seen as they should (let me know if I'm wrong please).

Do we want to keep that behavior? If we do, then we don't want to burden clients with making style decisions based on modules.

I suspect this is going to be a requirement for mobile web marvin given current mobile web acts by including all necessary styles, so I'm asking all of this to see how we will do it. Making the styles meta data, would mean we can do the most performant thing, but pay a cost of complexity. Just exposing the link tag with the styles URL on the HTML somewhere would work too, being a simpler solution, but a bit more expensive with regards to performance (bigger CSS transfer size on every article).

So, pretty much depending on what the apps are going to do with this in the future, we can work on designing a good solution. As an interim solution, we could just expose the link[rel=stylesheet] tags in PCS and accept the tradeoffs.

If it is happening and we need to do it, is it something we should put behind the PCS (as you asked above) so it is centralized rather than leaving to the clients?

I believe it should, but I may be wrong. You are better aware of its scope and if this data fits on it, so I think you are better equipped to say where it should go.

Hopefully I've explained myself half-well, let me know if it is still unclear.

In T173821#3587859, @Jhernandez wrote:

AFAIK, the apps avoid doing any decisions by preloading CSS and accepting that some articles that need special styles won't be seen as they should (let me know if I'm wrong please).

This is correct.

Do we want to keep that behavior? If we do, then we don't want to burden clients with making style decisions based on modules.

This has been the strategy since the earliest days of the native apps. I believe that in the early days the idea was that the app could update its bundled styles periodically as directed by a config variable set in the MobileApp extension, but we've moved away from this idea.

I would welcome some more sophisticated approach than what we have now but agree that ultimately it'll probably still involve a tradeoff between completeness and page load performance.

Because they are very common. When the user visits an article X that needs extra styles, there are then 2 options. Ignore the need for those styles and show the article with the styles you've got (the current apps strategy), or load the needed styles to show the article as it was intended (the mobile web approach). I'm not sure what we'll need on marvin.

Ahhhhhhh… makes sense. Thanks for clarifying the different strategies

AFAIK, the apps avoid doing any decisions by preloading CSS and accepting that some articles that need special styles won't be seen as they should (let me know if I'm wrong please).

Do we want to keep that behavior? If we do, then we don't want to burden clients with making style decisions based on modules.

I would say that the current app strategy is becoming untenable… as app adoption grows, the same constraints of mobile web start to apply. So I would like to figure out a strategy that works for all the platforms. Your strategy based on the modules seems like a good idea.

If the client includes the full styles URL as it comes in the HTML, then it will be requesting all of the CSS,

Ok, I think I am starting to follow… So this seems like a key design flaw in how we serve CSS. Instead of separate URLs for each style sheet which could be cached, we request all the styles in a query URL which causes the behavior you are describing… fetching all styles all the time.

Having individual URLs per module in the HTML is not an option for production because of the amount of HTTP requests would be terrible for performance.

Ok this is good to know and will help guide the solution here. It seems like we are trying to strike a balance, yes? Get all the common styles in one sheet and then load the "special" styles individually?

So ideally, the client app/skin should be able to check the styles that the content needs, see if they are available already, and choose to fetch the ones it doesn't have.

So I'm thinking of how much we can offload to the PCS here… going off what you are saying, here is my first idea for the "common" styles:

1. We manually decide what the "common" styles are (we can start with what is bundled in the apps and modify that based on your current work)
2. We set up logic in PCS to get these basic styles from the MediaWiki API and cache them
3. We will expose the sheet as a single unchanging URL that can be easily cached by clients
4. We add this URL to the page HTML (this should be ok since the URL is unchanging and can be cached)
5. Apps can still bundle them, but would prefer whatever is at the URL as it may be newer.

For the "lesser" used styles, I'm also thinking if we can do this without burdening the clients… but I'm less sure, would this work?

1. When a page is first fetched in the PCS, we get all modules for that page
2. We request each sheet individually from the MW API, and cache each individually (If a style sheet was cached previously, we would not get it again)
3. We expose each sheet as a single unchanging URL so that it can be easily cached by clients
4. We add these URLs to the page HTML to reference these other sheets (this should be ok since the URL is unchanging and can be cached)

Basically, I'm trying to solve this doing what you are saying, except instead of doing it on each client, we do it in the service itself.

I don't see a way around sometimes having multiple URLs, but if we make the URLs unchanging and deterministic we should get the caching benefits (I think)

Does this sound viable to you or am I way off base here?

Thanks again for all the background… this makes a lot of sense to me now and was super helpful.

In T173821#3590409, @Fjalapeno wrote:

[...]

So I'm thinking of how much we can offload to the PCS here… going off what you are saying, here is my first idea for the "common" styles:

1. We manually decide what the "common" styles are (we can start with what is bundled in the apps and modify that based on your current work)

👍 Sounds good

1. We set up logic in PCS to get these basic styles from the MediaWiki API and cache them

This is one option, another one would just be provide the URL to the Mediawiki URL in the HTML and you wouldn't have to worry about fetching and exposing them at the services layer.

Of course handling the styles URL on PCS gives room for interesting considerations.

For example if we version the URLs by adding for example a content hash based on the files, then we could leverage of things like Cache-Control: immutable, and possibly make some performance improvements from that.

That'd be, instead of:

• <wp.domain>/w/load.php?modules=mediawiki.skinning.content.parsoid&...

Something like:

• <pcs.domain>/styles/page.4ab87fb12cd.css
  • 4ab87fb12cd being the hash of the contents of the MediaWiki URL with the modules we decide are the common

This is very interesting but it also may have implications on the styles cache eviction, so I think it would need to be discussed/planned in more detail to find holes in the approach.

1. We will expose the sheet as a single unchanging URL that can be easily cached by clients

👍 sounds good. To be more specific, by unchanging do you mean:

1. The contents are fixed by URL ([file].[contenthash].css) and when the contents change, the URL changes?
   • Could benefit from better caching (see previous point), but needs consideration for cache invalidation or asset regeneration if evicted and requested.
2. The URL is always the same, and the contents may change?
   • A prettier alias to the MediaWiki URL

1. We add this URL to the page HTML (this should be ok since the URL is unchanging and can be cached)

👍

Apps can still bundle them, but would prefer whatever is at the URL as it may be newer.

For the "lesser" used styles, I'm also thinking if we can do this without burdening the clients… but I'm less sure, would this work?

We could go through the trouble, or we could just rely on giving the clients the module names, or a bunch of single module urls and have them fetch the styles (automagically via browser, or else).

I'm not sure what would be the best approach. It would be good to get stats about the modules included in parsoid output for all pages of a big wiki, to get an idea of what are some of the statistically significant styles and if they should go to the common CSS.

1. When a page is first fetched in the PCS, we get all modules for that page
2. We request each sheet individually from the MW API, and cache each individually (If a style sheet was cached previously, we would not get it again)
3. We expose each sheet as a single unchanging URL so that it can be easily cached by clients
4. We add these URLs to the page HTML to reference these other sheets (this should be ok since the URL is unchanging and can be cached)

That'd work. Styles would then be exposed as link[rel=stylesheet] tags on the HTML, right?

Basically, I'm trying to solve this doing what you are saying, except instead of doing it on each client, we do it in the service itself.

Sure, making these decisions in one place is better than having to replicate them across platforms.

You can also improve the client's capabilities by doing things like the content hash on the URLs, and serving the assets with cache-control: immutable, so there are many things that can be improved at the service level.

I don't see a way around sometimes having multiple URLs, but if we make the URLs unchanging and deterministic we should get the caching benefits (I think)

Agreed, it is a tradeoff, but one I think is worth. As I mentioned above, getting some data about the styles modules and their significance over the whole article base, could give us a good idea of which modules could go to common directly to avoid having many URLs that will be requested very often. Bonus points if we weight the data by pageviews.

Does this sound viable to you or am I way off base here?

Thanks again for all the background… this makes a lot of sense to me now and was super helpful.

Yes! Thanks for taking the time to read it through. Seems like we're starting to have enough info to draft a proposal or plan some research tasks.

Discussed this in the services meeting. One thing that came up is cache invalidation. Currently there is no event we can use to evict the CSS from the cache the it is updated.

So if we go with the plan to have static URLS, we have 2 options:

1. Find a way to invalidate the cache reliably
2. Expire the cache with a sufficient TTL in Varnish

#1 would be preferable, as w could update the cache immediately when CSS changes. #2 has the downside of the CSS being out of sync with the content for an amount of time.

However, @mobrovac pointed out that #1 has technical difficulties to overcome.

@mobrovac you mentioned something along the lines of the CSS being in Github is what is complicating us triggering an event. I'm not super familiar with how we manage CSS right now (aside from the fact it is delivered by ResourceLoader). Can you elaborate on this? Thanks

CSS files are considered to be part of the codebase. That means that they (can) change with each code deploy. Consequently, there are no events happening in the system when a particular source file changes (and given the amount of files we have in all of the repositories combined, it's not even feasible to do so). In order words, CSS files are not considered content per se. If we wanted to promote them to that status, option #2 is probably the most pragmatic and easiest to implement in the short term. To implement #1 reliably, we could explore setting up some hooks during the deployment process that could/would check the CSS files' last modified time. However, this assumes that (i) the CSS files are all part of the same service codebase; (ii) all of them are in the same directory (for setting up the checks more easily); and (iii) we can actually pull it off with Scap.

Wrt caching, the URI structure and the variants it can have are also important. What were you thinking here? Would there be a set of CSS files that would be accessed more often than others? How does the client's skin influence the choice?

Yeah it is a complicated topic. Besides the style in the codebases, there are also styles in "content", the common.css, or the skin css (Vector.css or Minerva.css), and soon there will be styles in templates...

At the very least, the smallest change we could do is to expose the link[rel=stylesheet] tag in the HTML as parsoid serves it.

Wrt caching, the URI structure and the variants it can have are also important. What were you thinking here?

@mobrovac Nothing concrete… if you have some thoughts here to help guide the design it would be helpful.

Would there be a set of CSS files that would be accessed more often than others?

Yes, What @Jhernandez and I were thinking was to identify and bundle all "common" CSS into a file and host that at a single URL (This is essentially what the apps do now but they bundle it).

How does the client's skin influence the choice?

Currently, I am thinking we would gear this towards the Reading clients (apps and mobile web) and their skin choice. That way it is something we can predict and cache.

In T173821#3614777, @Fjalapeno wrote:

Wrt caching, the URI structure and the variants it can have are also important. What were you thinking here?

@mobrovac Nothing concrete… if you have some thoughts here to help guide the design it would be helpful.

A straw-man to kick-start the bike-shedding session: https://{domain}/api/rest_v1/(media|data)/css/{file}{/skin}. We could put it either under data or media, depending on how we look at it, I guess :) The skin could be optional with a sane default (as I am assuming here that native clients don't have the concept of skin as MW does).

Would there be a set of CSS files that would be accessed more often than others?

Yes, What @Jhernandez and I were thinking was to identify and bundle all "common" CSS into a file and host that at a single URL (This is essentially what the apps do now but they bundle it).

That sounds sane and cache-friendly, but the caveat here is that bundling different CSS files into one makes tracking changes harder.

How does the client's skin influence the choice?

Currently, I am thinking we would gear this towards the Reading clients (apps and mobile web) and their skin choice. That way it is something we can predict and cache.

+1

A straw-man to kick-start the bike-shedding session: https://{domain}/api/rest_v1/(media|data)/css/{file}{/skin}.

Thanks for the info @mobrovac. Something along those lines looks fine to me. Apps do use some CSS from Minerva I believe, this may or may not change, not sure yet. Regardless, I guess a URI like this gives us the option of supporting other skins as needed in the future.

That sounds sane and cache-friendly, but the caveat here is that bundling different CSS files into one makes tracking changes harder.

Yep… I guess if we are just going with just expiring the cache with a reasonable TTL, then maybe it doesn't mater if we can track changes at all.

It sounds like there are two separate questions:

1. How to return the list of non-standard (page specific) CSS / JS modules needed to render a specific piece of content. (Examples: HTTP headers, HTML link or meta tags.)
2. How to fetch missing (or outdated) modules. Currently, the normal practice is to fetch modules from ResourceLoader. Client side logic can take into account cache state, or replace specific modules with variants tailored to the use case. CDN caching is supported for ResourceLoader requests.

The discussion here seems to be primarily about alternate solutions for 2), possibly optimized for mobile use cases. What would be the issue with using ResourceLoader requests for now?

@GWicke I think @Jhernandez's comment was pretty illustrative to me. Was there something else beyond his reasoning there that you were looking for?

https://phabricator.wikimedia.org/T173821#3587859

@Fjalapeno, that comment touches on 1), but as I said to me it looks like the API focused discussion has moved to 2). Either way, I am not sure we need a new API for either 1) or 2).

In T173821#3616072, @GWicke wrote:

It sounds like there are two separate questions:

1. How to return the list of non-standard (page specific) CSS / JS modules needed to render a specific piece of content. (Examples: HTTP headers, HTML link or meta tags.)

I'm not sure that is entirely correct… this ticket is about how to list ALL styles needed by a page. Of which page specific CSS is a subset. Either way, the current idea from @Jhernandez's comment:

…Styles would then be exposed as link[rel=stylesheet] tags on the HTML…

It's just that "common styles" would be bundled into a single file at a single URL

2. How to fetch missing (or outdated) modules. Currently, the normal practice is to fetch modules from ResourceLoader. Client side logic can take into account cache state, or replace specific modules with variants tailored to the use case. CDN caching is supported for ResourceLoader requests.

Yes, this is served by ResourceLoader, and @Jhernandez stated why this is a problem:

… If the client includes the full styles URL as it comes in the HTML, then it will be requesting all of the CSS, when it already has most of it from the bundled ones on the app, or previously visited pages … Having individual URLs per module in the HTML is not an option for production because of the amount of HTTP requests would be terrible for performance.…

So if we make the request in a single ResourceLoader URL, then the CSS can't be cached in a useful way, and if we fetch them individually for each module, then it is too many API calls.

In T173821#3622153, @GWicke wrote:

@Fjalapeno, that comment touches on 1), but as I said to me it looks like the API focused discussion has moved to 2). Either way, I am not sure we need a new API for either 1) or 2).

The solution being discussed is to bundle the common CSS into a single file, which we will need to re-host at a URL… it's not really an API as much as a hosted file, but I think we do need a service to do this.

If I recall correctly, ResourceLoader client code on desktop already looks at a list of modules needed in a given page, checks client side caches, and fetches the remaining modules from the RL API (in a single call), and caches those modules separately in localstorage. Given that this discussion is making no reference to this, I am getting the impression that this understanding might be wrong. Could you clarify?

Either way, I think it is worth considering separating general skin styles (which might be packaged / structured differently depending on skins) from page / content specific modules. For most composition & custom formatting tasks, we are only interested in the list of additional dependencies (as metadata), and can safely ignore the general skin modules, as they are handled through other, context-specific means.

In T173821#3622245, @GWicke wrote:

If I recall correctly, ResourceLoader client code on desktop already looks at a list of modules needed in a given page, checks client side caches, and fetches the remaining modules from the RL API (in a single call), and caches those modules separately in localstorage. Given that this discussion is making no reference to this, I am getting the impression that this understanding might be wrong. Could you clarify?

I believe you are right and this is how it works on desktop. The issue is that this logic is problematic (especially in a mobile context) because this is one of the things blocking the render path: Being unable to display the page while the client determines what modules it needs and then constructs a URL to fetch missing modules using a ResourceLoader URL which by its nature is useless for local caching. I think getting away from this is mostly implicit in the premise of this ticket.

The main thrust of the solution being proposed is that much of this logic can be performed on behalf of the client (what modules are needed for this page) and then add the URLs to the CSS files directly to the HTML. Then allow the clients can simply use their HTTP cache for determining which CSS files they don't have cached locally, just like any other file. This removes client specific logic in favor of using standard browser based mechanisms for obtaining these CSS files.

Either way, I think it is worth considering separating general skin styles (which might be packaged / structured differently depending on skins) from page / content specific modules. For most composition & custom formatting tasks, we are only interested in the list of additional dependencies (as metadata), and can safely ignore the general skin modules, as they are handled through other, context-specific means.

Agreed, and there is definitely a different path for the general styles (bundling common files into a single file at a single URL) vs page/content specific styles (individual files at individual URLs) laid out here.

Am I missing anything else? I think this is addressing your points, we can also discus at the next services meeting, too

The only styles a web app should really be concerned with are the modules that vary per page. You can get these from the API e.g. https://en.m.wikipedia.org/w/api.php?formatversion=2&format=json&action=parse&prop=text|modules&page=Paris

Everything else, I would expect the client to take care of (other styles are provided by the 'skin' or an 'extension' but I'd imagine in a client the client is acting as both skin and extension(s)).

In T173821#3622245, @GWicke wrote:

If I recall correctly, ResourceLoader client code on desktop already looks at a list of modules needed in a given page, checks client side caches, and fetches the remaining modules from the RL API (in a single call), and caches those modules separately in localstorage. Given that this discussion is making no reference to this, I am getting the impression that this understanding might be wrong. Could you clarify?

AFAIK this is correct. That code doesn't run on the native apps for example, so the opted for bundling a subset of hand picked styles. As @Fjalapeno mentioned (emphasis mine):

The main thrust of the solution being proposed is that much of this logic can be performed on behalf of the client (what modules are needed for this page) and then add the URLs to the CSS files directly to the HTML. Then allow the clients can simply use their HTTP cache for determining which CSS files they don't have cached locally, just like any other file. This removes client specific logic in favor of using standard browser based mechanisms for obtaining these CSS files.

Thanks for the info @Jdlrobson, I've added it to https://www.mediawiki.org/wiki/Reading/Web/Projects/NewMobileWebsite/API_Inventory to keep track.

Waiting on CSS development to start… blocking on that