Page MenuHomePhabricator

Mobile site should be rewritten in PHP
Closed, ResolvedPublic

Description

The current mobile site (e.g., http://en.m.wikipedia.org/) is written in Ruby. Having the site in Ruby causes a number of headaches (for code review, code deployment, and server maintenance) given Wikimedia's mostly PHP-based infrastructure.

I've created a page at http://www.mediawiki.org/wiki/Mobile_site_rewrite for notes about this.


Version: .5
Severity: normal
URL: http://www.mediawiki.org/wiki/Mobile_site_rewrite

Details

Reference
bz25558

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 21 2014, 11:21 PM
bzimport set Reference to bz25558.

hcatlin wrote:

MZ, its great to see someone standing up and taking on this huge project
that so many feel so passionately about. I can't wait for the day when I
can switch off the mobile site.

Goodluck on the new project you're starting!

Oh, a couple notes on things that will make it hard to get into the current
architecture (and no one ever seems to think of).

Right now, each path comes back with a single resulting HTML page. Except,
on mobile, we have up to 20 different pages that might come back as determined
by the user agent. Some of them are 'similar' pages to each other (aka, most webkit
style browsers get something similar with only small tweaks) and others are entirely
different formats.

That doesn't mean that you can't use PHP for it. Ruby presents no advantages there.
But, it does mean that its extremely difficult to use the mainline wikipedia architecture
as its kind of a 'simple case' setup.

Oh right, and a ton of different mobile applications (and other third party applications)
use the software's API. Its the API that our native phone apps use and others have taken
to use it. So, most of the popular native phone apps on all platforms use the current
site's API. Its pretty simple, but you have to make sure that you either communicate updates
with all of them or mimic the current API as to not seriously break stuff.

Goodluck on the "Big Re-Write!"

ayg wrote:

Filing a bug need not imply a willingness to fix it. Bug trackers are meant to track things that need to be done, even if they aren't yet assigned to anyone.

I think they can mostly be replaced with skins (which should branch appropiately).

Mobile does some things the hard way.

(In reply to comment #1)

MZ, its great to see someone standing up and taking on this huge project
that so many feel so passionately about. I can't wait for the day when I
can switch off the mobile site.

Hi! I don't think I'll be the person working on this (though stranger things have happened). If this sits around long enough, it might make for a really nice Google Summer of Code project, though. I've listed it here: http://www.mediawiki.org/wiki/Summer_of_Code_2011/Project_ideas (You could even be a mentor!)

Right now, each path comes back with a single resulting HTML page. Except,
on mobile, we have up to 20 different pages that might come back as determined
by the user agent. Some of them are 'similar' pages to each other (aka, most
webkit style browsers get something similar with only small tweaks) and
others are entirely different formats.

Are these differences documented? Also, is it possible currently to emulate these settings without faking the user-agent? Perhaps with a URL parameter of some kind for easier debugging? Having a way to A-B test would certainly make a rewrite easier. :-)

Oh right, and a ton of different mobile applications (and other third party
applications) use the software's API. Its the API that our native phone apps
use and others have taken to use it. So, most of the popular native phone
apps on all platforms use the current site's API. Its pretty simple, but you
have to make sure that you either communicate updates with all of them or
mimic the current API as to not seriously break stuff.

Good point. I didn't realize that the current mobile site had an API. I added keeping the API working to the rewrite specifications here: http://www.mediawiki.org/wiki/Mobile_site_rewrite#Rewrite_specifications

How is the API accessed and is it documented anywhere?

Goodluck on the "Big Re-Write!"

Thanks!

(In reply to comment #3)

I think they can mostly be replaced with skins (which should branch
appropiately).

I added this idea here: http://www.mediawiki.org/wiki/Mobile_site_rewrite#Future_implementation

Since the community voted on this to be a strategic priority we'll actually be hiring someone to focus on the development work. The timeline is to get them in and running Q4 of this year .. which is of course .. right now :)

Stay tuned for updates.

(In reply to comment #5)

Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)
Stay tuned for updates.

Out of curiosity, is there a link to where the community "voted" on this? (I presume somewhere on strat wiki, but a quick search couldn't find it)

(In reply to comment #5)

Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)
Stay tuned for updates.

I was indication last night that the foundation didn't really care about the mobile platform and weren't putting money towards it for at least 2 years, Can you provide some indication of where this vote was?

Jorm added a comment.Oct 19 2010, 3:05 AM

This I found easily enough: http://strategy.wikimedia.org/wiki/Strategic_Plan/Role_of_the_WMF#Invest_in_mobile_and_offline_products_to_broaden_reach_to_new_populations

It doesn't discuss rewriting the mobile site but doing so (rebuilding it so that it could be maintained by other php engineers) would easily fall into the remit of the plan.

A quick search brings up

http://strategy.wikimedia.org/wiki/Proposal:Mobile_platform_flexibilities.

and their are other mobile priorities listed through the wiki.

Rewriting the gateway is just the initial step to making requests like that easier for both ops and development.

I'll be writing up a project page for it in the next weeks and will post the link as it starts coming together so that others can participate and comment.

(In reply to comment #9)

A quick search brings up
http://strategy.wikimedia.org/wiki/Proposal:Mobile_platform_flexibilities.
and their are other mobile priorities listed through the wiki.

So, to quote that page, this is a priority because "the chicks love generous, modern philanthropists" (sorry couldn't resist). On a serious note I think I mis-interpreted comment 5 as meaning there was a massive vote of wikimedians about mobile, rather than a more general, lots of people at strat thought it would be a good idea (which it is).

Rewriting the gateway is just the initial step to making requests like that
easier for both ops and development.

Not to mention mediawiki re-users.

I'll be writing up a project page for it in the next weeks and will post the
link as it starts coming together so that others can participate and comment.

Cool, I'll keep an eye out for that.

Cheers,

TheDJ added a comment.Oct 19 2010, 8:56 AM

(In reply to comment #5)

Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)
Stay tuned for updates.

Interesting... What type of new mobile server will we be wanting to create ? A full rewrite in php of the current Ruby server, or are we integrating it into standard mediawiki, by using different skins ?

  • Will we keep to the reprocessing method, or are we directly accessing db and wikicode parser?
  • Will we drop support for WAP/WML (since it is not html ? )
  • Will we be including support for more than just iOS/android ? (so multiple skins ?)

I worked a bit on device support for the Ruby version, and if needed, I can help with some of those peculiarities among the different devices. There are plenty of them :D

I think we should use directly the parser output (this means that -for some systems- the parser cache could be straightly used).
The reprocessing method of fetching a wikipedia page then applying changes is quite ugly.

Hampton, is the 'encode everything into entities' actually needed by some devices?

Will we be including support for more than just iOS/android ? (so multiple

skins ?)
There are 17 backends there: android capable iphone iphone2 kindle kindle2 nativeiphone netfront nokia_series_60 operamini palm_pre ps3 psp wap2 webkit wii wml.

I thought we would use a meta-skin. Some changes are just using a different CSS or changing a field type in the header (I don't think all of them are intentional).

About the WML, it's a cool feature, although I'm not sure how right it currently is (eg. it generates h3 without h2...). As it won't be much tested, it could break without anyone noticing for years (for instance, faking a wml request, I'm not getting the article content),
As for providing it, it seems easy to duplicate the reprocessing stage, and I think the parser provides enough hooks to do it there, too.

TheDJ added a comment.Oct 19 2010, 5:26 PM

Another place we might need reprocessing is for phones that only support XHTML Mobile Profile, and not full HTML. This is sometimes called WAP 2.0.

TheDJ added a comment.Oct 19 2010, 5:43 PM

There are 3 basic skins:

  • Webkit style (for iphone/android/palmOS and other touch based devices, but also Wii which uses Opera )
    • Relative differences between the devices has to do with size of the search field, input element with type search vs. input type text, webkit2 vs. webkit3 (issues with positioning of search button If i remember correctly. Most of these styles use Javascript collapsable buttons for the sections.

**Depending on screensize limitations, images are either full screenwidth or allow text flowing beside them.

  • Wii uses a simpler version of this skin, and has adapted fontsize to improve readability on Television screens. It shares the section collapsing idea of the iphone interface.
  • Simple style (for SonyE, PSP, PS3, Kindle, OperaMini, and basically all other non-touch based or small screen devices).

Most of these devices use XHTML MP and with the exception of OperaMini don't support full HTML. Some don't deal well with JS (crash). Most have keypad navigation trough use of accesskey properties in the navigation links of wiki interface.
PS3 uses this skin as it has a very limited browser based on NetFront (as do PSP and Sony Ericsson phones)

  • Third skin is WAP/WML, which clearly is even more simple and not really based on HTML even
    • This mode is not fully supported, because we don't support chunking large pages into multiple cards/requests. Works for smaller pages though. Backup is currently in mobile.wikipedia.org, which does support this mode well.
  • The fourth skin, you could say is a json api mode for the iphone application (and some third party widgets and other external tools.

I spoke with Hampton at Wikimania about this idea, and he seemed keen on it -
perhaps it's time to socialize it more broadly.

The mobile site should be able to render wikitext directly into it's desired
format(s), otherwise we are basically just screen-scraping ourselves.

Post-processing is always going to be a fragile hack. I believe the solution is
to create a wikitext parser/renderer architecture, where the parser expands
templates and then builds a document object model and then the renderer turns
that structure into code. Then we could simply extend the system to render to
an alternate markup which is ideal for the target.

I wrote a parser earlier this year that converts most wikitext into a document
object model. It needs more work but it's a good start, I will try and get it
on the SVN this week. Using the parser I wrote, I believe we could write a
renderer that can output the same HTML as the current parser does, as well as
one that outputs the same HTML that the mobile site does.

My 2 cents.

mike.lifeguard+bugs wrote:

If having this written in PHP is a priority, why was it permitted to be written in Ruby in the first place?

(In reply to comment #16)

If having this written in PHP is a priority, why was it permitted to be written
in Ruby in the first place?

I wasn't there, so this could well be wrong, but when it was first written the priority was just getting it in place, not necessarily longer-term maintenance.

The pilot project was a success, a need and use case for mobile front end has been seen and widely understood. Now, how will we keep up long-term maintenance?

I'm working with Tomasz to get this rewrite going and already have a prototype server allocated for it.

It was obtained, already written, along with the iPhone app that used it. There wasn't any permitting for something to be written in Ruby, but there was a choice to obtain something written in Ruby.

For those following this bug please check http://www.mediawiki.org/wiki/Mobile_site_rewrite for updates on the rewrite project. We'll be demoing the first version of it at the Berlin hackathon.

Good. Where's the code? PatchOutputMobile ?
Why does it edit the outputted html and parse it with DOM extension ??

What will happen next time an output is not XHTML conformant? (eg. unadvertedly by the sysops), whoops out goes your mobile skin (not sure how tolerant is loadHtml but it seems to at least give a bunch or warnings...)

I thought that "rewrite" meant "make it sensible". Yes, I see how is it copying the ruby code, that did itin the same (wrong) way. It should be hooking as a Skin (or Linker, not sure how you change it after Dantman splitted them).

(In reply to comment #20)

Good. Where's the code? PatchOutputMobile ?
Why does it edit the outputted html and parse it with DOM extension ??
What will happen next time an output is not XHTML conformant? (eg. unadvertedly
by the sysops), whoops out goes your mobile skin (not sure how tolerant is
loadHtml but it seems to at least give a bunch or warnings...)
I thought that "rewrite" meant "make it sensible". Yes, I see how is it copying
the ruby code, that did itin the same (wrong) way. It should be hooking as a
Skin (or Linker, not sure how you change it after Dantman splitted them).

Yup, the work so far has been in http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/PatchOutputMobile/. Patrick has been working with Brion and Hampton to move that along. Can you pass on some non conformant XHTML for us to test with? We'll take that on during the hackathon and see how badly it explodes.

ayg wrote:

loadHtml() is unlikely to break on malformed input. The bigger issue is that doing it this way is both fragile and requires more code, compared to just writing a skin. The only reason to hack the HTML output is if you're not familiar with the MediaWiki codebase or skin system, in which case that's easier than doing it properly -- but I'd hope Wikimedia has enough employees who understand MediaWiki to do it right.

I'd still hope we'd be looking at a more "skin" based solution... So other people can use it without spinning up other services and such

(In reply to comment #23)

I'd still hope we'd be looking at a more "skin" based solution... So other
people can use it without spinning up other services and such

Right. I started a thread on wikitech-l to discuss this: http://lists.wikimedia.org/pipermail/wikitech-l/2011-May/053305.html.

Reedy added a comment.May 22 2011, 6:11 PM
  • Bug 29086 has been marked as a duplicate of this bug. ***

This bug can be closed, right?

Let's not close it until MobileFrontend is the primary mobile interface.

(In reply to comment #27)

Let's not close it until MobileFrontend is the primary mobile interface.

Isn't this the purpose of Bug 30512?

(In reply to comment #28)

Isn't this the purpose of Bug 30512?

Ah, yes, thanks for the reminder!