The current mobile site (e.g., http://en.m.wikipedia.org/) is written in Ruby. Having the site in Ruby causes a number of headaches (for code review, code deployment, and server maintenance) given Wikimedia's mostly PHP-based infrastructure.
I've created a page at http://www.mediawiki.org/wiki/Mobile_site_rewrite for notes about this.
Version: .5
Severity: normal
URL: http://www.mediawiki.org/wiki/Mobile_site_rewrite
hcatlin wrote:
MZ, its great to see someone standing up and taking on this huge project
that so many feel so passionately about. I can't wait for the day when I
can switch off the mobile site.
Goodluck on the new project you're starting!
Oh, a couple notes on things that will make it hard to get into the current
architecture (and no one ever seems to think of).
Right now, each path comes back with a single resulting HTML page. Except,
on mobile, we have up to 20 different pages that might come back as determined
by the user agent. Some of them are 'similar' pages to each other (aka, most webkit
style browsers get something similar with only small tweaks) and others are entirely
different formats.
That doesn't mean that you can't use PHP for it. Ruby presents no advantages there.
But, it does mean that its extremely difficult to use the mainline wikipedia architecture
as its kind of a 'simple case' setup.
Oh right, and a ton of different mobile applications (and other third party applications)
use the software's API. Its the API that our native phone apps use and others have taken
to use it. So, most of the popular native phone apps on all platforms use the current
site's API. Its pretty simple, but you have to make sure that you either communicate updates
with all of them or mimic the current API as to not seriously break stuff.
Goodluck on the "Big Re-Write!"
ayg wrote:
Filing a bug need not imply a willingness to fix it. Bug trackers are meant to track things that need to be done, even if they aren't yet assigned to anyone.
I think they can mostly be replaced with skins (which should branch appropiately).
Mobile does some things the hard way.
(In reply to comment #1)
MZ, its great to see someone standing up and taking on this huge project
that so many feel so passionately about. I can't wait for the day when I
can switch off the mobile site.
Hi! I don't think I'll be the person working on this (though stranger things have happened). If this sits around long enough, it might make for a really nice Google Summer of Code project, though. I've listed it here: http://www.mediawiki.org/wiki/Summer_of_Code_2011/Project_ideas (You could even be a mentor!)
Right now, each path comes back with a single resulting HTML page. Except,
on mobile, we have up to 20 different pages that might come back as determined
by the user agent. Some of them are 'similar' pages to each other (aka, most
webkit style browsers get something similar with only small tweaks) and
others are entirely different formats.
Are these differences documented? Also, is it possible currently to emulate these settings without faking the user-agent? Perhaps with a URL parameter of some kind for easier debugging? Having a way to A-B test would certainly make a rewrite easier. :-)
Oh right, and a ton of different mobile applications (and other third party
applications) use the software's API. Its the API that our native phone apps
use and others have taken to use it. So, most of the popular native phone
apps on all platforms use the current site's API. Its pretty simple, but you
have to make sure that you either communicate updates with all of them or
mimic the current API as to not seriously break stuff.
Good point. I didn't realize that the current mobile site had an API. I added keeping the API working to the rewrite specifications here: http://www.mediawiki.org/wiki/Mobile_site_rewrite#Rewrite_specifications
How is the API accessed and is it documented anywhere?
Goodluck on the "Big Re-Write!"
Thanks!
(In reply to comment #3)
I think they can mostly be replaced with skins (which should branch
appropiately).
I added this idea here: http://www.mediawiki.org/wiki/Mobile_site_rewrite#Future_implementation
Since the community voted on this to be a strategic priority we'll actually be hiring someone to focus on the development work. The timeline is to get them in and running Q4 of this year .. which is of course .. right now :)
Stay tuned for updates.
(In reply to comment #5)
Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)Stay tuned for updates.
Out of curiosity, is there a link to where the community "voted" on this? (I presume somewhere on strat wiki, but a quick search couldn't find it)
(In reply to comment #5)
Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)Stay tuned for updates.
I was indication last night that the foundation didn't really care about the mobile platform and weren't putting money towards it for at least 2 years, Can you provide some indication of where this vote was?
This I found easily enough: http://strategy.wikimedia.org/wiki/Strategic_Plan/Role_of_the_WMF#Invest_in_mobile_and_offline_products_to_broaden_reach_to_new_populations
It doesn't discuss rewriting the mobile site but doing so (rebuilding it so that it could be maintained by other php engineers) would easily fall into the remit of the plan.
A quick search brings up
http://strategy.wikimedia.org/wiki/Proposal:Mobile_platform_flexibilities.
and their are other mobile priorities listed through the wiki.
Rewriting the gateway is just the initial step to making requests like that easier for both ops and development.
I'll be writing up a project page for it in the next weeks and will post the link as it starts coming together so that others can participate and comment.
(In reply to comment #9)
A quick search brings up
http://strategy.wikimedia.org/wiki/Proposal:Mobile_platform_flexibilities.
and their are other mobile priorities listed through the wiki.
So, to quote that page, this is a priority because "the chicks love generous, modern philanthropists" (sorry couldn't resist). On a serious note I think I mis-interpreted comment 5 as meaning there was a massive vote of wikimedians about mobile, rather than a more general, lots of people at strat thought it would be a good idea (which it is).
Rewriting the gateway is just the initial step to making requests like that
easier for both ops and development.
Not to mention mediawiki re-users.
I'll be writing up a project page for it in the next weeks and will post the
link as it starts coming together so that others can participate and comment.
Cool, I'll keep an eye out for that.
Cheers,
(In reply to comment #5)
Since the community voted on this to be a strategic priority we'll actually be
hiring someone to focus on the development work. The timeline is to get them in
and running Q4 of this year .. which is of course .. right now :)Stay tuned for updates.
Interesting... What type of new mobile server will we be wanting to create ? A full rewrite in php of the current Ruby server, or are we integrating it into standard mediawiki, by using different skins ?
I worked a bit on device support for the Ruby version, and if needed, I can help with some of those peculiarities among the different devices. There are plenty of them :D
I think we should use directly the parser output (this means that -for some systems- the parser cache could be straightly used).
The reprocessing method of fetching a wikipedia page then applying changes is quite ugly.
Hampton, is the 'encode everything into entities' actually needed by some devices?
Will we be including support for more than just iOS/android ? (so multiple
skins ?)
There are 17 backends there: android capable iphone iphone2 kindle kindle2 nativeiphone netfront nokia_series_60 operamini palm_pre ps3 psp wap2 webkit wii wml.
I thought we would use a meta-skin. Some changes are just using a different CSS or changing a field type in the header (I don't think all of them are intentional).
About the WML, it's a cool feature, although I'm not sure how right it currently is (eg. it generates h3 without h2...). As it won't be much tested, it could break without anyone noticing for years (for instance, faking a wml request, I'm not getting the article content),
As for providing it, it seems easy to duplicate the reprocessing stage, and I think the parser provides enough hooks to do it there, too.
Another place we might need reprocessing is for phones that only support XHTML Mobile Profile, and not full HTML. This is sometimes called WAP 2.0.
There are 3 basic skins:
**Depending on screensize limitations, images are either full screenwidth or allow text flowing beside them.
Most of these devices use XHTML MP and with the exception of OperaMini don't support full HTML. Some don't deal well with JS (crash). Most have keypad navigation trough use of accesskey properties in the navigation links of wiki interface.
PS3 uses this skin as it has a very limited browser based on NetFront (as do PSP and Sony Ericsson phones)
I spoke with Hampton at Wikimania about this idea, and he seemed keen on it -
perhaps it's time to socialize it more broadly.
The mobile site should be able to render wikitext directly into it's desired
format(s), otherwise we are basically just screen-scraping ourselves.
Post-processing is always going to be a fragile hack. I believe the solution is
to create a wikitext parser/renderer architecture, where the parser expands
templates and then builds a document object model and then the renderer turns
that structure into code. Then we could simply extend the system to render to
an alternate markup which is ideal for the target.
I wrote a parser earlier this year that converts most wikitext into a document
object model. It needs more work but it's a good start, I will try and get it
on the SVN this week. Using the parser I wrote, I believe we could write a
renderer that can output the same HTML as the current parser does, as well as
one that outputs the same HTML that the mobile site does.
My 2 cents.
mike.lifeguard+bugs wrote:
If having this written in PHP is a priority, why was it permitted to be written in Ruby in the first place?
(In reply to comment #16)
If having this written in PHP is a priority, why was it permitted to be written
in Ruby in the first place?
I wasn't there, so this could well be wrong, but when it was first written the priority was just getting it in place, not necessarily longer-term maintenance.
The pilot project was a success, a need and use case for mobile front end has been seen and widely understood. Now, how will we keep up long-term maintenance?
I'm working with Tomasz to get this rewrite going and already have a prototype server allocated for it.
It was obtained, already written, along with the iPhone app that used it. There wasn't any permitting for something to be written in Ruby, but there was a choice to obtain something written in Ruby.
For those following this bug please check http://www.mediawiki.org/wiki/Mobile_site_rewrite for updates on the rewrite project. We'll be demoing the first version of it at the Berlin hackathon.
Good. Where's the code? PatchOutputMobile ?
Why does it edit the outputted html and parse it with DOM extension ??
What will happen next time an output is not XHTML conformant? (eg. unadvertedly by the sysops), whoops out goes your mobile skin (not sure how tolerant is loadHtml but it seems to at least give a bunch or warnings...)
I thought that "rewrite" meant "make it sensible". Yes, I see how is it copying the ruby code, that did itin the same (wrong) way. It should be hooking as a Skin (or Linker, not sure how you change it after Dantman splitted them).
(In reply to comment #20)
Good. Where's the code? PatchOutputMobile ?
Why does it edit the outputted html and parse it with DOM extension ??What will happen next time an output is not XHTML conformant? (eg. unadvertedly
by the sysops), whoops out goes your mobile skin (not sure how tolerant is
loadHtml but it seems to at least give a bunch or warnings...)I thought that "rewrite" meant "make it sensible". Yes, I see how is it copying
the ruby code, that did itin the same (wrong) way. It should be hooking as a
Skin (or Linker, not sure how you change it after Dantman splitted them).
Yup, the work so far has been in http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/PatchOutputMobile/. Patrick has been working with Brion and Hampton to move that along. Can you pass on some non conformant XHTML for us to test with? We'll take that on during the hackathon and see how badly it explodes.
ayg wrote:
loadHtml() is unlikely to break on malformed input. The bigger issue is that doing it this way is both fragile and requires more code, compared to just writing a skin. The only reason to hack the HTML output is if you're not familiar with the MediaWiki codebase or skin system, in which case that's easier than doing it properly -- but I'd hope Wikimedia has enough employees who understand MediaWiki to do it right.
I'd still hope we'd be looking at a more "skin" based solution... So other people can use it without spinning up other services and such
(In reply to comment #23)
I'd still hope we'd be looking at a more "skin" based solution... So other
people can use it without spinning up other services and such
Right. I started a thread on wikitech-l to discuss this: http://lists.wikimedia.org/pipermail/wikitech-l/2011-May/053305.html.
(In reply to comment #27)
Let's not close it until MobileFrontend is the primary mobile interface.
Isn't this the purpose of Bug 30512?
(In reply to comment #28)
Isn't this the purpose of Bug 30512?
Ah, yes, thanks for the reminder!