@Aklapper Thx for pointing me to this, I have updated the task with the expected information.

@abi_ Thank you!

@Cparle Thank you for the feedback.

@ArielGlenn Thank you for your interest and your patience. Here it is https://phabricator.wikimedia.org/tag/affects-kiwix-and-openzim/

@Ramsey-WMF Hi, what is the status on this? I kind of believe the work has made progresses... but this is tracked in an other ticket!?

@Andrew Thank you for your offer. In 2019 we have invested to improve the performance of our MWoffliner scraper. This effort is not over, there is 2-3 additional important things I want to get done before asking you guys to do more work ;) For the moment the overall VM setup we have (mwoffliner project) is pretty much OK and I'm so thankful you provide it. But of course improving performance of software changes the HW footprint and we will probably later have to rebuild/changes our VM landscape to be more efficient.

In T91976#5780336, @Krenair wrote:

In T91976#5778398, @bd808 wrote:

This would be some type of major re-write of how OpenStack manages instances or somehow creating, registering, and using a new image on the fly.

s/image/flavour/?

In T91976#5778480, @Kelson wrote:

@BD2412 Thank you for answering my request. I understand that we have to stick to OpenStack concepts. That said I have to emphasis that this approach IMO leads to a waste of HW resources. In addition, it sounds also a bit "outdated" to me, if I compare to other cloud solutions which provide a small(er) HW configuration granularity.

Can you give some examples? AWS has pre-defined instance sizes.

@BD2412 Thank you for answering my request. I understand that we have to stick to OpenStack concepts. That said I have to emphasis that this approach IMO leads to a waste of HW resources. In addition, it sounds also a bit "outdated" to me, if I compare to other cloud solutions which provide a small(er) HW configuration granularity.

@Aklapper Looks good to me. Thx.

I have forgotten to write that, after discussing this topic with @Aklapper, it looks like a solution might be:

• A dedicated tag "Kiwix & openZIM"
• A dedicated workboard where we would be able to put the tickets we would like to see fixed/implemented in priority.

@Dzahn Thank you. I have been able to connect to the admin UI.

@Aklapper Happy you tackle that issue. I wanted to open that ticket myself for quite a long time already. You can put me in the list of admins.

I don't think the last fix deployment has fixed anything regarding that ticket. The Parsoid output is still broken so far I can see.

@bearND Thank you for the explanation

@fgiunchedi You are perfectly right. This seems to be an invalid bug. In fact I made the error because "File:Poaceae Habitus 2010-10-03 FuentedeSanLorenzoSierraMadrona.jpg" has been redirected from "File:Isoetes setaceum Habitus 2010-10-03 FuentedeSanLorenzoSierraMadrona.jpg". This is probably a consequence of T226931

@ssastry Thank you!

@Aklapper The whole offline/Kiwix stuff is handled by the new-readers team. That said, totally unsure if this is right to put it here. I would really like to have a tag or project to gather all openZIM/Kiwix impacting tickets. Would that be somehow possible?

Like T217540 is one good source of broken/outdated HTML (here with invalid images) delivered by Parsoid. This pretty annoyous for offline version of our wikis.

@ssastry I don't really understand why this ticket is closed. There is no agreement that this is a bug?

@mobrovac Thx, so I see that this concerns 307 web sites and for us, as around 20 are meta web sites that we don't want to scrape anyway, this is around 280 wikis we hardly can scrape.

BTW, it.wikibooks.org seems to be also impacted. Do we have a list somewhere of all the wikis where the mobile API endpoints are not "properly" available?

This problem stops us to provide a new offline version of Wikispecies.

@Ariel Sorry this was a problem on our end. Is fixed now.

What happens with qstat, is that still available?

@Audiodude thx

@bd808 Thx, I have successfully created the new WP1 instance.

Thx to all the participants!

@Bstorm As info, around 2 weeks ago we have introduced mwoffliner 1.8 on our VPS boxes. This new version should require between 3 and 5 times less IO activity than before. We should also not store a lot of data for longer than 1 month like before.

@bd808 I think a m1.xlarge might be enough for now, but not so sure how long it will last. Your analysis is good but you are missing an information. The jobs which need the most memory are not running on the grid. The datamining jobs were running on the pre-existing mwoffliner VPS boxes, but as we use these boxes more intensively than before, I want (or have) to stop to do so.

@GTirloni I wonder if these files are legacy after we migrated all our WM in December. If the files are older than 2019, then you can remove them I think.

@Krenair Yes, we fully understand that and this is our fault. We will have to deal with a few days downtime.

@Krenair Any chance to have a valid homepage? Even if the wiki is closed?

@akosiaris Problem seems to be fixed from our end. Thank you very much.

I'm not sure to fully understand the technical explanation. Is the problem confirmed? If "yes", what is the plan to solve it?

@Samwilson yes the qrpedia repo on Wikimedia source forge is the good one.

Here is how mine works now, id is the Twitter user:

<includeonly><a class="twitter-timeline" href="https://twitter.com/<!--{$id}-->?ref_src=twsrc%5Etfw"></a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="https://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script></includeonly>

@Reedy Thank you for your quick feedback. I was not aware that Mediawiki HHVM support will be deprecated. As a consequence, I have replaced HHVM with php-fpm and the problem disappeared.

Here a better example:

<quiz>
{ test }
</quiz>

All 5 boxes have been migrated to Debian Stretch

@Automactic I have added you to the project achttps://wikitech.wikimedia.org/wiki/Special:NovaProject. If you have configure your SSH key, I think you should have access to the VMs through SSH.

I have migrated one of the 5, the 4 others will follow.

@Krenair Thank you for the reminder as I was believing the deadline is somewhere in Spring 2019. Will do the necessary in November.

@Ramesh2744 Thank you very much for the information.

@Ramesh2744 Given a title (multimedia file) retrieve author/license (ideally title/description with corresponding language if available) in a machine readable format.