@Andrew Thank you for your offer. In 2019 we have invested to improve the performance of our MWoffliner scraper. This effort is not over, there is 2-3 additional important things I want to get done before asking you guys to do more work ;) For the moment the overall VM setup we have (mwoffliner project) is pretty much OK and I'm so thankful you provide it. But of course improving performance of software changes the HW footprint and we will probably later have to rebuild/changes our VM landscape to be more efficient.

In T91976#5780336, @Krenair wrote:

In T91976#5778398, @bd808 wrote:

This would be some type of major re-write of how OpenStack manages instances or somehow creating, registering, and using a new image on the fly.

s/image/flavour/?

In T91976#5778480, @Kelson wrote:

@BD2412 Thank you for answering my request. I understand that we have to stick to OpenStack concepts. That said I have to emphasis that this approach IMO leads to a waste of HW resources. In addition, it sounds also a bit "outdated" to me, if I compare to other cloud solutions which provide a small(er) HW configuration granularity.

Can you give some examples? AWS has pre-defined instance sizes.

@BD2412 Thank you for answering my request. I understand that we have to stick to OpenStack concepts. That said I have to emphasis that this approach IMO leads to a waste of HW resources. In addition, it sounds also a bit "outdated" to me, if I compare to other cloud solutions which provide a small(er) HW configuration granularity.

@Aklapper Looks good to me. Thx.

I have forgotten to write that, after discussing this topic with @Aklapper, it looks like a solution might be:

• A dedicated tag "Kiwix & openZIM"
• A dedicated workboard where we would be able to put the tickets we would like to see fixed/implemented in priority.

@Dzahn Thank you. I have been able to connect to the admin UI.

@Aklapper Happy you tackle that issue. I wanted to open that ticket myself for quite a long time already. You can put me in the list of admins.

I don't think the last fix deployment has fixed anything regarding that ticket. The Parsoid output is still broken so far I can see.

@bearND Thank you for the explanation

@fgiunchedi You are perfectly right. This seems to be an invalid bug. In fact I made the error because "File:Poaceae Habitus 2010-10-03 FuentedeSanLorenzoSierraMadrona.jpg" has been redirected from "File:Isoetes setaceum Habitus 2010-10-03 FuentedeSanLorenzoSierraMadrona.jpg". This is probably a consequence of T226931

@ssastry Thank you!

@Aklapper The whole offline/Kiwix stuff is handled by the new-readers team. That said, totally unsure if this is right to put it here. I would really like to have a tag or project to gather all openZIM/Kiwix impacting tickets. Would that be somehow possible?

Like T217540 is one good source of broken/outdated HTML (here with invalid images) delivered by Parsoid. This pretty annoyous for offline version of our wikis.

@ssastry I don't really understand why this ticket is closed. There is no agreement that this is a bug?

@mobrovac Thx, so I see that this concerns 307 web sites and for us, as around 20 are meta web sites that we don't want to scrape anyway, this is around 280 wikis we hardly can scrape.

BTW, it.wikibooks.org seems to be also impacted. Do we have a list somewhere of all the wikis where the mobile API endpoints are not "properly" available?

This problem stops us to provide a new offline version of Wikispecies.

@Ariel Sorry this was a problem on our end. Is fixed now.

What happens with qstat, is that still available?

@Audiodude thx

@bd808 Thx, I have successfully created the new WP1 instance.

Thx to all the participants!

@Bstorm As info, around 2 weeks ago we have introduced mwoffliner 1.8 on our VPS boxes. This new version should require between 3 and 5 times less IO activity than before. We should also not store a lot of data for longer than 1 month like before.

@bd808 I think a m1.xlarge might be enough for now, but not so sure how long it will last. Your analysis is good but you are missing an information. The jobs which need the most memory are not running on the grid. The datamining jobs were running on the pre-existing mwoffliner VPS boxes, but as we use these boxes more intensively than before, I want (or have) to stop to do so.

@GTirloni I wonder if these files are legacy after we migrated all our WM in December. If the files are older than 2019, then you can remove them I think.

@Krenair Yes, we fully understand that and this is our fault. We will have to deal with a few days downtime.

@Krenair Any chance to have a valid homepage? Even if the wiki is closed?

@akosiaris Problem seems to be fixed from our end. Thank you very much.

I'm not sure to fully understand the technical explanation. Is the problem confirmed? If "yes", what is the plan to solve it?

@Samwilson yes the qrpedia repo on Wikimedia source forge is the good one.

Here is how mine works now, id is the Twitter user:

<includeonly><a class="twitter-timeline" href="https://twitter.com/<!--{$id}-->?ref_src=twsrc%5Etfw"></a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="https://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script></includeonly>

@Reedy Thank you for your quick feedback. I was not aware that Mediawiki HHVM support will be deprecated. As a consequence, I have replaced HHVM with php-fpm and the problem disappeared.

Here a better example:

<quiz>
{ test }
</quiz>

All 5 boxes have been migrated to Debian Stretch

@Automactic I have added you to the project achttps://wikitech.wikimedia.org/wiki/Special:NovaProject. If you have configure your SSH key, I think you should have access to the VMs through SSH.

I have migrated one of the 5, the 4 others will follow.

@Krenair Thank you for the reminder as I was believing the deadline is somewhere in Spring 2019. Will do the necessary in November.

@Ramesh2744 Thank you very much for the information.

@Ramesh2744 Given a title (multimedia file) retrieve author/license (ideally title/description with corresponding language if available) in a machine readable format.

@daniel Thx, any clue about a specific task on Phabricator? Commons:Structured_data is a promising project but seems to be quite big and running already for quite a long time. It seems difficult to me to know (be informed) at which time the feature I need will be/has been implemented?

Detected using MWoffliner and reported here too https://github.com/openzim/mwoffliner/issues/185

Any news on this? (CC @atgo)

I have restarted the Web service manually and it is back.

@Krenair How is the procedure to wipeout the old ubuntu and get a 18.04? Can I do that on my own?

What does it means for me?
Should I do something myself?
Can we plan it around the end of the year?

@bd808 Thx, works fine now.

@bd808 Thx. Done at /home/kelson/T202877.

I do not have setup 2FA in the past and if Id it was inadvertently. I'm not interested in this level of security for wikitech. I for sure do not have complete any 2FA setup and do not have any scratch codes in my email box. I simply did not want this and still do not want this.

@Quiddity No, I use my Wikimedia/Wikipedia password. Whatever I use, I do not want that. How to deactivate the 2-factor authentication?

@Aklapper I have no token and if I give a random string it says "Verification failed".

@Jhernandez I understand that priorities need to be set but this is quite sad and a really concrete problem for offline projects. Parsoid output is the only one output which allows to properly re-render our textual content. Not providing a proper mobile version seems to me quite a problem considering how people access the content these days. For openZIM/Kiwix problem, it means higher maintenance costs (if possible at all) to try to maintain a mobile rendering similar as online. CC @atgo.

@Legoktm Thank you for your solution proposal. How would that behave if users want to download an image file? Would that mean that they get a file name with something like hash_value.jpg ?

I have open a ticket related to this here https://phabricator.wikimedia.org/T199140

After having migrated to nginx (from apache), it seems to work!?!?!

@Aklapper Yes, you can close the ticket, there is nothing more I can do on my side.

I just go an email with subject "482 Wiki-offline-reader-l moderator request(s) waiting". Please remove me from the list of owners (see my last comment).

@Dzahn @herron Could you please in addition remove my email address to the list of owner?

@MarcoAurelio I would be more comfortable if the messages were kept in an web available archive.

@bd808 perfect, will remember for next time. Thx.

@bd808 It works now, and if I understand right you have restarted the webservice on enwp10 side itself? Do you confirm? What is the exact command you have run?

I confirm the problem but I have almost no clue why it does not work. I do not have touched anything on the system recently. On the other side, I'm not sure how the stuff should work as I do not have built it myself.

Thank you all for the quick and adapted answer. It works well and we finally have been able to create the ZIM of ba.wikibooks.org successfully.

I have archive the openzim (related) projects on phabricator. So basically, what is needed on the mirror, is also to archive it there... whatever it means removal or close all tickets/PR (I have moved/merged them) and put the project in read-only.

My Github account is "kelson42"