Page MenuHomePhabricator

Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats
Closed, ResolvedPublic

Event Timeline

Xqt created this task.May 20 2017, 1:17 PM
Restricted Application added a project: Analytics. · View Herald TranscriptMay 20 2017, 1:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Hmm, i get this

[Sat May 20 14:40:43.698258 2017] [:error] [pid 24237] [client 10.68.21.68:59064] PHP Fatal error: Uncaught Error: Call to undefined function data_dumper() in /var/www/wikistats/api.php:51\nStack trace:\n#0 {main}\n thrown in /var/www/wikistats/api.php on line 51

in the logs.

Change 354709 had a related patch set (by Paladox) published:
[operations/debs/wikistats@master] fix undefined data_dumper

https://gerrit.wikimedia.org/r/354709

Just for the record, *this* Wikistats bears no relation to *the* Wikistats, which is stats.wikimedia.org. I just found out about this new project, but it is indeed not maintained as far as I know. I'm interested to know who built it, what the purpose was, and if there are any users. It's pretty confusing, because we're building what we're calling Wikistats 2.0, so there's a name clash.

Dzahn claimed this task.May 20 2017, 4:31 PM
Dzahn removed subscribers: Milimetric, Xqt.
Dzahn added a comment.May 20 2017, 4:36 PM

A couple things:

  • this has nothing to do with Analytics wikistats, please do not add those tags and people
  • it is not unmaintained, i maintain it
  • it is not new, it exists since a loooong time (yes, aware of the name clash but that is called wikistats since before it even moved to WMF infra years ago and that's why it has a different tag)
  • it was deliberate to deactivate this feature and i'm not going to activate it again by simply uncommenting it, it was insecure
  • i will come up with something better to replace it but that's not high prio for me, so some patience please
  • i am surprised (mostly in a good way) that this was noticed so quickly, which shows me it's actually used, something that i was wondering about

A couple things:

  • this has nothing to do with Analytics wikistats, please do not add those tags and people
  • it is not unmaintained, i maintain it
  • it is not new, it exists since a loooong time (yes, aware of the name clash but that is called wikistats since before it even moved to WMF infra years ago and that's why it has a different tag)
  • it was deliberate to deactivate this feature and i'm not going to activate it again by simply uncommenting it, it was insecure
  • i will come up with something better to replace it but that's not high prio for me, so some patience please
  • i am surprised (mostly in a good way) that this was noticed so quickly, which shows me it's actually used, something that i was wondering about

Oh, unsecure? what about using mysqli functions instead of mysql functions?

Or switching to pdo.

Dzahn triaged this task as Normal priority.May 20 2017, 4:39 PM
Dzahn added a comment.May 20 2017, 4:43 PM

Oh, unsecure? what about using mysqli functions instead of mysql functions?
Or switching to pdo.

Ehm, you linked my own change above and if you'd look at the title of that it's "remove all mysql_ functions, use PDO". That is data_dumper.

xml_dumper is deactivated because it passes through a shell command. "$command="mysql -X "... I will get to it but not on the weekend.

Ah yeh. I knew that xml_dumper could have possibly been disabled due to it executing commands. :)

@Dzahn, the pywikibot framework uses the xml files and the CI Travis tests where broken since few days and our maintenance script doesn't work due to this problem. That's the reason I reported it.

Change 354709 abandoned by Paladox:
fix undefined data_dumper

https://gerrit.wikimedia.org/r/354709

Dzahn added a subscriber: Xqt.May 21 2017, 12:02 AM

@Xqt Sorry as well, not your fault since it's easy to confuse them. I appreciate you reported it and it's good to know it is actually used, i'll work on it soon. In the future just use the "Labs-project-Wikistats" tag and also feel free to directly subscribe me or assign. Thanks.

Change 355736 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/debs/wikistats@master] add new csv_dumper function, remove old code for xml_dump

https://gerrit.wikimedia.org/r/355736

Change 355736 merged by jenkins-bot:
[operations/debs/wikistats@master] add new csv_dumper function, remove old code for xml_dump

https://gerrit.wikimedia.org/r/355736

Dzahn added a comment.May 26 2017, 1:57 AM

Hi @Xqt CSV format works again, using new code, but sticking to the same format, outputs all columns it can find.

Now using PDO / getColumnMeta and fputcsv.

The "SSV" format is deactivated because _just_ for a different delimiter another format seems is not really needed, unless you disagree.

About XML i'll see separately in a bit.

Change 355740 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/debs/wikistats@master] go back to data_dumper, add ssv format back

https://gerrit.wikimedia.org/r/355740

Change 355740 merged by Dzahn:
[operations/debs/wikistats@master] go back to data_dumper, add ssv format back

https://gerrit.wikimedia.org/r/355740

Dzahn added a comment.May 26 2017, 3:06 AM

Hi again, changed my mind about the SSV thing, that also works again now (URLs as before) after additional change above.

Xqt added a comment.May 26 2017, 5:56 AM

I only use xml format for the bots. I'll wait a bit. Thanks a lot.

Change 355742 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikistats: install php7.0-xml if on stretch

https://gerrit.wikimedia.org/r/355742

Dzahn added a comment.May 26 2017, 7:42 PM

@Xqt I expected you to use the CSV for some reason. Didn't expect XML, does this mean you need it to be the exact same format as before? Actually the old XML format from mysql seems not that great and now i would create it from PHP and the tags would change/more meaningful than before.

Change 355742 merged by Dzahn:
[operations/puppet@production] wikistats: install php7.0-xml if on stretch

https://gerrit.wikimedia.org/r/355742

Xqt added a comment.May 28 2017, 7:15 AM

@Dzahn: I'll check our wikistats library. I think there is a csv interface too. Maybe I get it activated.

Change 355743 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/debs/wikistats@master] (WIP) - re-add XML output

https://gerrit.wikimedia.org/r/355743

Change 358150 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikistats: cron for XML dumps (WIP)

https://gerrit.wikimedia.org/r/358150

@Xqt I will reactivate XML soon, i'll just run the mysql -X command as before but let it happen like once per hour in a cron. That will be easier and you will have the same format as before.

Change 358150 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikistats: add cron jobs for XML dumps

https://gerrit.wikimedia.org/r/358150

Change 358150 merged by Dzahn:
[operations/puppet@production] wikistats: add cron jobs for XML dumps

https://gerrit.wikimedia.org/r/358150

Change 359640 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/debs/wikistats@master] re-activate XML dump links, to new target in ./xml/

https://gerrit.wikimedia.org/r/359640

Change 355743 abandoned by Dzahn:
(WIP) - re-add XML output

https://gerrit.wikimedia.org/r/355743

Change 359640 merged by Dzahn:
[operations/debs/wikistats@master] re-activate XML dump links, to new target in ./xml/

https://gerrit.wikimedia.org/r/359640

Change 359642 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikistats: disable cron for wikia - dump file too large (for now)

https://gerrit.wikimedia.org/r/359642

Change 359642 merged by Dzahn:
[operations/puppet@production] wikistats: disable cron for wikia - dump file too large (for now)

https://gerrit.wikimedia.org/r/359642

Dzahn added a comment.Jun 17 2017, 2:47 AM

@Xqt The XML links on the index page work again, the content should be exactly like you were used to it before the interruption.

The URLs changed though, example: http://wikistats.wmflabs.org/xml/wikipedias.xml

(instead of that old api.php link ending in format=xml)

I started working on redirects for the old URL but not done yet... should be like this: ?

RewriteCond %{QUERY_STRING} table=(\w)&format=xml$
RewriteRule api.php ./xml/%1.xml? [R=301,L]

(only one table is missing for now, Wikia, it's just too large, needs compression)

Xqt closed this task as Resolved.Jun 17 2017, 10:20 AM

@Dzahn: I changed the url for the maintenance script in https://gerrit.wikimedia.org/r/#/c/359653/ and this should be enough form me. This task is solved now for me. Many thanks investigating into this matter.

Dzahn added a comment.Jun 20 2017, 7:56 PM

Nice, i see the subtask is resolved too. cool! ( i should still do the rewrites when i get to it)

Also, Xqt i like your avatar.