Change 250878 had a related patch set uploaded (by Hoo man):
Use Zend to create the DCAT-AP RDF on snapshot1003

https://gerrit.wikimedia.org/r/250878

Change 250878 merged by Dzahn:
Use Zend to create the DCAT-AP RDF on snapshot1003

https://gerrit.wikimedia.org/r/250878

The cause here is that the php command switched from Zend PHP 5.3 to HHVM when snapshot1003 got upgraded. I don't know whether this is a bug in your code or in HHVM or something else. Please note that we don't run the latest HHVM.

For reference:

hoo@snapshot1003:~$ php5 --version
PHP 5.5.9-1ubuntu4.13 (cli) (built: Sep 29 2015 15:24:49) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies

hoo@snapshot1003:~$ hhvm --version
HipHop VM 3.6.5 (rel)
Compiler: 1442470674_297057398
Repo schema: 599829f574c631684b463e9fe5f9093266631b9c
Extension API: 20150212

I manually recreated the RDF using Zend php now and changed the binary being used to Zend php. I don't know if there's more to do here (like fixing an underlying problem in the script) or whether we can just close this.

I think this issue with HHVM was fixed in this diff

In T117534#1781061, @Lokal_Profil wrote:

I think this issue with HHVM was fixed in this diff

But that patch is part of HHVM 3.6.5 (which we use), so I don't think that's it.

A bit more digging produces https://github.com/facebook/hhvm/issues/4336 which I found through the unmerged patch at https://reviews.facebook.net/D29529

What ever happened here? Is this still an issue?

Just tested it and it still is.

For reference, tested dcat 92ab37d94edefc36107eceb2197252dd9825af8b with:

hoo@snapshot1003:~$ hhvm --version
HipHop VM 3.12.1 (rel)
Compiler: 3.12.1+dfsg-1
Repo schema: 8aa242a1c26f122330b9188732cdd84948fd7706

Tested, per @ArielGlenn's request:

hoo@mwdebug1001:~/dcat$ hhvm --version
HipHop VM 3.18.1 (rel)
Compiler: 3.18.1+dfsg-1+wmf1
Repo schema: 47e86d9e41dac7800783dc589035875b00f7231c

The output still contains various xmlns:rdf="", like <dcterms:license rdf:resource="http://creativecommons.org/publicdomain/zero/1.0/" xmlns:rdf=""/>.

This could probably be worked around as noted in the comment on https://github.com/facebook/hhvm/issues/4336 (look for Workaround).

Seems to be fixed, at least in the current file.

In T117534#3709102, @nichtich wrote:

Seems to be fixed, at least in the current file.

IIRC we now generate the dcat files using a special setup which avoids hhvm. This task is about fixing the underlying issue so that that special fix isn't needed.

@ArielGlenn @hoo As this is an effect of hhvm would converting the script to python be a viable solution or is PHP a must for it to fit with other dump related mechanisms? (Haven't checked if it is viable codewise but asking to see what options there are).

It makes no difference to my setup if you hand me a python script or a php one. @hoo may have something to say about it however.

In T117534#3749216, @Lokal_Profil wrote:

@ArielGlenn @hoo As this is an effect of hhvm would converting the script to python be a viable solution or is PHP a must for it to fit with other dump related mechanisms? (Haven't checked if it is viable codewise but asking to see what options there are).

I'm not as proficient in Python (compared to PHP) and Python is not used very widely within Wikimedia (operations/dumps uses it though). If this makes maintaining/ developing the tool easier, I don't see a strong reason not to go for Python.

The script is not interfering with MediaWiki internally, so it's fine for it to not be in PHP.

Note: When rewriting this, we might also want to use the opportunity to make it customizable so that it works for all of Wikimedia in the end.

Creating RDF/XML by hand is bad practice anyway, no matter in which programming language. The current script could be modified to use Purtle like other Wikimedia software, this would also solve the bug.

Change 391489 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[operations/dumps/dcat@master] Work around HHVM bug by using XMLWriter::writeAttribute()

https://gerrit.wikimedia.org/r/391489

Change 391489 merged by Hoo man:
[operations/dumps/dcat@master] Work around HHVM bug by using XMLWriter::writeAttribute()

https://gerrit.wikimedia.org/r/391489

Change 391808 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Snapshot: Use canonical php (= hhvm) for dcat

https://gerrit.wikimedia.org/r/391808

Change 391808 merged by ArielGlenn:
[operations/puppet@production] Snapshot: Use canonical php (= hhvm) for dcat

https://gerrit.wikimedia.org/r/391808

This was reverted back to using php5 in https://phabricator.wikimedia.org/T182348

The parent task has been Declined; is it worth keeping this open given that we're moving away from HHVM to Zend PHP 7?

I am not sure if I am simply rephrasing what @Jdforrester-WMF asked above, but any way: Are we positive this issue would go away if we updated box the command runs on to PHP 7?

meh. I meant, if we moved the task to run on PHP 7 machine, that would mean we could still get correct XML, right? I.e. it does not require PHP 5?

In T117534#3892964, @WMDE-leszek wrote:

meh. I meant, if we moved the task to run on PHP 7 machine, that would mean we could still get correct XML, right? I.e. it does not require PHP 5?

That should be, but I didn't check that, yet.

In T117534#3894564, @hoo wrote:

In T117534#3892964, @WMDE-leszek wrote:

meh. I meant, if we moved the task to run on PHP 7 machine, that would mean we could still get correct XML, right? I.e. it does not require PHP 5?

That should be, but I didn't check that, yet.

Have you had a chance yet? Right now this is the only open task underneath T86081; if it's an on-going issue, should we stash it under T172165 too? That's coming up very quickly…

In T117534#4067258, @Jdforrester-WMF wrote:

In T117534#3894564, @hoo wrote:

In T117534#3892964, @WMDE-leszek wrote:

meh. I meant, if we moved the task to run on PHP 7 machine, that would mean we could still get correct XML, right? I.e. it does not require PHP 5?

That should be, but I didn't check that, yet.

Have you had a chance yet? Right now this is the only open task underneath T86081; if it's an on-going issue, should we stash it under T172165 too? That's coming up very quickly…

This is going to come up relatively soon, as we are going to move the snapshot hosts (and thus this job) to PHP7 this coming quarter.

$ cat test.sh 
#!/bin/bash

tmpDir=`mktemp -d`
touch $tmpDir/foo.json.gz
touch $tmpDir/bar.ttl.gz
touch $tmpDir/bar.ttl.bz2

php DCAT.php --config=./config.example.json --dumpDir="$tmpDir" --outputDir=/tmp &&
rm -rf $tmpDir &&
echo "Find the output at /tmp/dcatap.rdf" &&
exit 0

rm -rf $tmpDir
echo Error

Identical output with PHP 5.5.9-1ubuntu4.24 and PHP 7.1.15 (cli):

$ bash test.sh 
Find the output at /tmp/dcatap.rdf
$ md5sum /tmp/dcatap.rdf 
f488c991f5a7dfd58f05ac2fc48c3706  /tmp/dcatap.rdf

@ArielGlenn Do we want to conclude this?

As far as I'm concerned we can 'decline' it since we'll be moving to php7 for dumps this quarter.