Page MenuHomePhabricator

Wikimedia Portals repo has trouble running on Windows OS
Closed, ResolvedPublic

Description

Problem

The Wikimedia Portals repo is largely a Node application that produces static HTML/CSS/JS, so cross-OS compatibility shouldn't be an issue.
However, there are a few places where unix-centric shortcuts are used instead of employing the cross-OS Node functions
These include basically all usage of exec():

function garbageCollect() {
	exec( 'find cache -mtime +' + ( DAYS + 2 ) + ' -delete', function ( error ) {
		if ( error ) {
			console.error( 'Error deleting old cached stats', error ); // eslint-disable-line no-console
		}
	} );
}
if ( fs.existsSync( translationPath ) ) {
	exec( 'find ' + translationPath + ' -mindepth 1 -delete' );
} else {
	fs.mkdirSync( translationPath );
}
  • The double ampersands and backslashed paths in package.json may also be problematic.
	"scripts": {
		"start": "gulp update-stats && gulp watch --portal=wikipedia.org & npm run dev-server",
		"dev-server": "es-dev-server --app-index ./dev/wikipedia.org/index.html --preserve-symlinks=true --watch --open",
		"test": "gulp -- lint",
		"build-all-portals": "gulp -- lint && gulp -- update-stats && gulp --portal=all -- fetch-meta && gulp --portal=wikipedia.org -- default"
	},
  • /dev/wikipedia.org/portal is a symlink, which is a UNIX specific construct.

Event Timeline

@Jdrewniak , Have a look at this package node-cmd.
I was wondering if this could be useful in our case. This could help in running cli/bash style commands in windows.

I also read the official NodeJs documentation ( child_process_spawning_bat_and_cmd_files_on_windows ), which could be useful too.

As far as paths are concerned, I think path.resolve() could solve that.

Even though I am not using Windows but I'd still love to solve this issue.

Jdrewniak triaged this task as Medium priority.Mar 14 2021, 2:43 PM

@Tsiruot unfortunately the node-cmd package won't help much.

What we have to do is find the Node equivalent of some of the unix commands, like find. I don't mind using a helper library to help us out with that, like glob, but replacing a command like $> find ./cache -mtime +2 -delete is much more complicated, because what that does is find all the files in the /cache directory that have been modified in the last two days and deletes them. Using the Node File System module, we would have to first find all the files (let's say using glob), then loop through them, stat them, check their mtime, and then unlink them.

@Jdrewniak I tried to implement this and it is working fine. But the lines of code have increased significantly.

This will find all the files in /cache directory, options is an object for cwd
glob('*', options, findFiles)

This will find stats of all the files. Notice the resolvePath() for absolute path.

function findFiles(err, files) {
		if(err) console.log(err);
		else {
			files.forEach(el => {
				fs.stat(resolvePath(el) , findStats.bind(this, el));
			})
		}}

This will check number of days that have been passed since the file was last modified.

	function checkStats(stats) {
		var days = ((Date.now() - stats.mtime.getTime()) / (24*60*60*1000));
		if(days > 2) return true;
		else return false;
	}

This will delete the file from the directory.

	function deleteFile(el) {
		fs.unlink(resolvePath(el), (err) => {
			console.error( 'Error deleting old cached stats', err );
		});
	}

This might not be the best approach but just a rough implementation

Also, the resolvePath() function

	function resolvePath(file) {
		var p = path.resolve(__dirname, '.' , 'cache', file);
		return p;
	}

@Jdrewniak I tried to implement this and it is working fine. But the lines of code have increased significantly.

This will find all the files in /cache directory, options is an object for cwd
glob('*', options, findFiles)

This will find stats of all the files. Notice the resolvePath() for absolute path.

function findFiles(err, files) {
		if(err) console.log(err);
		else {
			files.forEach(el => {
				fs.stat(resolvePath(el) , findStats.bind(this, el));
			})
		}}

This will check number of days that have been passed since the file was last modified.

	function checkStats(stats) {
		var days = ((Date.now() - stats.mtime.getTime()) / (24*60*60*1000));
		if(days > 2) return true;
		else return false;
	}

This will delete the file from the directory.

	function deleteFile(el) {
		fs.unlink(resolvePath(el), (err) => {
			console.error( 'Error deleting old cached stats', err );
		});
	}

This might not be the best approach but just a rough implementation

Also, the resolvePath() function

	function resolvePath(file) {
		var p = path.resolve(__dirname, '.' , 'cache', file);
		return p;
	}

As i just notice this issue... as i am a window user i have also faced this issue.... and i can help you to reslove this issue....

This comment was removed by Tsiruot.

Change 683481 had a related patch set uploaded (by Ishan Saini; author: Ishan Saini):

[wikimedia/portals@master] Remove exec() and add deleteFiles() function

https://gerrit.wikimedia.org/r/683481

Change 683481 merged by jenkins-bot:

[wikimedia/portals@master] Remove exec() and add deleteFiles() function

https://gerrit.wikimedia.org/r/683481