Page MenuHomePhabricator
Create Task
Maniphest T224652

Setup the MLEB instance as a test instance for the Translate extension
Closed, ResolvedPublicJul 3 2019
Actions

Description

As discussed, please setup the MLEB instance to pull and deploy the latest code from the Translate extension.

Details

Due Date
Jul 3 2019, 12:30 PM

Related Objects
Search...

Event Timeline

abi_ created this task.May 30 2019, 8:03 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 30 2019, 8:03 AM
KartikMistry added a comment.May 30 2019, 8:07 AM

Do we need specific branch or code should be on master? Also, please add config we need for Translate (LocalSettings.php)

KartikMistry added a project: Language-Team (Language-2019-April-June).May 30 2019, 8:08 AM
KartikMistry moved this task from Backlog to In Progress on the Language-Team (Language-2019-April-June) board.Jun 13 2019, 8:25 AM
abi_ added a comment.EditedJun 16 2019, 12:40 PM

Hi Kartik,

The branch to deploy is the master branch. You'll also need to deploy the ULS extension.

Use the following configuration under LocalSettings.php,

// Groups
$wgGroupPermissions['translator']['translate'] = true;
$wgGroupPermissions['translator']['skipcaptcha'] = true; // Bug 34182: needed with ConfirmEdit

$wgGroupPermissions['translate-admin']['translate'] = true;
$wgGroupPermissions['translate-admin']['pagetranslation'] = true;
$wgGroupPermissions['translate-admin']['translate-manage'] = true;
$wgGroupPermissions['translate-admin']['translate-messagereview'] = true;
$wgGroupPermissions['translate-admin']['skipcaptcha'] = true;

$wgTranslateDocumentationLanguageCode = 'qqq';

// Workflow states
$wgTranslateWorkflowStates = [
	'unset' => [ 'color' => 'FF0000' ], // red
	'proofreading' => [ 'color' => 'd7f7f3' ], // light blue
	'ready' => [ 'color' => '00FF00' ], // green
	'inprogress' => [
		'color' => 'FFFF00', // yellow
	],
	'state conditions' => [
		[ 'ready', [ 'PROOFREAD' => 'MAX' ] ],
		[ 'proofreading', [ 'TRANSLATED' => 'MAX' ] ],
		[ 'unset', [ 'UNTRANSLATED' => 'MAX', 'OUTDATED' => 'ZERO', 'TRANSLATED' => 'ZERO' ] ],
		[ 'inprogress', [ 'UNTRANSLATED' => 'NONZERO' ] ],
	]
];

We'll need the following users,

  1. One user with the group - translator
  2. One user with the group - translate-admin.

Setup two skins - Timeless & Vector.

Ensure caching is setup.

If possible, setting up the TTM Server with the ElasticSearch backend would also be good. This is documented here. We can then add the following configuration under LocalSettings.php,

$wgTranslateTranslationServices['TTMServer'] = array(
	'type' => 'ttmserver',
	'class' => 'ElasticSearchTTMServer',
	'cutoff' => 0.75,
);

Please let me know if there is anything I can help you with.

KartikMistry added a comment.Jun 18 2019, 8:13 AM

Instance is setup at https://language-translate.wmflabs.org

Pending tasks:

  • Caching.
  • TTM Server/ElasticSearch
abi_ triaged this task as Medium priority.Jun 19 2019, 8:26 AM
abi_ set Due Date to Jul 2 2019, 6:30 PM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 19 2019, 8:26 AM
abi_ added a comment.Jun 19 2019, 12:30 PM

For caching, plan is to setup Redis and then,

// Caching
$wgObjectCaches['redis'] = array(
    'class' => 'RedisBagOStuff',
    'servers' => array( '127.0.0.1:6379' ),
    'persistent' => true,
);
$wgMainCacheType = CACHE_ANYTHING;

// This is equivalent to redis_local in production, since MediaWiki-Vagrant
// only has one data center.
$wgMainStash = 'redis';

// Avoid user request serialization and other slowness
$wgSessionCacheType = 'redis';
$wgSessionsInObjectCache = true;

// Jobqueue
$wgJobTypeConf['default'] = array(
  'class'       => 'JobQueueRedis',
  'daemonized'  => true,
  'redisServer' => '127.0.0.1',
  'redisConfig' => array( 'connectTimeout' => 2, 'compression' => 'gzip' ),
);

$wgJobQueueAggregator = array(
  'class'        => 'JobQueueAggregatorRedis',
  'redisServers' => array( '127.0.0.1' ),
  'redisConfig'  => array( 'connectTimeout' => 2 ),
);

The job runner service configuration can be taken from Mediawiki Vagrant here.

KartikMistry updated the task description. (Show Details)Jun 27 2019, 7:42 AM
KartikMistry updated the task description. (Show Details)
KartikMistry updated the task description. (Show Details)Jun 27 2019, 8:22 AM
KartikMistry updated the task description. (Show Details)
KartikMistry updated the task description. (Show Details)Jun 27 2019, 8:24 AM
KartikMistry changed Due Date from Jul 2 2019, 6:30 PM to Jul 3 2019, 12:30 PM.
Pginer-WMF edited projects, added Language-Team (Language-2019-July-September); removed Language-Team (Language-2019-April-June).Jul 1 2019, 12:32 PM
Pginer-WMF moved this task from Backlog to In Progress on the Language-Team (Language-2019-July-September) board.
KartikMistry updated the task description. (Show Details)Jul 3 2019, 6:56 AM
KartikMistry added a comment.Jul 3 2019, 7:02 AM

I installed elasticsearch (6.5.2) from upstream Debian repository (Version as per: https://www.mediawiki.org/wiki/Extension:CirrusSearch#Dependencies) and service doesn't start.

# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-07-03 07:02:02 UTC; 2s ago
     Docs: http://www.elastic.co
  Process: 22177 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 22177 (code=exited, status=1/FAILURE)

Jul 03 07:01:58 language-translate systemd[1]: Started Elasticsearch.
Jul 03 07:02:02 language-translate systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 03 07:02:02 language-translate systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 03 07:02:02 language-translate systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
KartikMistry added a comment.Jul 3 2019, 7:20 AM

From /var/log/elasticsearch/elasticsearch.log:

[2019-07-03T07:17:15,209][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:4, file:/var/lib/elasticsearch/nodes/0/_state/node-4.st]]; nested: IOException[failed to read [id:4,
file:/var/lib/elasticsearch/nodes/0/_state/node-4.st]]; nested: XContentParseException[[-1:36] [node_meta_data] unknown field [node_version], parser not found];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:4, file:/var/lib/elasticsearch/nodes/0/_state/node-4.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:394) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:302) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.2.jar:6.5.2]
        ... 6 more
Caused by: java.io.IOException: failed to read [id:4, file:/var/lib/elasticsearch/nodes/0/_state/node-4.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:394) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:302) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.2.jar:6.5.2]
        ... 6 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not found
        at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:347) ~[elasticsearch-x-content-6.5.2.jar:6.5.2]
        at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.5.2.jar:6.5.2]
        at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:394) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:302) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.2.jar:6.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.2.jar:6.5.2]
        ... 6 more
KartikMistry added a comment.Jul 3 2019, 7:46 AM

elasticsearch is up and running now.

KartikMistry moved this task from In Progress to In Review on the Language-Team (Language-2019-July-September) board.Jul 3 2019, 7:49 AM
Nikerabbit added a comment.Jul 3 2019, 12:10 PM

The version-incompatible indexes were wiped to make ElasticSearch to start up.

KartikMistry updated the task description. (Show Details)Jul 8 2019, 2:30 PM
abi_ added a comment.EditedJul 8 2019, 6:26 PM

Hi @KartikMistry,

Thanks for setting up this instance. Everything looks good and seems to be working well.

Tested

  • Saw that the latest code was deployed on the instance
  • Checked other plugins via Special:Version page
  • Logged in as translator and was able to create page with translation tags.
  • Logged in as translate-admin and was able to mark page for translation.
  • Was able to translate strings as translate-admin and translator.
  • Message group state is updating properly.
  • Was able to toggle themes between Vector and Timeless.
  • Was able to search for translations on the Search translations page.
  • Was able to use the Page Translation special page.
  • Was able to update the credentials for both the users.

TODO

It would be good if the following could be done,

  • Need to install / enable TwnMainPage extension
  • An admin user, for example to change the links on the sidebar.
  • SSH Access for Niklas and me.

Current users

  1. translate-admin
  2. translator

Please get in touch with @abi_ to get the credentials for those users.

KartikMistry added a comment.EditedJul 9 2019, 4:45 AM
  • An admin user, for example to change the links on the sidebar.

admin user is there. I'll send credential.

  • SSH Access for Niklas and me.

It is Lab access, anyone from Language team can ssh to it. eg ssh language-translate.eqiad.wmflabs

KartikMistry added a comment.Jul 9 2019, 4:56 AM
  • Need to install / enable TwnMainPage extension

Done. Let me know if anything specific is needed for this setup.

abi_ added a comment.Jul 9 2019, 4:59 AM
In T224652#5316001, @KartikMistry wrote:
  • Need to install / enable TwnMainPage extension

Done. Let me know if anything specific is needed for this setup.

Yes, I should have mentioned this initially,

$wgTranslateUseSandbox = true;
$wgGroupPermissions['translate-admin']['translate-sandboxmanage'] = true;  // Add this role to the existing group
KartikMistry added a comment.Jul 9 2019, 5:07 AM
In T224652#5316002, @abi_ wrote:
$wgTranslateUseSandbox = true;
$wgGroupPermissions['translate-admin']['translate-sandboxmanage'] = true;  // Add this role to the existing group

Done!

abi_ added a comment.Jul 9 2019, 5:17 AM

Thanks, this is now mostly done. But I'm getting an exception when trying to register using https://language-translate.wmflabs.org/index.php/Special:MainPage

It looks like a permission issue, I'll investigate some more and get back to you.

abi_ added a subtask: T227637: MLEB instance requires createaccount rights for all users for TwnMainPage extension.Jul 10 2019, 7:29 AM
abi_ added a comment.Jul 10 2019, 7:37 AM

I spent some time fixing the related issue but was not very satisfied with the solution I came up with. I've created a subtask, and investigate the issue further later.

@KartikMistry, for now we can,

  1. Enable createaccount right for all
$wgGroupPermissions['*']['createaccount'] = true;
  1. Setup password authentication on Apache. We can create a single user and the language team can use that for testing.
KartikMistry added a comment.Jul 11 2019, 5:27 AM
In T224652#5320208, @abi_ wrote:
  1. Enable createaccount right for all
  2. Setup password authentication on Apache. We can create a single user and the language team can use that for testing.

Done!

abi_ closed this task as Resolved.Jul 11 2019, 2:40 PM

Information regarding MLEB testing instance: https://www.mediawiki.org/wiki/User:APatro_(WMF)/MLEB_Testing_Instance

This is working well for now, so resolving this task.

Thanks @KartikMistry

• Petar.petkovic moved this task from In Review to Done on the Language-Team (Language-2019-July-September) board.Jul 12 2019, 3:33 PM
abi_ mentioned this in T228830: Update MLEB Test instance to be similar to the Translate instance.Jul 24 2019, 7:23 AM
abi_ closed subtask T227637: MLEB instance requires createaccount rights for all users for TwnMainPage extension as Resolved.Feb 5 2020, 8:27 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits