Toolforge: figure out how to work with the new domain in the grid
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Mar 23 2021, 12:42 PM

Description

The Toolforge grid migration from Stretch to Buster implies we are migrating away from the tools.eqiad.wmflabs domain into tools.eqiad1.wikimedia.cloud. In toolsbeta, the whole grid uses toolsbeta.eqiad1.wikimedia.cloud already.

CURRENT STATUS AS OF TASK FILLING:

grid master == stretch
grid shadow == buster

Some issues detected:

the hiera key that stores the master will need refresh when the grid master is migrated to buster, for example:

-sonofgridengine::gridmaster: tools-sgegrid-master.tools.eqiad.wmflabs
+sonofgridengine::gridmaster: tools-sgegrid-master.tools.eqiad1.wikimedia.cloud

the grid master is somehow encoding the domain in the configuration:

aborrero@tools-sgegrid-master:~$ sudo qconf -ss | grep sgegrid
tools-sgegrid-master.tools.eqiad.wmflabs
tools-sgegrid-shadow.tools.eqiad.wmflabs
aborrero@tools-sgegrid-master:~$ sudo qconf -dh tools-sgegrid-shadow.tools.eqiad.wmflabs
can't resolve hostname "tools-sgegrid-shadow.tools.eqiad.wmflabs"
aborrero@tools-sgegrid-master:~$ sudo qconf -ah tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud
tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud added to administrative host list
aborrero@tools-sgegrid-master:~$ sudo qconf -ss | grep sgegrid
tools-sgegrid-master.tools.eqiad.wmflabs
tools-sgegrid-shadow.tools.eqiad.wmflabs

aborrero@tools-sgegrid-shadow:~$ qstat -f
error: commlib error: access denied (server host resolves rdata host "tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud" as "tools-sgegrid-shadow.tools.eqiad.wmflabs")
error: unable to contact qmaster using port 6444 on host "tools-sgegrid-master.tools.eqiad.wmflabs"

I've detected a few places where this might be hardcoded:

aborrero@tools-sgegrid-master:/var/lib/gridengine$ cat default/common/shadow_masters
tools-sgegrid-master.tools.eqiad.wmflabs
tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud
tools-sgegrid-shadow.tools.eqiad.wmflabs

the master daemon insists on the old domain:

Mar 23 12:46:07 tools-sgegrid-master sge_qmaster[3764]: can't resolve host name "tools-sgegrid-shadow.tools.eqiad.wmflabs": undefined commlib error code
Mar 23 12:46:07 tools-sgegrid-master sge_qmaster[3764]: can't resolve host name "tools-sgegrid-shadow.tools.eqiad.wmflabs": undefined commlib error code

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
		Restricted Task
Resolved	Bstorm	T169289 Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues
Resolved	MoritzMuehlenhoff	T169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS
Resolved	Bstorm	T203254 labstore1004 and labstore1005 high load issues following upgrades
Open	None	T257945 NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers
Resolved	Andrew	T289883 cloud-vps: actively deprecate/phase out use of Debian Stretch
Resolved	taavi	T306099 Cloud VPS "tools" project Stretch deprecation
Resolved	taavi	T306100 Cloud VPS "toolsbeta" project Stretch deprecation
Resolved	taavi	T275864 Toolforge: migrate to Debian Buster or later
Stalled	None	T249949 Stop using integration/composer and then archive the repo
In Progress	taavi	T287900 Switch Toolforge installation of "composer" to use the Debian package
Stalled	None	T305834 Cloud VPS: drop wmflabs names from profile::resolving::domain_search
Resolved	aborrero	T277653 Toolforge: add Debian Buster to the grid and eliminate Debian Stretch
Resolved	aborrero	T278232 Toolforge: figure out how to work with the new domain in the grid

Event Timeline

aborrero created this task.Mar 23 2021, 12:42 PM

aborrero updated the task description. (Show Details)Mar 23 2021, 12:47 PM

This apparently fixed itself, but some weird things remain:

aborrero@tools-sgegrid-shadow:~$ qstat
error: commlib error: access denied (server host resolves rdata host "tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud" as "tools-sgegrid-shadow.tools.eqiad.wmflabs")
error: unable to contact qmaster using port 6444 on host "tools-sgegrid-master.tools.eqiad.wmflabs"

aborrero@tools-sgegrid-master:~$ sudo systemctl restart gridengine-master.service
[..]
Mar 23 12:46:07 tools-sgegrid-master sge_qmaster[3764]: can't resolve host name "tools-sgegrid-shadow.tools.eqiad.wmflabs": undefined commlib error code
Mar 23 12:46:07 tools-sgegrid-master sge_qmaster[3764]: can't resolve host name "tools-sgegrid-shadow.tools.eqiad.wmflabs": undefined commlib error code
[..]

aborrero@tools-sgegrid-shadow:~$ qstat
error: denied: host "tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud" is neither submit nor admin host

aborrero@tools-sgegrid-master:~$ sudo qconf -ah tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud
tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud added to administrative host list

aborrero@tools-sgegrid-shadow:~$ qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
task@tools-sgeexec-0901.tools. BI    0/8/50         0.75     lx-amd64      
---------------------------------------------------------------------------------

aborrero@tools-sgegrid-master:$ qconf -ss | grep sgegrid
tools-sgegrid-master.tools.eqiad.wmflabs
tools-sgegrid-shadow.tools.eqiad.wmflabs

aborrero updated the task description. (Show Details)Mar 23 2021, 12:52 PM

RhinosF1 added a subscriber: RhinosF1.Mar 25 2021, 4:33 PM

In both tools and toolsbeta projects we have now buster master/shadow servers. They are working fine.

We don't plan on introducing new domains in the foreseeable future. So I think this task can be closed, even though no concrete solution was found, we don't expect to find this problem ever again.

Toolforge: figure out how to work with the new domain in the gridClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Toolforge: figure out how to work with the new domain in the grid
Closed, ResolvedPublic
Actions

Related Objects
Search...