Page MenuHomePhabricator

missing maps postgres passwords in clouddb-services
Closed, ResolvedPublic

Description

puppet onclouddb1003.clouddb-services.eqiad1.wikimedia.cloud fails like this:

Function lookup() did not find a value for the name 'profile::wmcs::services::postgres::osm_password'

There are four allied passwords in the manifest; probably all four of them were wiped by T254491

$osm_password = hiera('profile::wmcs::services::postgres::osm_password'),
$kolossos_password = hiera('profile::wmcs::services::postgres::kolossos_password'),
$aude_password = hiera('profile::wmcs::services::postgres::aude_password'),
$planemad_password = hiera('profile::wmcs::services::postgres::planemad_password'),

I'm not clear on if anyone is currently using those passwords or if they can be rotated.

Event Timeline

Andrew created this task.Jun 9 2020, 7:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 9 2020, 7:07 PM
Andrew updated the task description. (Show Details)Jun 9 2020, 7:08 PM

@Andrew: Can you please associate at least one active project with this task, so others can find it? Thanks.

Bstorm added a subscriber: Bstorm.Jun 9 2020, 10:05 PM

$kolossos_password = hiera('profile::wmcs::services::postgres::kolossos_password'),
$aude_password = hiera('profile::wmcs::services::postgres::aude_password')
These two are read-write accounts for both Toolforge tools and maps things. They also represent unique data on the server. I think I can find them. I'll check the others.

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T22:55:27Z] <bstorm_> Reset the passwords for T254931

bd808 renamed this task from missing postgres passwords on to missing postgres passwords in clouddb-services.Jun 9 2020, 11:02 PM
bd808 added a project: Cloud-VPS.
bd808 added a project: Data-Services.
bd808 moved this task from Backlog to Maps on the Data-Services board.
Bstorm renamed this task from missing postgres passwords in clouddb-services to missing maps postgres passwords in clouddb-services.Jun 9 2020, 11:24 PM
Bstorm triaged this task as Unbreak Now! priority.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 9 2020, 11:24 PM
Bstorm closed this task as Resolved.Jun 9 2020, 11:25 PM
Bstorm claimed this task.
Bstorm added a comment.EditedJun 9 2020, 11:29 PM

This would have also fixed the Wikilabels db server's puppet.

jbond added a subscriber: jbond.Jun 10 2020, 9:28 AM

@Bstorm thanks for fixing this however could you let me know what you did as i notice private/labs is still missing from labtestpuppetmaster2001 and I'm unsure if its still needed there?

It would also be useful to know in general what labtestpuppetmaster is still used for?

thanks

@Bstorm thanks for fixing this however could you let me know what you did as i notice private/labs is still missing from labtestpuppetmaster2001 and I'm unsure if its still needed there?

It would also be useful to know in general what labtestpuppetmaster is still used for?

Let me chime in.

In past times, the general puppetmaster for cloud VMs (in project that dodn't have a local puppetmaster) was an actual hardware server. This got eventually moved to VMs inside the cloud (puppetmasters in https://openstack-browser.toolforge.org/project/cloudinfra). This is for our eqiad1 openstack deployment. We have another openstack deployment in codfw, called codfw1dev. More info here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments

We are in the process of migrating this actual hardware server puppetmaster (labtestpuppetmaster2001) to virtual machines inside the codfw1dev deployment itself, see T242607: Create in-cloud puppetmaster for codfw1dev