Volans (Riccardo Coccioli)
Operations Software Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Feb 10 2016, 11:25 AM (147 w, 6 d)
Availability
Available
IRC Nick
volans
LDAP User
Volans
MediaWiki User
RCoccioli (WMF) [ Global Accounts ]

Recent Activity

Yesterday

Volans updated the task description for T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.
Mon, Dec 10, 7:42 PM · Patch-For-Review, Operations, Operations-Software-Development
Volans added projects to T182028: DNS repo: add CI checks for obvious configuration errors: DNS, Traffic.
Mon, Dec 10, 1:07 PM · Traffic, DNS, Patch-For-Review, Operations-Software-Development, Operations

Sun, Dec 9

Volans edited P7896 DNS zone validator output example.
Sun, Dec 9, 11:18 AM

Sat, Dec 8

CDanis awarded T182028: DNS repo: add CI checks for obvious configuration errors a Love token.
Sat, Dec 8, 4:51 PM · Traffic, DNS, Patch-For-Review, Operations-Software-Development, Operations
Volans edited P7896 DNS zone validator output example.
Sat, Dec 8, 3:42 PM
Volans created P7896 DNS zone validator output example.
Sat, Dec 8, 3:42 PM

Fri, Dec 7

Volans added a comment to P5608 Update production known hosts.

@Volans should this really be world-editable? 🤔

Fri, Dec 7, 10:07 PM
Volans changed the edit policy for P5608 Update production known hosts.
Fri, Dec 7, 10:06 PM
Volans added a comment to T182028: DNS repo: add CI checks for obvious configuration errors.

FYI I've added this small section to the docs for running the script:
https://wikitech.wikimedia.org/wiki/DNS#Linting_the_zone_files

Fri, Dec 7, 11:25 AM · Traffic, DNS, Patch-For-Review, Operations-Software-Development, Operations

Thu, Dec 6

Volans added a comment to T209921: ms-be2047 spontaneous reboots.

@Papaul @fgiunchedi Today the RAID alarm was continuously flapping and created a ton of tasks (see above) that I asked mo.brovac to close as he had access to the batch edit interface in Phabricator.
I've disabled the event handler for the 2 RAID checks in Icinga for this host. Please remember to re-enable them once fixed.

Thu, Dec 6, 11:35 AM · Operations, ops-codfw

Wed, Dec 5

Volans added a comment to T211213: Cumin: excluding a deleted OpenStack project in aliases.yaml causes cumin to fail.

This is totally expected, that alias query is using the global grammar mixing the results of three different queries to OpenStack according to the provided boolean operators.
The current openstack grammar allow to query with the given parameters either on all projects or a specific one. All projects but some is not a feature of the current OpenStack grammar in Cumin.
So not sure what is the request here,
As to avoid the error, a quick git grep on the Puppet repo when deleting a project should be enough. In fact contintcloud is still mentioned in another two places in the Puppet repo beside this one.

Wed, Dec 5, 4:18 PM · Patch-For-Review, Operations-Software-Development

Mon, Dec 3

Volans created P7881 Netbox extract serial and asset tag.
Mon, Dec 3, 11:56 PM
Volans added a comment to T188868: Use Pwned Passwords API to check password strength.

Note to self: do not reply to a complex topic on a Friday night
True, if the usernames and user activities are public there are lots of information available for a malicious HIBP site.

Mon, Dec 3, 3:16 PM · MediaWiki-User-login-and-signup, MediaWiki-Authentication-and-authorization, Security-Core

Fri, Nov 30

Volans added a comment to T189641: Service for checking the Pwned Passwords database.

Why "k-anonimity offers very little defense"?

It only means the attacker has to try k passwords instead of one. With k being a few hundred, that only makes a realistic difference if you use super strict rules for bad logins (e.g. lock out the user completely after some amount of tries, like banks do). I guess a wiki could use very aggressive bad password throttling specifically for login attempts with weak passwords (a few per day or month) and in that case it would make a difference, but not with the default setup.

Fri, Nov 30, 11:20 PM · User-Tgr, WMF-Legal, Patch-For-Review, Services, Security, MediaWiki-User-login-and-signup, MediaWiki-Authentication-and-authorization, Security-General
Volans added a comment to T189641: Service for checking the Pwned Passwords database.

Sure. But using that involves non-trivial tradeoffs (you are sending your password hashes to a third party - k-anonimity offers very little defense if the service is malicious) which is why I have misgivings about making it an easily enabled core option, with the more secure alternative being less easy to enable (or even discover).

Fri, Nov 30, 10:32 PM · User-Tgr, WMF-Legal, Patch-For-Review, Services, Security, MediaWiki-User-login-and-signup, MediaWiki-Authentication-and-authorization, Security-General
Volans added a comment to T210486: Audit "misc" cluster hosts.

I will leave dbmonitor ones for @Volans to decide!

Fri, Nov 30, 11:51 AM · User-Marostegui, Patch-For-Review, Operations

Thu, Nov 29

Volans added a comment to T189641: Service for checking the Pwned Passwords database.

Consideration should be given to whether having to maintain two backends is worth whatever savings come from not implementing the local service as being compatible with api.pwnedpasswords.com.

Thu, Nov 29, 9:37 PM · User-Tgr, WMF-Legal, Patch-For-Review, Services, Security, MediaWiki-User-login-and-signup, MediaWiki-Authentication-and-authorization, Security-General
Volans added a comment to T189641: Service for checking the Pwned Passwords database.

I wasn't aware of this task, but I've contacted the Security team few months ago with more or less the same idea. Hence here my a-bit-more-than-2 cents:

Thu, Nov 29, 7:45 PM · User-Tgr, WMF-Legal, Patch-For-Review, Services, Security, MediaWiki-User-login-and-signup, MediaWiki-Authentication-and-authorization, Security-General

Wed, Nov 28

Volans added a comment to T210380: Icinga downtime script should fail on the passive hosts.

@Dzahn thanks for all the fixes!

Wed, Nov 28, 7:13 PM · Patch-For-Review, monitoring, Operations
Volans updated subscribers of T210566: Netbox should use CN rather than UID for LDAP login username.

IIRC it was decided to use the UID, cc @faidon

Wed, Nov 28, 12:21 PM · netops, Operations

Tue, Nov 27

Volans removed a project from T210474: Make failures on foreachwiki more obvious the deployer: Operations-Software-Development.
Tue, Nov 27, 9:47 AM · Deployments

Mon, Nov 26

Volans triaged T210380: Icinga downtime script should fail on the passive hosts as Normal priority.
Mon, Nov 26, 9:40 AM · Patch-For-Review, monitoring, Operations
Volans created T210380: Icinga downtime script should fail on the passive hosts.
Mon, Nov 26, 9:40 AM · Patch-For-Review, monitoring, Operations

Fri, Nov 23

Volans added a comment to T210288: cumin on labspuppemaster doesn't work anymore for projects migrated to eqiad1-r.

I brought this up few weeks ago in the WMCS-admin IRC channel, explaining also that cumins puppetization uses the hiera variable profile::openstack::main::region in modules/profile/manifests/openstack/main/cumin/master.pp and to feel free to change/override it at will based on the migration.
The other short term option is to generate two different config files like config-eqiad.yaml and config-eqiad1-r.yaml and maybe adding a bash alias ease of use.

Fri, Nov 23, 2:08 PM · Operations-Software-Development, cloud-services-team (Kanban), Cloud-VPS
Volans closed T208267: Requesting access to netbox for bd808 as Resolved.
Fri, Nov 23, 10:32 AM · Patch-For-Review, LDAP-Access-Requests, Operations, SRE-Access-Requests
Volans added a comment to T208267: Requesting access to netbox for bd808.

Added read-only access to cn=wmf and confirmed it works as expected allowing people to login but in read-only mode. Edit/delete/add buttons are not shown and accessing edit pages redirect to the login page. Same for the django admin panel.

Fri, Nov 23, 9:37 AM · Patch-For-Review, LDAP-Access-Requests, Operations, SRE-Access-Requests

Wed, Nov 21

Volans added a comment to T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.

@crusnov for the puppettization I think we could go with a simple git clone and setting netbox config accordingly. You can see as an example how the cookbooks in profile::spicerack are deployed.

Wed, Nov 21, 10:30 AM · Patch-For-Review, Operations, Operations-Software-Development
Volans added a comment to T209921: ms-be2047 spontaneous reboots.

ms-be2047 reported down by Icinga since few minutes, unable to ssh, black screen at the console so far.

Wed, Nov 21, 10:14 AM · Operations, ops-codfw
Volans updated the task description for T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.
Wed, Nov 21, 10:11 AM · Patch-For-Review, Operations, Operations-Software-Development
Volans added a comment to T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.

I went ahead and created the repo for the reports at:
https://gerrit.wikimedia.org/r/admin/projects/operations/software/netbox-reports

Wed, Nov 21, 10:04 AM · Patch-For-Review, Operations, Operations-Software-Development

Tue, Nov 20

Volans updated the task description for T205897: Netbox: fill network topology.
Tue, Nov 20, 8:55 PM · Operations

Mon, Nov 19

Volans added a comment to T209757: Notifications disablement via puppet not working on icinga.

@Volans I reported this very issue, believing firmly this was a bug on our icinga installation and was discarded very dismissively. I make mistakes and I am not always right, but when I claim there is a bug I normally don't do it lightly. Specially because this is not the first nor the second time such a bad state on the alerting system happened to us. And given we are such a heavy users of it, so we notice every small quirk quickly.

Mon, Nov 19, 8:18 PM · Patch-For-Review, Operations, Icinga, monitoring
Volans added a comment to T205898: Netbox: explore NAPALM integration.

Regardless, I think this all boils down to these two questions:

  • Is it worth our time/effort to pursue this NAPALM exploration further? Having NAPALM in our toolbox may or may not be interesting for netops by itself, so we should factor that in in our decision.
Mon, Nov 19, 4:48 PM · Patch-For-Review, Operations
Volans added a comment to T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.

My proposal is to start with 1+2, 6 and 8.

Mon, Nov 19, 4:28 PM · Patch-For-Review, Operations, Operations-Software-Development

Sat, Nov 17

Volans triaged T209757: Notifications disablement via puppet not working on icinga as High priority.

Things that I've found so far, some may be unrelated but still need a fix anyway.

Sat, Nov 17, 8:41 AM · Patch-For-Review, Operations, Icinga, monitoring
Volans created T209758: parsoid-rt repeated failures on ruthenium (parsoid::testing).
Sat, Nov 17, 8:16 AM · Parsoid, Operations
Volans updated the task description for T209757: Notifications disablement via puppet not working on icinga.
Sat, Nov 17, 7:17 AM · Patch-For-Review, Operations, Icinga, monitoring

Thu, Nov 15

Volans added a comment to T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency.

As we'll be tackling this shortly, we should start deciding which report we want to write and what kind of puppetization and deployment method we want to choose.
The last bit might vary a bit also based on how we want to run those reports (manually via UI on demand, manually or automatically via HTTP API and/or CLI). See https://netbox.readthedocs.io/en/stable/additional-features/reports/#running-reports for more details.
I'll try to summarize here a few options.

Thu, Nov 15, 4:18 PM · Patch-For-Review, Operations, Operations-Software-Development

Wed, Nov 14

Volans added a project to T208706: Degraded RAID on analytics1039: Analytics.

Adding analytics, Luca and Otto in case it was missed. Also puppet has issues because of RO filesystem.

Wed, Nov 14, 6:46 PM · Patch-For-Review, Analytics, Operations, DC-Ops, ops-eqiad

Mon, Nov 12

Volans added a comment to T209265: Validate no namespaced keys are present in hieradata/*.yaml.

Regarding the few that I know:

  • profile::openstack::main::cumin::auth_group: cumin_masters doesn't actually seems to be defined elsewhere, it should probably be moved like the ones below
  • profile::openstack::main::cumin::project_pub_key: undef and profile::openstack::main::cumin::project_masters: [] seem to be defined in hieradata/eqiad/profile/openstack/main/cumin.yaml so could probably be removed easily
  • profile::netbox::netbox_server: netmon1002.wikimedia.org doesn't seem to be referenced anywhere
Mon, Nov 12, 3:22 PM · Patch-For-Review, Puppet
Volans merged T209279: debmonitor search yields nothing into T198592: Debmonitor: add search capability.
Mon, Nov 12, 2:01 PM · Patch-For-Review, Operations-Software-Development
Volans merged task T209279: debmonitor search yields nothing into T198592: Debmonitor: add search capability.
Mon, Nov 12, 2:01 PM · Operations
Volans added a comment to T209279: debmonitor search yields nothing.

I guess he's referring to the search bar at the top-right, pending code review since July ;)

Mon, Nov 12, 2:00 PM · Operations
Volans added a project to T209189: Revisit and update python testing in puppet: Operations.

Thanks @Bstorm for formalizing our random IRC chat into this proposal 😉

Mon, Nov 12, 12:50 PM · Operations, cloud-services-team (Kanban), Puppet, Proposal

Nov 9 2018

Volans added a comment to T209182: netbox won't allow me to upload photos of the rack.

@RobH yep, known issue, the immediate fix was already scheduled in https://gerrit.wikimedia.org/r/c/operations/puppet/+/463820 but then we decided to go directly in the direction of using swift as a backend for attachments to avoid to setup and rsync between the two netbox hosts. For that I preferred to leave it "broken" on purpose to avoid having then to migrate existing attachments to swift. I just din't had yet time to set it up, I hope in the next week or two to be able set everything up, but please let me know also how much is a blocker so I can prioritize accordingly.

Nov 9 2018, 8:56 PM · Operations

Nov 7 2018

Volans added a comment to T208824: rename tegmen to icinga2001 and reinstall it with stretch.

[Sorry hit submit too early...]
So either shutdown and run the decom script and then reimage with --new or follow the steps that Luca has outlined there the last time he did it.

Nov 7 2018, 3:53 PM · Patch-For-Review, monitoring, Operations
Volans added a comment to T208824: rename tegmen to icinga2001 and reinstall it with stretch.

Shutting down doesn't remove it from puppetdb, revoke it's puppet certificate and remove it from debmonitor though.

Nov 7 2018, 3:52 PM · Patch-For-Review, monitoring, Operations
Volans added a comment to T208729: Onboarding Chris Danis (CDanis).

Need one more signature on my GPG key before pwstore access can be granted

Looping in @Volans

Nov 7 2018, 11:17 AM · User-CDanis, Patch-For-Review, SRE-Access-Requests, Operations, User-herron, LDAP-Access-Requests
Volans added a comment to T208824: rename tegmen to icinga2001 and reinstall it with stretch.

@Dzahn see https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging too ;)

Nov 7 2018, 10:07 AM · Patch-For-Review, monitoring, Operations

Nov 6 2018

Volans updated subscribers of T208884: Puppet errors on automation-framework project.

@GTirloni yeah, sorry for the trouble, I know about them, just didn't had the time yet to fix them as the local PuppetDB is broken (I think it happened during the migration to the new region).
I will not spend time to fix the immediate PuppetDB failure as it's an old one anyway and we're in the process to upgrade the local PuppetDB used by the local Puppet master to the same version of production in the next few days.
If this is a blocker for anything please let me know so that I can point those instances temporarily to the Cloud puppetmasters and then back to the local one once we have the new PuppetDB up and running.

Nov 6 2018, 9:31 PM · Cloud-VPS
Volans updated subscribers of T208861: cumin: Support multiple OpenStack regions.
Nov 6 2018, 4:05 PM · Patch-For-Review, Operations-Software-Development

Nov 5 2018

Volans updated subscribers of T208783: Migrate tests from nose to pytest.

In general I'm all in for the nose -> pytest migration and pytest is what we're using in a lot of other projects.
Regarding the Puppet repo specifically though there are multiple angle to look at, that makes me wonder if those more complex scripts that requires testing shouldn't be inside the puppet repo in the first place. Also I think that if we start touching it we shouldn't just blindly replace nose with pytest but instead re-think the whole Python testing within the Puppet repo.

Nov 5 2018, 8:54 PM · Operations

Nov 2 2018

aborrero awarded T179816: Cumin: create external backend for WMCS Puppet API a Like token.
Nov 2 2018, 1:05 PM · Operations-Software-Development

Oct 31 2018

Volans added a comment to T208462: Error Unknown column ipb_sitewide in field list on query.

Just for a quick reference the alter to create the table (confirmed also by the history on neodymium) should be:

set session sql_log_bin=0; ALTER TABLE  ipblocks   ADD ipb_sitewide bool NOT NULL default 1;
Oct 31 2018, 9:03 PM · DBA, Anti-Harassment, Operations
Volans updated subscribers of T208462: Error Unknown column ipb_sitewide in field list on query.

I've quickly audited the ipblocks.frm on all cored DBs in all shards (s1-s8) for all schemas and the only one missing (apart schemas that don't have it either on the masters because not in all.dblist) is ruwikiquote on db2050.
To do it quickly (as I'm not anymore familiar with the current tooling around DB stuff) I did the poor's man approach running things like:

sudo cumin 'C:mariadb::heartbeat%shard = s3' "grep -c 'ipb_sitewide' /srv/sqldata*/*/ipblocks.frm"

I've then checked with a similar approach the dbstores, and again, only dbstore2002 for s3 has that field missing.

Oct 31 2018, 8:43 PM · DBA, Anti-Harassment, Operations

Oct 30 2018

Volans added a comment to T201247: Sporadic puppet failures.

@Andrew did it reoccurred during last week? do you have a list of hostnames+time by any chance?

Oct 30 2018, 11:25 PM · cloud-services-team (Kanban), Operations

Oct 29 2018

Volans closed T199413: Systemd restart loop of timer filled the disk on tegmen as Resolved.

This hasn't repro in months and we're moving to stretch on the Icinga hosts. Resolving for now, feel free to re-open if this happens again.

Oct 29 2018, 3:26 PM · Patch-For-Review, Icinga, monitoring, Operations

Oct 27 2018

Volans added a comment to T202782: upgrade icinga server to stretch and replace einsteinium.

Mentioned in SAL (#wikimedia-operations) [2018-10-27T00:00:06Z] <mutante> icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

Oct 27 2018, 9:53 AM · Patch-For-Review, monitoring, Operations
Volans added a comment to T208100: cumin tries to downtime Icinga even with --no-downtime.

Isn't the issue that despite saying --no-downtime it tries to set a downtime?

Oct 27 2018, 9:48 AM · Operations, Operations-Software-Development

Oct 26 2018

Volans added a comment to T208100: cumin tries to downtime Icinga even with --no-downtime.

Actually the --new might not work either as the host is in puppetdb, sorry for the wrong suggestion.
Anyway this is kinda unrelated to the reimage script as the issue is that we don't monitor the other icinga hosts from the active one, so it's really a corner case and not sure it should be fixed hardcoding this weirdness into the reimage script.

Oct 26 2018, 10:44 PM · Operations, Operations-Software-Development
Volans added a comment to T208100: cumin tries to downtime Icinga even with --no-downtime.

See the --new option

Oct 26 2018, 10:42 PM · Operations, Operations-Software-Development

Oct 24 2018

Volans created T207898: Cumin PuppetDB backend: allow to filter by last run metadata.
Oct 24 2018, 9:45 PM · Operations-Software-Development
Volans added a comment to T116580: monitor postgresql replication status.

I guess the description should be updated, as we have more installations in prod now, and we actually already have a check for replication, see modules/postgresql/manifests/slave/monitoring.pp

Oct 24 2018, 12:58 PM · monitoring, Operations

Oct 23 2018

Volans committed rCUMIN0e5fc0075dc2: tests: remove pylint skip-file (authored by Volans).
tests: remove pylint skip-file
Oct 23 2018, 4:49 PM

Oct 22 2018

Volans updated subscribers of T205867: Expand Spicerack library and SRE Cookbooks - Q2 2018-19 Goal.
Oct 22 2018, 1:50 PM · Operations-Software-Development, Operations, Goal
Volans updated subscribers of T205868: Expand Netbox usage - Q2 2018-19 Goal.
Oct 22 2018, 1:49 PM · Operations, Operations-Software-Development, Goal
Volans claimed T205884: Spicerack: split wmf-auto-reimage-lib into Spicerack modules.
Oct 22 2018, 9:22 AM · Patch-For-Review, Operations-Software-Development

Oct 19 2018

Volans merged T161545: Cumin: PuppetDB backend, allow to specify boolean values for resource parameters into T207037: Cumin: allow to query for Puppet primitive types.
Oct 19 2018, 11:38 AM · Patch-For-Review, Operations-Software-Development
Volans merged task T161545: Cumin: PuppetDB backend, allow to specify boolean values for resource parameters into T207037: Cumin: allow to query for Puppet primitive types.
Oct 19 2018, 11:38 AM · Operations-Software-Development
Volans closed T201346: rack/setup/install clustermgmt1001.eqiad.wmnet (new cumin master) as Resolved.
Oct 19 2018, 11:35 AM · ops-eqiad, Operations-Software-Development, Operations

Oct 18 2018

Volans added a comment to T202051: db2042 (m3) master RAID battery failed.

Opened T207417 for the ferm part.

Oct 18 2018, 8:30 PM · User-Banyek, Operations, ops-codfw, DBA
Volans created T207417: ferm fail to start at boot in some cases.
Oct 18 2018, 8:30 PM · Operations
Volans added a comment to T202051: db2042 (m3) master RAID battery failed.

db2042 failed to start ferm at reboot due to a DNS timeout query:

Oct 18 15:53:04 db2042 ferm[837]: DNS query for 'prometheus2003.codfw.wmnet' failed: query timed out
[...SNIP...]
Oct 18 15:53:04 db2042 systemd[1]: Failed to start ferm firewall configuration.

Apparently the 2 icinga checks that report it were not noticed as probably the host was downtimed for the programmed maintenance.
I've manually started ferm and it all worked fine but it has been without ferm since the reboot.
I'm opening a separated task to fix the puppet/systemd side of it

Oct 18 2018, 8:25 PM · User-Banyek, Operations, ops-codfw, DBA
Volans added a comment to T207385: Create a check on the DC failover script to see if codfw -> eqiad replication is working before failing over to codfw (considering eqiad as the active DC by default).

Sure, we can add a step that checks the parser cache replication/heartbeat.
Could you precisely outline in which phase we need to check what and also update the SwitchDatacenter wiki page so that is clear that the step is needed even before we automate that in the cookbooks?

From the top of my head I think it should go where the check if all the masters are up to date is.
Can you give me link to the switch wiki page?

Thank you!

Oct 18 2018, 1:32 PM · Operations-Software-Development, Datacenter-Switchover-2018
Volans added a comment to T207273: Parser cache hit ratio alerting.

That's exactly what I meant, we should have this check independently and adding other checks to the other part described in T207385 to prevent it.

Oct 18 2018, 1:10 PM · monitoring, DBA
Volans added a comment to T207385: Create a check on the DC failover script to see if codfw -> eqiad replication is working before failing over to codfw (considering eqiad as the active DC by default).

Sure, we can add a step that checks the parser cache replication/heartbeat.
Could you precisely outline in which phase we need to check what and also update the SwitchDatacenter wiki page so that is clear that the step is needed even before we automate that in the cookbooks?

Oct 18 2018, 1:05 PM · Operations-Software-Development, Datacenter-Switchover-2018
Volans added a comment to T207273: Parser cache hit ratio alerting.

My suggestion for this kind of check was not for the passive dc, but mainly the active one to make sure that the parser caches are properly used. We might have changes in mediawiki that will change the hit ratio over time and it could go below a threshold that causes issues.
I think it might be useful in general to have this check and it would have also immediately alarm after the switch to tell us the real cause of the issue. It's not meant to prevent it, for that we'll have the other ones (replication/heartbeat/cookbook)

Oct 18 2018, 1:04 PM · monitoring, DBA
Volans updated the task description for T205868: Expand Netbox usage - Q2 2018-19 Goal.
Oct 18 2018, 10:49 AM · Operations, Operations-Software-Development, Goal
Volans closed T205896: Netbox: upgrade to the latest version (>= 2.4) as Resolved.

Netbox has been upgraded to upstream 2.4.6. Report any issue you might found.

Oct 18 2018, 10:49 AM · Patch-For-Review, Operations
Volans closed T205896: Netbox: upgrade to the latest version (>= 2.4) , a subtask of T205868: Expand Netbox usage - Q2 2018-19 Goal, as Resolved.
Oct 18 2018, 10:49 AM · Operations, Operations-Software-Development, Goal

Oct 17 2018

Volans updated the task description for T207009: Onboarding Cas Rusnov.
Oct 17 2018, 7:07 PM · Patch-For-Review, Operations
Volans updated the task description for T207009: Onboarding Cas Rusnov.
Oct 17 2018, 6:54 PM · Patch-For-Review, Operations
Gerrit Code Review <gerrit@wikimedia.org> committed rOSNB68447d54688a: Modify access rules (authored by Volans).
Modify access rules
Oct 17 2018, 1:17 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rOSNB782edc05634f: Modify access rules (authored by Volans).
Modify access rules
Oct 17 2018, 1:17 PM
Volans committed rOSNBc46280c3e60a: Merge tag 'v2.4.6' (authored by Volans).
Merge tag 'v2.4.6'
Oct 17 2018, 11:27 AM
Gerrit Code Review <gerrit@wikimedia.org> committed rOSNB831404375237: Modify access rules (authored by Volans).
Modify access rules
Oct 17 2018, 11:27 AM
Volans committed rOSNBb806e6115e68: Revert "Add Wikimedia's initial data" (authored by Volans).
Revert "Add Wikimedia's initial data"
Oct 17 2018, 11:25 AM
Volans added a reverting change for rOSNB6a1caa720b90: Add Wikimedia's initial data: rOSNBb806e6115e68: Revert "Add Wikimedia's initial data".
Oct 17 2018, 11:25 AM
Volans committed rOSNBb6a2931e0dc5: Revert "Allow custom fields in the Device CSV form" (authored by Volans).
Revert "Allow custom fields in the Device CSV form"
Oct 17 2018, 11:25 AM
Volans added a reverting change for rOSNB07a7facdf44d: Allow custom fields in the Device CSV form: rOSNBb6a2931e0dc5: Revert "Allow custom fields in the Device CSV form".
Oct 17 2018, 11:25 AM

Oct 16 2018

Volans updated the task description for T207009: Onboarding Cas Rusnov.
Oct 16 2018, 9:13 PM · Patch-For-Review, Operations
Volans changed the visibility for P5608 Update production known hosts.
Oct 16 2018, 9:04 PM
Volans added a comment to T206992: Create replication icinga check for the Parsercache hosts.

I think we could also consider adding an alert based on the hit ratio of the parsercache caches (we already have the data in grafana)

Oct 16 2018, 4:24 PM · Patch-For-Review, Wikimedia-Incident, User-Banyek, DBA

Oct 15 2018

Volans updated the task description for T207009: Onboarding Cas Rusnov.
Oct 15 2018, 7:02 PM · Patch-For-Review, Operations
Volans updated subscribers of T207009: Onboarding Cas Rusnov.
Oct 15 2018, 6:29 PM · Patch-For-Review, Operations
Volans updated the task description for T207009: Onboarding Cas Rusnov.
Oct 15 2018, 6:28 PM · Patch-For-Review, Operations
Volans added a member for WMF-NDA-Requests: crusnov.
Oct 15 2018, 6:28 PM
Volans added a member for WMF-NDA: crusnov.
Oct 15 2018, 6:27 PM
Volans added a member for acl*sre-team: crusnov.
Oct 15 2018, 6:26 PM
Volans triaged T207037: Cumin: allow to query for Puppet primitive types as Normal priority.
Oct 15 2018, 2:20 PM · Patch-For-Review, Operations-Software-Development