Dec 17 2019
@Igorkim78 can you document the config changes here?
Dec 16 2019
Dec 13 2019
Dec 12 2019
Dec 11 2019
This is resolved now.
I worked with Filippo to fix this via the following commands:
Dec 9 2019
Dec 6 2019
Dec 5 2019
onimisionipe@cloudelastic1002:~$ sudo smartctl -H /dev/sdb smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-9-amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Dec 4 2019
Dec 3 2019
Nov 28 2019
Nov 25 2019
I have deployed Lucas's patch. Kindly test this to confirm everything is fine.
Nov 20 2019
Nov 18 2019
Nov 15 2019
Nov 13 2019
@Igorkim78 file has been uploaded.
Nov 6 2019
Sorry all.. I just did another deployment and patch has been correctly applied. Also test works fine now
I've deployed @Lucas_Werkmeister_WMDE patch. But I can still run:
Nov 5 2019
T230746 is almost here. So we won't be resharding any index for now
Nov 4 2019
see https://phabricator.wikimedia.org/T237228 for current OSM replication issues
I'm closing this task as there are icinga alerts for osm sync
Nov 1 2019
Oct 29 2019
Oct 27 2019
This is causing mjolnir deploy directory to become unavailable/missing and also causing puppet to fail.
Oct 21 2019
Oct 16 2019
Oct 10 2019
Oct 9 2019
@dduvall Thanks!. I removed the test stage also forced devdeps to install. We should definitely look at a better way to handle this later. but Its fine as it is.
Currently, Build is passing but not publishing yet. Do we need to enable CI publish stage for the repo?
@dduvall Thanks. I will implement this.
Oct 3 2019
This issue has come up again. Currently, we have only enwiki_content_1546970425 unassigned with too many shards  allocated to this node for index [enwiki_content_1546970425], index setting index.routing.allocation.total_shards_per_node=1] error from _cluster/allocation/explain.
Oct 1 2019
Post merge builds seems to fail.
Sep 24 2019
We should talk to elastic to see how we can move this forward.
Currently, we require jackson-databind 2.8.11 and jackson-annotation 2.8.11 for JsonLayout to work when using SyslogAppender. Version 2.8.6 is provided by debian for this packages. We should use the correct version to make sure everything work as expected.
Sep 23 2019
Sep 20 2019
Sep 18 2019
Sep 16 2019
Sep 12 2019
@Ladsgroup there's no TLS termination on that port for now. We should have and I will work on it in the nearest future. Please use HTTP for now
Sep 11 2019
Sep 10 2019
Sep 9 2019
Sep 6 2019
This is a know issue. The SRE team is finding a quick solution to restore these services. Thanks
JsonLayout requires other dependencies for log4j. This include jackson databind. See https://logging.apache.org/log4j/2.x/runtime-dependencies.html.
- Rebuild log4j with this dependencies
- Fall back to shipping logs with PatternLayout.
Sep 4 2019
Not sure but seems we are missing some configs in our config.yaml patch
Sep 3 2019
rsyslog Json requires the @cee token which must be provided according to standard via profile::rsyslog::udp_localhost_compat. Let's use profile::rsyslog::udp_json_logback_compat instead as it permits parsing of json from log4j without the token.
Sep 2 2019
Aug 29 2019
On another note, I think this check make sense for other clusters as well
elastic1029 is back on icinga showing memory errors. see https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=elastic1029&service=Memory+correctable+errors+-EDAC-
Aug 28 2019
My screenshot from Windows 10/Version 76.0.3809.100 (Official Build) (64-bit)
I changed the priority of this to normal. Feel free to change it as you see fit