User Details
- User Since
- Jan 7 2019, 1:06 PM (22 w, 6 d)
- Availability
- Available
- IRC Nick
- jbond42
- LDAP User
- Jbond
- MediaWiki User
- JBond (WMF) [ Global Accounts ]
Fri, Jun 7
Awesome, ill keep the ticket open so people now im playing with the other two servers
Sorry for this i have just pushed https://gerrit.wikimedia.org/r/c/operations/puppet/+/515017 so this should only be broken for sarin and neodyium now, sorry for the interruption and ping me if you still see issues
Thu, Jun 6
going to re-image this server to stretch, testing changes to late_command.sh
im going to reimage this server to test the following change
https://gerrit.wikimedia.org/r/c/operations/puppet/+/514689
Wed, Jun 5
Tue, Jun 4
A CR for this has been deployed so the spam should be gone. please reopen if spam persists
Mon, Jun 3
I looked at this a bit more today and my initial analysts was wrong. the facts do actually resolve they just take longer when there are disk issues. Further i was unable to find a command to test if we have this issues that would be generic enough to apply across the fleet. As such for now i have created a patch to increase the timeout values but I'm definitely open to better suggestions
Thu, May 30
Tue, May 28
Fri, May 24
ack, thanks for the clarification
Thu, May 23
just watching ripe presentation and thought this may be of interest https://ripe78.ripe.net/archives/video/106
Wed, May 22
@Tgr while reviewing the change created by volans i noticed that currently wikimedia.modular.im. dose not exist. We should ensure this exists and belongs to the wikimedia foundation before adding the authorization. further having followed the links it looks like the hosted service from modular.im would actually be a riot.im box and would have the name wikimedia.riot.im
Tue, May 21
I have done a bit of testing with the lvm_vgs. under the hood this command runs vgs -o name --noheadings, if i run this from the command line and time it i get the following
May 15 2019
have created changes for pcc and archive-project-volumes
utils/pcc
left this till last as im not sure how/where its called
May 14 2019
after speching with alex it seems that facter2 set ipaddress6 to undef if there where only link-local adrdesses however in facter3 it has the value of the link-local address (however more investigation required)
git/puppet [ grep -r ipaddress6 ./modules review/jbond/update_python ] 9:57 PM
./modules/base/lib/facter/interface_primary.rb:Facter.add('ipaddress6') do
./modules/base/lib/facter/interface_primary.rb: # Do not rely on ipaddress6_#{interface_primary}, as its underlying
./modules/interface/manifests/add_ip6_mapped.pp: $ipv6_address = inline_template("<%= require 'ipaddr'; (IPAddr.new(scope.lookupvar(\"::ipaddress6_${interface}\")).mask(64) | IPAddr.new(@v6_mapped_lower64)).to_s() %>")
./modules/profile/templates/exim/exim4.conf.mailman.erb: interface = <; <%= @ipaddress %> ; <%= @ipaddress6 %>
./modules/profile/templates/exim/exim4.conf.mailman.erb: interface = <; <%= @ipaddress %> ; <%= @ipaddress6 %>
./modules/profile/manifests/dnsrecursor.pp: $facts['ipaddress6'],
./modules/profile/manifests/dnsrecursor.pp: ::dnsrecursor::monitor { [ $facts['ipaddress'], $facts['ipaddress6'] ]: }
./modules/profile/manifests/pybal.pp: 'bgp-nexthop-ipv6' => inline_template("<%= require 'ipaddr'; (IPAddr.new(@ipaddress6).mask(64) | IPAddr.new(\"::\" + @ipaddress.gsub('.', ':'))).to_s() %>"),
./modules/profile/manifests/openstack/base/pdns/auth/service.pp: dns_auth_ipaddress6 => $facts['ipaddress6'],
./modules/ssh/manifests/server.pp: if $::ipaddress6 == undef {
./modules/ssh/manifests/server.pp: $aliases = [ $::hostname, $::ipaddress, $::ipaddress6 ]
./modules/standard/spec/default_module_facts.yml:ipaddress6: 2001:db8::42
./modules/calico/templates/initscripts/calico-node.systemd.erb: -e IP6=<%= @ipaddress6 %> \
./modules/pdns_server/templates/pdns.conf.erb:<% if @dns_auth_ipaddress6 then %>local-ipv6=<%= @dns_auth_ipaddress6 %><% end %>
./modules/pdns_server/manifests/init.pp:# - $dns_auth_ipaddress6:IPv6 address PowerDNS will bind to and send packets from
./modules/pdns_server/manifests/init.pp: $dns_not to check theses code points
As i was on clinic duty last week i decided to have a crack at doing the manual work to rename everything in the repo so that they can be checked by CI. Thanks to all the reviewers i have mostly got that complete with only two changes still wanting review. I have also created a change to add a CI check to ensure python files have the correct extension. It would be good to get the latter committed asap to prevent more files from being added.
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/508855 which has now been rolled back
Confirm that is responsible for the faliure, the other re-arms where done by myself
The change above failed as the phsicalDrive regex dose not take into account the the number of drives per span. however further investigation shows we have this discrepancy on other servers
logs are now been sent to kafka, however we still need to role the profile::firewall::logging module to all infrastructure which is been tracked in T116011
May 10 2019
This should be complete now please allow up to 30 minutes for puppet to role out the change any issues after that please re-open the ticket
latest package has been uploaded re open if further problems
@greg can you please approve jfishback addition to the deployment group
May 9 2019
Ok great thanks for the update
I guess the removal needs merging?
Is this ticket complete? can it be closed, if not what further actions are required?
@greg this is essentially a request to add cparle to the deployment group. As the manager of Release engineering are you the relevent person to approve this access request? Thanks
I have created the list and you should have received the admin password. I have set some of the privacy options but the majority of settings have been left to the default values as such please check all config options to ensure the list is configured as desired. Pleas also add a description and update the mailing list page. Let me know if there are any issues
Hi @RStallman-legalteam can you confirm NDA status please
Can this task be made public (via "Edit Task > Visible To")?
done
May 8 2019
correct pull request https://github.com/puppetlabs/facter/pull/1775
@Aklapper thanks for the explanation i hadn't realised that restriction.
I have created the following patch not sure why it wasn't auto tagged here
https://gerrit.wikimedia.org/r/c/operations/puppet/+/508820
did this issue get fixed? i just checked scandium and parsoid-vd has been running for 3 weeks
Is there any further action for this ticket or can we close it?
Are there any more actions required here or can we close this ticket?
Resolving as this looks fixed, i have checked the logs and the last error i see relating to this is from 2019-05-01 20:51:17 so im pretty sure its fixed but please reopen if im wrong
May 7 2019
! In T221529#5144319, @crusnov wrote:
Has anyone checked if the 5xx errors happen to coincide with puppet-merge happening?
I think we should patch facter once your PR is reviewed/merged upstream to address this for good. But I think it's fine to proceed with the facter rollout given that this is harmless log spam and only affects ~ 10% of our servers (mc and cp hosts).
Resolving this and will track the root problem in https://phabricator.wikimedia.org/T222356
this has been uploaded let me know if there are any issues
May 3 2019
I have updated the new package and removed the thirdparty package let me know if you see any issues
May 2 2019
This is due to a bug in facter fundamentally cased because there are an even number or words in the output of ip route show. I have proposed a fix upstream and i believe the warning is harmless (in fact my PR removes the warning all together). If we want to work around this issue more we have a number of options
- patch or facter package (@MoritzMuehlenhoff opinion?)
- update modules/interface/manifests/route.pp to remove the lock keyword. lock stops the kernel from updating the MTU via PTMU. I'm unsure if that may cause issues @ayounsi could likely add more information here?
i have added a plaster to the smart-data-dump to stop the spam and will investigate the underlining issues further via T222326
Apr 30 2019
Apr 29 2019
Have not had time to look at this in depth yet however i did just notice an issue while applying a refactor change[1]
intend to get this in a second but in case i forget the following DNS entries need cleaning up
Apr 26 2019
Looking at cumin1001 I noticed that the log prefix at the end of the input chan is "fw-out-drop" and the output chain is empty with an accept policy. Is "out" indeed the direction in this case? Or would dropped packets logged by the input chain be considered "in"?