since making puppetmaster1003 an active puppet master the following error is occasionally observed
Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Unsupported facts format
When looking at the puppet master apache logs we see the following
2019-09-23T17:13:54 39001 2620:0:861:107:10:64:48:144 proxy-server/500 283 POST http://puppet:8140/puppet/v3/catalog/ms-be1039.eqiad.wmnet?environment=production -application/json - - Puppet/5.5.10 Ruby/2.3.3-p222 (x86_64-linux-gnu) -- - - 2620:0:861:107:10:64:48:144
This error is coming from lib/puppet/indirector/catalog/compiler.rb after a debian patch was applied meaning the 4.8.2 puppet master only supports pson or msgpack (i think that's what the first if catches). however puppet master 5.5.10 supports json. I believe that agents are negotiating with puppetmaster1003 and choosing a serialization of json, however when they come to submit there facts they hit a 4.8.2 master which doesn't understand json.
The recommended fix is to set preferred_serialization_format = pson until the migration to 5.5.10 is complete