From all postgres read replicas on eqiad we get similar errors in the logs:
root@maps1005:/home/jgiannelos# tail /var/log/postgresql/postgresql-11-main.log 2021-08-27 11:09:45 GMT LOG: started streaming WAL from primary at 617/59000000 on timeline 1 2021-08-27 11:09:45 GMT FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000061700000059 has already been removed 2021-08-27 11:09:50 GMT LOG: started streaming WAL from primary at 617/59000000 on timeline 1 2021-08-27 11:09:50 GMT FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000061700000059 has already been removed 2021-08-27 11:09:55 GMT LOG: started streaming WAL from primary at 617/59000000 on timeline 1 2021-08-27 11:09:55 GMT FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000061700000059 has already been removed 2021-08-27 11:10:00 GMT LOG: started streaming WAL from primary at 617/59000000 on timeline 1 2021-08-27 11:10:00 GMT FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000061700000059 has already been removed 2021-08-27 11:10:05 GMT LOG: started streaming WAL from primary at 617/59000000 on timeline 1 2021-08-27 11:10:05 GMT FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000061700000059 has already been removed
here are the logs from the master node
2021-08-27 11:10:53 GMT [9775]: [2-1] user=replication,db=[unknown],app=[unknown],client=10.64.16.27 LOG: replication connection authorized: user=replication 2021-08-27 11:10:53 GMT [9775]: [3-1] user=replication,db=[unknown],app=walreceiver,client=10.64.16.27 ERROR: requested WAL segment 0000000100000622000000F8 has already been removed 2021-08-27 11:10:53 GMT [9775]: [4-1] user=replication,db=[unknown],app=walreceiver,client=10.64.16.27 LOG: disconnection: session time: 0:00:00.004 user=replication database= host=10.64.16.27 port=37506 2021-08-27 11:10:53 GMT [9776]: [2-1] user=replication,db=[unknown],app=[unknown],client=10.64.16.6 LOG: replication connection authorized: user=replication 2021-08-27 11:10:53 GMT [9776]: [3-1] user=replication,db=[unknown],app=walreceiver,client=10.64.16.6 ERROR: requested WAL segment 0000000100000622000000E6 has already been removed 2021-08-27 11:10:53 GMT [9776]: [4-1] user=replication,db=[unknown],app=walreceiver,client=10.64.16.6 LOG: disconnection: session time: 0:00:00.003 user=replication database= host=10.64.16.6 port=52490 2021-08-27 11:10:53 GMT [9777]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=10.64.0.18 LOG: connection received: host=10.64.0.18 port=53734 2021-08-27 11:10:53 GMT [9777]: [2-1] user=replication,db=[unknown],app=[unknown],client=10.64.0.18 LOG: replication connection authorized: user=replication 2021-08-27 11:10:53 GMT [9777]: [3-1] user=replication,db=[unknown],app=walreceiver,client=10.64.0.18 ERROR: requested WAL segment 0000000100000622000000CF has already been removed 2021-08-27 11:10:53 GMT [9777]: [4-1] user=replication,db=[unknown],app=walreceiver,client=10.64.0.18 LOG: disconnection: session time: 0:00:00.003 user=replication database= host=10.64.0.18 port=53734