Change Details

#pywikibot runs a relatively large set of post-merge tests via GitHub Actions. Some of these tests use Beta Cluster wikis as their target for end-to-end testing of various features. The efforts to exclude unwanted bots from {T393487} have recently blocked some of the IP addresses used by GitHub Actions. GitHub Actions uses Microsoft Azure to host many (all?) of it's runners. There are over 5000 (!) IP ranges listed at https://api.github.com/meta that GitHub Actions might make requests from. Some potential options: * Allowlist 5000+ CIDR ranges and keep that list updated. * Setup [[https://docs.github.com/en/actions/how-tos/hosting-your-own-runners|self-hosted GitHub runners]] for use by https://github.com/wikimedia/ organization projects. * Add a SOCKS5 proxy to the appropriate test suites to tunnel traffic to an exit that is unlikely to be blocked * Migrate all of these tests to a CI platform that is "in-house" (Zuul or GitLab CI) and unlikely to be blocked See also: * {T399367} * {T399415}

#pywikibot runs a relatively large set of post-merge tests via GitHub Actions. Some of these tests use Beta Cluster wikis as their target for end-to-end testing of various features. The efforts to exclude unwanted bots from {T393487} have recently blocked some of the IP addresses used by GitHub Actions. GitHub Actions uses Microsoft Azure to host many (all?) of it's runners. There are over 5000 (!) IP ranges listed at https://api.github.com/meta that GitHub Actions might make requests from. Some potential options: * Allowlist 5000+ CIDR ranges and keep that list updated. * Setup [[https://docs.github.com/en/actions/how-tos/hosting-your-own-runners|self-hosted GitHub runners]] for use by https://github.com/wikimedia/ organization projects. * Add a SOCKS5 proxy to the appropriate test suites to tunnel traffic to an exit that is unlikely to be blocked * Migrate all of these tests to a CI platform that is "in-house" (Zuul or GitLab CI) and unlikely to be blocked See also: * {T399367} * {T399415} **Steps to implement SOCKS proxy from GitHub action to WMCS** [] @xqt makes a Developer account specifically to act as the credential holder that can build an ssh SOCKS5 tunnel from GitHub Actions to bastion.wmcloud.org. [] @xqt adds an ssh public key to that new Developer account and keeps track of the associated public key for the GitHub Actions configuration. [] @xqt asks @bd808 to make the new Developer account a member of the bastion project so it can ssh in. [] @bd808 does the needful [x] @xqt figures out how to add config to the GitHub Actions to establish an ssh tunnel from the Action runner to bastion.wmcloud.org. A pure cli way to do this would be something like `ssh -o StrictHostKeyChecking=accept-new -f -N -D 127.0.0.1:1080 -i $PRIVATE_KEY_FILE $USER@bastion.wmcloud.org` ** `-o StrictHostKeyChecking=accept-new`: Accept offered host key for any host not already in the known hosts file ** `-f`: Background ssh process after connecting ** `-N`: Do not exec a remote command ** `-D 127.0.0.1:1080`: Create a SOCKS5 proxy listening on 127.0.0.1:1080 and terminated on the ssh connected host ** `-i $PRIVATE_KEY_FILE`: Use the private key in $PRIVATE_KEY_FILE [x] @Xqt adds the needed equivalent of `export HTTPS_PROXY="socks5h://127.0.0.1:1080"` to the GitHub Actions to tell requests to proxy traffic though the tunnel and do DNS resolution on the proxy termination side so that the internal network IPs are contacted when traffic flows over the tunnel. There are some weird things that might happen if the DNS is done outside the Cloud VPS network. Public IPv4 addresses in Cloud VPS work in ways that are sometimes confusing. (Copied from 399485#11011491 to keep track of the steps)

#pywikibot runs a relatively large set of post-merge tests via GitHub Actions. Some of these tests use Beta Cluster wikis as their target for end-to-end testing of various features. The efforts to exclude unwanted bots from {T393487} have recently blocked some of the IP addresses used by GitHub Actions. GitHub Actions uses Microsoft Azure to host many (all?) of it's runners. There are over 5000 (!) IP ranges listed at https://api.github.com/meta that GitHub Actions might make requests from. Some potential options: * Allowlist 5000+ CIDR ranges and keep that list updated. * Setup [[https://docs.github.com/en/actions/how-tos/hosting-your-own-runners|self-hosted GitHub runners]] for use by https://github.com/wikimedia/ organization projects. * Add a SOCKS5 proxy to the appropriate test suites to tunnel traffic to an exit that is unlikely to be blocked * Migrate all of these tests to a CI platform that is "in-house" (Zuul or GitLab CI) and unlikely to be blocked See also: * {T399367} * {T399415} **Steps to implement SOCKS proxy from GitHub action to WMCS** [] @xqt makes a Developer account specifically to act as the credential holder that can build an ssh SOCKS5 tunnel from GitHub Actions to bastion.wmcloud.org. [] @xqt adds an ssh public key to that new Developer account and keeps track of the associated public key for the GitHub Actions configuration. [] @xqt asks @bd808 to make the new Developer account a member of the bastion project so it can ssh in. [] @bd808 does the needful [x] @xqt figures out how to add config to the GitHub Actions to establish an ssh tunnel from the Action runner to bastion.wmcloud.org. A pure cli way to do this would be something like `ssh -o StrictHostKeyChecking=accept-new -f -N -D 127.0.0.1:1080 -i $PRIVATE_KEY_FILE $USER@bastion.wmcloud.org` ** `-o StrictHostKeyChecking=accept-new`: Accept offered host key for any host not already in the known hosts file ** `-f`: Background ssh process after connecting ** `-N`: Do not exec a remote command ** `-D 127.0.0.1:1080`: Create a SOCKS5 proxy listening on 127.0.0.1:1080 and terminated on the ssh connected host ** `-i $PRIVATE_KEY_FILE`: Use the private key in $PRIVATE_KEY_FILE [x] @Xqt adds the needed equivalent of `export HTTPS_PROXY="socks5h://127.0.0.1:1080"` to the GitHub Actions to tell requests to proxy traffic though the tunnel and do DNS resolution on the proxy termination side so that the internal network IPs are contacted when traffic flows over the tunnel. There are some weird things that might happen if the DNS is done outside the Cloud VPS network. Public IPv4 addresses in Cloud VPS work in ways that are sometimes confusing. (Copied from 399485#11011491 to keep track of the steps)