Change Details

We are splitting RESTBase in two components - the (public) REST API router and the storage service (cf. {T220449}). This task is about deploying the front-end REST router in Kubernetes. == Service Info **Service name**: RESTRouter (name still under discussion, cf. T220761) **Owners**: @Pchelolo and @mobrovac (#core_platform_team) **Repository**: `mediawiki/services/restbase` **ETA**: by the end of Q4 FY18/19 **Description**: RESTRouter is the routing part of (the current) RESTBase. It accepts external requests, validates them (performs access checks if needed) and performs all the business logic related to the request: it looks up the storage for possible data hits and, if needed, issues requests to back-end services to complete the requests, sending the response to storage prior to returning it to the client. == Deployment Plan RESTRouter will effectively take over request handling from RESTBase, so we will need to divert traffic to it without interruption. First, we deploy RESTRouter to k8s. Next, we expose the storage routes in RESTBase (cf. [PR #1103](https://github.com/wikimedia/restbase/pull/1103)) and test RESTRouter for load (options include synthetic traffic, mirroring, using only background updates/internal requests). Then, we include RESTRouter as an additional back-end for RESTBase. We do more load-tests. Finally, a new LVS is created for RESTRouter and Varnish and all back-end services are instructed to use the new LVS end point instead of the current one. In the post-deploy clean-up step, we remove public route handling from RESTBase, effectively turning it into the back-end storage service. Benchmarking: - https://wikitech.wikimedia.org/wiki/User:Alexandros_Kosiaris/Benchmarking_kubernetes_apps - P8425

We are splitting RESTBase in two components - the (public) REST API router and the storage service (cf. {T220449}). This task is about deploying the front-end REST router in Kubernetes. = Service Info = **Service name**: RESTRouter (name still under discussion, cf. T220761) **Owners**: @Pchelolo and @mobrovac (#core_platform_team) **Repository**: `mediawiki/services/restbase` **ETA**: by the end of Q4 FY18/19 **Description**: RESTRouter is the routing part of (the current) RESTBase. It accepts external requests, validates them (performs access checks if needed) and performs all the business logic related to the request: it looks up the storage for possible data hits and, if needed, issues requests to back-end services to complete the requests, sending the response to storage prior to returning it to the client. = Deployment Plans = Restrouter migration plans. Some parts are the same for all plans. Those are listed below * First, we deploy RESTRouter to k8s. * we expose the storage routes in RESTBase (cf. [PR #1103](https://github.com/wikimedia/restbase/pull/1103)) * test RESTRouter for load (options include synthetic traffic, mirroring, using only background updates/internal requests). == Plan 1 == Have restbase listen on both 7231 and 7233 and configure LVS restbase.svc.$::site.wmnet to also use 7233 Instantiate restrouter on a new LVS IP and DNS (restrouter.svc.$::site.wmnet) and have it talk to restbase.svc.$::site.wmnet:7233 Move services 1 by 1 to restrouter.discovery.wmnet (the site aware discovery records for restrouter.svc.$::site.wmnet) === Pros === * Move is gradual on a service level. Services are migrated one by one based on their configuration unearthing potential problems one by one * The currently stable and battle tested restbase installation is kept around even while more and more services are moved around * It's rather easy configuration wise, rather easy to do in steps * No downtime for services. === Cons === * The migration might take time as when issues arise, but at least blockers will be service specific * There is no gradual traffic switchover. For every service it's a "canary host first", then all or nothing approach. Even the canary host is depending on DC between 13% and 25% of traffic == Plan 2 == Have restbase listen on both 7231 and 7233 Add a new LVS IP on the restbase hosts and name it restbase-backend.svc.$::site.wmnet Configure restrouter to connect to restbase-backend.svc.$::site.wmnet:7233 Add the LVS IP for restbase.svc.$::site.wmnet to kubernetes hosts Add the kubernetes hosts to LVS for restbase.svc.$::site.wmnet Slowly migrate the traffic from the current restbase hosts to kubernetes hosts === Pros === * The services see 0 changes. Everything happens transparently to them. * The move of traffic is gradual allowing to rollback quickly and easily, as well as pause the migration * No downtime for services === Cons === * Rather convoluted configuration wise, with some margin for mistakes * All or nothing approach as far as services go. No way to distinguish between them * The migration might take a long time as when issues arise they will probably be global blockers for all services * Rollbacks are possible, but if issues arise, it's probably going to be a full rollback to the old installation * The ending restbase.svc.$::site.wmnet DNS does not reflect the actual software powering the frontend, aka restrouter possibly leading to future misunderstandings/confusion = Post migration = In the post-deploy clean-up step, we remove public route handling from RESTBase, effectively turning it into the back-end storage service. Comment from Giuseppe: I think plan 1 is much simpler. It requires more patches and more attention to not leave anything behind, but it's probably the better plan. Please be mindful that restrouter will need to be terminating SSL as well, like restbase does. I vote plan 1. Marko: * my vote goes for plan 1 as well, even though it will probably take longer, it makes it clear to all parties involved that changes are happening; that means that also service owners will be more aware in case of problems so they will be easier to detect * i agree that the end result is better with restrouter.svc than restbase.svc RESTRouter will effectively take over request handling from RESTBase, so we will need to divert traffic to it without interruption. Benchmarking: - https://wikitech.wikimedia.org/wiki/User:Alexandros_Kosiaris/Benchmarking_kubernetes_apps - P8425

We are splitting RESTBase in two components - the (public) REST API router and the storage service (cf. {T220449}). This task is about deploying the front-end REST router in Kubernetes. === Service Info = **Service name**: RESTRouter (name still under discussion, cf. T220761) **Owners**: @Pchelolo and @mobrovac (#core_platform_team) **Repository**: `mediawiki/services/restbase` **ETA**: by the end of Q4 FY18/19 **Description**: RESTRouter is the routing part of (the current) RESTBase. It accepts external requests, validates them (performs access checks if needed) and performs all the business logic related to the request: it looks up the storage for possible data hits and, if needed, issues requests to back-end services to complete the requests, sending the response to storage prior to returning it to the client. == Deployment Plan Deployment Plans = Restrouter migration plans. Some parts are the same for all plans. Those are listed below * First, we deploy RESTRouter to k8s. * we expose the storage routes in RESTBase (cf. [PR #1103](https://github.com/wikimedia/restbase/pull/1103)) * test RESTRouter for load (options include synthetic traffic, mirroring, using only background updates/internal requests). == Plan 1 == Have restbase listen on both 7231 and 7233 and configure LVS restbase.svc.$::site.wmnet to also use 7233 RESTRouter will effectively take over request handling from RESTBase,Instantiate restrouter on a new LVS IP and DNS (restrouter.svc.$::site.wmnet) and have it talk to restbase.svc.$::site.wmnet:7233 Move services 1 by 1 to restrouter.discovery.wmnet (the site aware discovery records for restrouter.svc.$::site.wmnet) === Pros === * Move is gradual on a service level. Services are migrated one by one based on their configuration unearthing potential problems one by one * The currently stable and battle tested restbase installation is kept around even while more and more services are moved around * It's rather easy configuration wise, rather easy to do in steps * No downtime for services. === Cons === * The migration might take time as when issues arise, but at least blockers will be service specific * There is no gradual traffic switchover. For every service it's a "canary host first", then all or nothing approach. so we will need to divert traffic to it without interruption.Even the canary host is depending on DC between 13% and 25% of traffic == Plan 2 == Have restbase listen on both 7231 and 7233 Add a new LVS IP on the restbase hosts and name it restbase-backend.svc.$::site.wmnet Configure restrouter to connect to restbase-backend.svc.$::site.wmnet:7233 Add the LVS IP for restbase.svc.$::site.wmnet to kubernetes hosts Add the kubernetes hosts to LVS for restbase.svc.$::site.wmnet Slowly migrate the traffic from the current restbase hosts to kubernetes hosts First, we deploy RESTRouter to k8s. Next, we expose the storage routes in RESTBase (cf. [PR #1103](https://github.com/wikimedia/restbase/pull/1103)) and test RESTRouter for load (options include synthetic traffic, mirroring, using only background updates/internal requests). Then, we include RESTRouter as an additional back-end for RESTBase. We do more load-tests. Finally, a new LVS is created for RESTRouter and Varnish and all back-end services are instructed to use the new LVS end point instead of the current one. In the post-deploy clean-up step, we remove public route handling from RESTBase, effectively turning it into the back-end storage service.=== Pros === * The services see 0 changes. Everything happens transparently to them. * The move of traffic is gradual allowing to rollback quickly and easily, as well as pause the migration * No downtime for services === Cons === * Rather convoluted configuration wise, with some margin for mistakes * All or nothing approach as far as services go. No way to distinguish between them * The migration might take a long time as when issues arise they will probably be global blockers for all services * Rollbacks are possible, but if issues arise, it's probably going to be a full rollback to the old installation * The ending restbase.svc.$::site.wmnet DNS does not reflect the actual software powering the frontend, aka restrouter possibly leading to future misunderstandings/confusion = Post migration = In the post-deploy clean-up step, we remove public route handling from RESTBase, effectively turning it into the back-end storage service. Comment from Giuseppe: I think plan 1 is much simpler. It requires more patches and more attention to not leave anything behind, but it's probably the better plan. Please be mindful that restrouter will need to be terminating SSL as well, like restbase does. I vote plan 1. Marko: * my vote goes for plan 1 as well, even though it will probably take longer, it makes it clear to all parties involved that changes are happening; that means that also service owners will be more aware in case of problems so they will be easier to detect * i agree that the end result is better with restrouter.svc than restbase.svc RESTRouter will effectively take over request handling from RESTBase, so we will need to divert traffic to it without interruption. Benchmarking: - https://wikitech.wikimedia.org/wiki/User:Alexandros_Kosiaris/Benchmarking_kubernetes_apps - P8425