This is a summary of the Slack discussion between Eric Evans and myself that led to the creation of this task. @Eevans , please correct anything I misrepresented, or elaborate on anything you feel needs more detail.

• the previous router (httprouter) was compatible with the standard (net/http) library, which facilitated more easily switch out router packages if an issue was encoutered
• fashttp (at least the way we're using it) is not compatible. Compare this handler (from when we were using httprouter) to this one (with fasthttp). The fasthttp one is fasthttp-specific, meaning that if we switched routers again, we'd have to change all the handlers. The httprouter one is not, meaning that we could switch router packages to any other compatible router without changing handlers. In other words, using an incompatible interface now may complicate our future choices
• open question: is there a way to use fasthttp in a compatible way?
• there is a proliferation of http abstractions for Go, perhaps indicating an insufficiency in the standard abstractions (otherwise why would so many people create more?). However, is there any way in which the standard abstractions do not meet our actual needs?
• while changing routers again would not be convenient, it would be much easier now (even considering just AQS 2.0 and not other projects) than it would be once we finish and deploy the services- changing abstractions on a WMF-wide basis once numerous services use whatever pattern is present in the Go service scaffolding might be a significant effort lasting years
• when we previously changed to fasthttp, we had more limited testing. We now have much more extensive testing, including Emeka's test suite, which would give us more confidence against regressions if we choose to again change routers
• our decision to switch away from httprouter seems solid. Returning to that project does not appear to be an option
• fasthttp itself says, in its own documentation, "For most cases net/http is much better as it's easier to use and can handle more cases. For most cases you won't even notice the performance difference." The bit about performance difference is a little scary, given WMF's scale.

In T328166#8565674, @BPirkle wrote:

This is a summary of the Slack discussion between Eric Evans and myself that led to the creation of this task. @Eevans , please correct anything I misrepresented, or elaborate on anything you feel needs more detail.

Having worked on the OG (Kask, the first golang service at wmf), the earliest iteration(s) of service-scaffold-golang & servicelib-golang, and AQS 2.0, I might also be able to provide some historical context.

This is touched on below, but there is a lot of proliferation when it comes to HTTP frameworks & abstractions, and yet it is still very common/possible (and the recommendation de jour in some circles) to simply use net/http. Kask is an example of a service that uses no third-party abstractions.

When we spun up the first of the AQS 2.0 services though, an early issue that came up was routing (aka "mux"). Go's standard library doesn't include any abstractions for routing based on HTTP method, or parsing resources (which is pretty standard stuff). There is nothing to stop you from implementing this, but I confess that it doesn't feel like something you should have to. We did some research and found httprouter. It seemed perfect at the time: It was very minimal in scope and didn't have an extensive graph of transitive dependencies, and it was ubiquitous, even serving as the underlying mux for a number of popular full-featured frameworks. More importantly, it allowed us to fill in the missing piece —routing— while implementing our endpoints using the http.Handler interface. Alas there were issues with httprouter, and as I understand it, concerns about the lack responsiveness upstream and as a result the long-term viability of the project.

• the previous router (httprouter) was compatible with the standard (net/http) library, which facilitated more easily switch out router packages if an issue was encoutered

Correct. The handlers are where the bulk of our code lives. If you are tightly coupled to a third-party library that later needs to be replaced it'll require what may be a massive undertaking (it may for all intents and purposes be entirely impractical). The router (mux) is where that code is wired up to resources (typically a few lines of main main()).

• fashttp (at least the way we're using it) is not compatible. Compare this handler (from when we were using httprouter) to this one (with fasthttp). The fasthttp one is fasthttp-specific, meaning that if we switched routers again, we'd have to change all the handlers. The httprouter one is not, meaning that we could switch router packages to any other compatible router without changing handlers. In other words, using an incompatible interface now may complicate our future choices

I'd treat the need to someday deprecate fasthttp as a given; I'd think of it in terms of when rather than an if. If you deem this overly cynical, I'd point you to...well, httprouter. 😁

• open question: is there a way to use fasthttp in a compatible way?
• there is a proliferation of http abstractions for Go, perhaps indicating an insufficiency in the standard abstractions (otherwise why would so many people create more?). However, is there any way in which the standard abstractions do not meet our actual needs?

As I alluded to above, the only concrete ask I've had (so far) is for a fuller featured mux. I see (for example) templating and database connectivity as out of scope, and middleware is easily implemented by chaining handlers. I think at least some of these are an attempt at recreating someone's favorite interface from another language.

• while changing routers again would not be convenient, it would be much easier now (even considering just AQS 2.0 and not other projects) than it would be once we finish and deploy the services- changing abstractions on a WMF-wide basis once numerous services use whatever pattern is present in the Go service scaffolding might be a significant effort lasting years
• when we previously changed to fasthttp, we had more limited testing. We now have much more extensive testing, including Emeka's test suite, which would give us more confidence against regressions if we choose to again change routers
• our decision to switch away from httprouter seems solid. Returning to that project does not appear to be an option

FWIW, there are other muxes that support the Handler interface (https://github.com/gorilla/mux for example).

• fasthttp itself says, in its own documentation, "For most cases net/http is much better as it's easier to use and can handle more cases. For most cases you won't even notice the performance difference."

This seems pretty damning.

The bit about performance difference is a little scary, given WMF's scale.

Allow me to assuage your concerns then. 😉

Have a look at the session storage dashboard. It's doing on the order of something like 2k/s request. 96% of the requests are between 1-2.5ms latency, and each includes a request to Cassandra(!) It handles a lot of requests, with not just low latency, but very predictable latency (a very narrow distribution of latencies), and it does this with modest resource utilization.

On criteria for evaluating routers, here are a few things off the top of my head that we could consider.

• actively maintained (as evidenced by recent commits and reasonably prompt evaluation of pull requests)
• performant (not sure how best to evaluate this)
• compatibility with standard interface
• popularity (somewhat dubious, but not entirely irrelevant. If lots of people use a router, there is probably a reason, and it is also less likely to be abandoned)
• features (path variables? regexp-based routing? host-based routing? custom routing rules? how does it handle conflicts? And so on.)

To be clear, these are evaluation criteria, not requirements, and should be interpreted in the overall context of our needs. For example, suppose one of the routers scores very highly on everything except compatibility. We might decide that poor compatibility does not outweigh the other benefits, and choose to use it anyway. On the other hand, if one of the routers scores very poorly on performance, we probably won't use it no matter how good it is at everything else.

Also, we may need to consider which features we care about and which we don't. For example, path variables seem like a must-have, but I'm less convinced on host-based routing.

What other criteria should we add to the list?

In T328166#8576921, @BPirkle wrote:

On criteria for evaluating routers, here are a few things off the top of my head that we could consider.

• actively maintained (as evidenced by recent commits and reasonably prompt evaluation of pull requests)
• performant (not sure how best to evaluate this)

My 2¢ would be to all but drop this from the list of requirements. Unless it's horribly broken, it's going to be fast enough (elevating literally every other concern). @Clarakosi did extensive performance testing of NodeJS & Python frameworks in the run up to Kask. We chose Go then because session storage was very sensitive to latency, and it vastly outstripped the best performance we could manage otherwise.

• compatibility with standard interface
• popularity (somewhat dubious, but not entirely irrelevant. If lots of people use a router, there is probably a reason, and it is also less likely to be abandoned)
• features (path variables? regexp-based routing? host-based routing? custom routing rules? how does it handle conflicts? And so on.)

Regexp sounds like a YAGNI to me. The need for matching with that granularity seems like it would be an indication of poorly chosen resources (though I'd be totally open to being shown otherwise).

To be clear, these are evaluation criteria, not requirements, and should be interpreted in the overall context of our needs. For example, suppose one of the routers scores very highly on everything except compatibility. We might decide that poor compatibility does not outweigh the other benefits, and choose to use it anyway. On the other hand, if one of the routers scores very poorly on performance, we probably won't use it no matter how good it is at everything else.

Also, we may need to consider which features we care about and which we don't. For example, path variables seem like a must-have, but I'm less convinced on host-based routing.

Agreed; Given our focus on micro-services, I can't think of a scenario where the use of host-based routing would be necessary.

What other criteria should we add to the list?

Having worked in Go for few years and used both net/http , I would like to speak about somethings which I feel are limitations of nethttp :-
Context support in request
To make a request with a context , we need to use copy of a Request with Request.WithContext. This is a new request pointer which can not change the original *http.Request.
Routing
Like @Eevans correctly pointed

routing (aka "mux"). Go's standard library doesn't include any abstractions for routing based on HTTP method, or parsing resources (which is pretty standard stuff)

No default timeouts in clients
This problem arises mainly in talking to remote services , and because of no deafult time-out , it may cause the application to hang.
Headers
net/http headers are stored in a map[string][]string. So the server must parse all the headers, convert them from []byte to string and put them into the map before calling user-provided request handler

While fasthttp also solved our long standing encoding issues , other advantages I feel fasthttp has are -

• Really fast for small to medium requests. A benchmarking done on this link -

https://morioh.com/p/aa306d9f8d29

• The RequestHandler accepts only one argument - RequestCtx. It contains all the functionality required for http request processing and response writing.

• The worker pool zero allocation model

The workers are already initialized and can be used and served to request .

Now coming to one of the open questions about compatible way of using the fasthttp with net/http
I had like to mention since the way of working is completely different from that of net/http , we can't change the implementation much but would like to mention of this adaptor for converting the net/http to fasthttp but not the other way around
https://pkg.go.dev/github.com/valyala/fasthttp/fasthttpadaptor

The criterion @BPirkle mentioned for choosing routers seems perfect to me and would like to know that should we go ahead and start comparing some of the popular web frameworks for go now?

In T328166#8636733, @SGupta-WMF wrote:

Having worked in Go for few years and used both net/http , I would like to speak about somethings which I feel are limitations of nethttp :-
Context support in request
To make a request with a context , we need to use copy of a Request with Request.WithContext. This is a new request pointer which can not change the original *http.Request.

I'm not sure I understand the problem here, requests by their very nature should be immutable, shouldn't they? What problems are created by net/http and/or solved by fasthttp?

[ ... ]

No default timeouts in clients
This problem arises mainly in talking to remote services , and because of no deafult time-out , it may cause the application to hang.

Could you expound on this? What is the client in this instance, what is the remote service? Are we talking about initiating connections to remote services from within an HTTP handler?

Headers
net/http headers are stored in a map[string][]string. So the server must parse all the headers, convert them from []byte to string and put them into the map before calling user-provided request handler

Can you expound on this one too? Since headers are inherently a string/string associative array, this seems like the correct behavior.

While fasthttp also solved our long standing encoding issues , other advantages I feel fasthttp has are -

• Really fast for small to medium requests. A benchmarking done on this link -

https://morioh.com/p/aa306d9f8d29

• The RequestHandler accepts only one argument - RequestCtx. It contains all the functionality required for http request processing and response writing.

• The worker pool zero allocation model

The workers are already initialized and can be used and served to request .

Performance is fasthttp's raison d'être (see https://github.com/valyala/fasthttp#faq); I have no doubt that it performs better. The question is, do we need the additional performance, and what are we willing to give up in exchange? Our most performance-sensitive service (Kask/session storage) is written in Go using net/http and its performance is widely regarded to be exceptional.

And again, the fasthttp folks themselves make this very same argument, prominently, in the first paragraph of their README:

For most cases net/http is much better as it's easier to use and can handle more cases. For most cases you won't even notice the performance difference.

Now coming to one of the open questions about compatible way of using the fasthttp with net/http
I had like to mention since the way of working is completely different from that of net/http , we can't change the implementation much but would like to mention of this adaptor for converting the net/http to fasthttp but not the other way around
https://pkg.go.dev/github.com/valyala/fasthttp/fasthttpadaptor

Ostensibly, this would solve my concerns¹, but quoting the docstrings, this doesn't seem to be what their intended for:

While this function may be used for easy switching from net/http to fasthttp, it has the following drawbacks comparing to using manually written fasthttp request handler:

• A lot of useful functionality provided by fasthttp is missing from net/http handler.
• net/http -> fasthttp handler conversion has some overhead, so the returned handler will be always slower than manually written fasthttp handler.

So it is advisable using this function only for quick net/http -> fasthttp switching. Then manually convert net/http handlers to fasthttp handlers according to https://github.com/valyala/fasthttp#switching-from-nethttp-to-fasthttp .

¹: To reiterate those concerns here: Go's http handler interface is idiomatic/ubiquitous, and (IME) capable. It is for all intents and purposes, The Standard™ (think WSGI for Python). Since this is where the vast majority of code we would write lives, the use of a standard interface like this safeguards the future. If we're tightly coupled to a framework-specific interface, we are tightly coupled to that framework. Given how resource constrained we are, this does not bode well if circumstances require us to make a change at some point in the future (and they will). To be clear, I'm not saying that we need to use net/http, only the http handler interface (which many frameworks honor).

The criterion @BPirkle mentioned for choosing routers seems perfect to me and would like to know that should we go ahead and start comparing some of the popular web frameworks for go now?

I'd like to see that comparison.

I compared some popular routers and have summarised the comparison here :-
gin-gonic/gin:

• It is fast and efficient due to its use of the httprouter library for request routing.
• It has a simple and intuitive API that makes it easy to use for developers.
• It provides middleware support for handling common tasks such as logging, authentication, and compression.
• It has a large community and extensive documentation.
• Over 50K stars on github
• Developers can use the standard http.Handler interface to create handlers that can be registered with this router.

gorilla/mux:

• It provides a powerful and flexible URL router that can handle complex request matching and URL generation.
• It has middleware support for handling common tasks such as logging, authentication, and CSRF protection.
• It supports a wide range of HTTP methods and headers.
• It has a large and active community.
• It has over 13k stars on GitHub
• Developers can use the standard http.Handler interface to create handlers that can be registered with this router.

labstack/echo:

• It is a lightweight and fast framework that is designed for building RESTful APIs.
• It has a simple and intuitive API that makes it easy to use for developers.
• It provides middleware support for handling common tasks such as logging, authentication, and CSRF protection.
• labstack/echo is a relatively new framework but with growing community members
• It has a small memory footprint and is suitable for running on low-end hardware.
• It has over 17k stars on GitHub
• Developers can use the standard http.Handler interface to create handlers that can be registered with this router.

fasthttp:

• It is a high-performance HTTP server library that is designed for serving HTTP requests quickly and efficiently.
• It provides a simple and lightweight API for handling HTTP requests and responses.
• It has a small memory footprint and is suitable for running on low-end hardware.
• It supports HTTP/1.x and HTTP/2 protocols.
• fasthttp is a library rather than a full-fledged framework, but it is also actively maintained and receives regular updates.
• the fasthttp repository on GitHub had over 18.7k stars
• Fasthttp provides its own implementation of the HTTP server and client libraries, which are not fully compatible with the net/http package

@BPirkle @Eevans Coming to the performance comparison bit , should we do any benchmarking or is there any other suggestion for evaluating performance?

Coming to the performance comparison bit , should we do any benchmarking or is there any other suggestion for evaluating performance?

I'm not aware of a canonical way of benchmarking performance in this situation (I'd love to be proven wrong). Erik suggested dropping that from the evaluation criterion, and I'm good with that as well. All the packages we're comparing are used widely enough that I'd be very surprised if there were significant enough differences to dictate our choice. But if we want to do a quick informal comparison just to make sure there are no extreme differences, here's how someone else did it. We might be able to borrow their approach.

On continuing the comparision, we have already established that the big drawback of fasthttp is that it does not comply to the standard interface. I suggest we compare the other three (labstack/echo, gorilla/mux, and gin-gonic/gin), pick our favorite, and compare it against fasthttp.

Transforming the bulleted list above into a table, in roughly the order they were mentioned in the bulleted lists:

I may have misrepresented some nuances of the differences between routers in this summary. If you see that I have done so in a meaningful way, please call that out.

Based on this table (and Surbhi's above comment) my impression is:

• gorilla/mux is the most powerful and flexible in terms of its capabilities, but that comes at the cost of complexity, memory footprint, and perhaps performance
• gin is an excellent choice for projects with straightforward needs
• echo's major benefit over gin is that echo is more established

Here are some urls I looked at for more info:

• https://mattermost.com/blog/choosing-a-go-framework-gin-vs-echo/#:~:text=The%20main%20difference%20between%20Gin,using%20Gin's%20built%2Din%20functions.
• https://blog.logrocket.com/building-microservices-go-gin/#:~:text=Gorilla%20mux%20doesn't%20offer,%2C%20authentication%2C%20and%20session%20management.
• https://markphelps.me/posts/4-things-to-consider-when-choosing-a-go-api-framework/

Disclaimer: I cannot vouch for the accuracy of any of these links. It is possible that they are dated, incorrect or biased.

The last link make the point that gorilla/mux is more of a "router", while gin and echo are more "frameworks". I'm not convinced that's a bad thing.

If we were selecting a router/framework for a single project with a focused, well-defined need, evaluation would be easier - we compare the needs of our project to our options. But we aren't doing that. We're selecting a router/framework for our scaffolding/servicelib, to be used across WMF for as-yet-undefined projects. This increases the importance of making a selection that can handle not only our current needs, but has the highest likelihood of handling unknown future needs.

Stated the other way around, it seems important that we pick something unlikely to be a blocker for future projects whose requirements are not yet known.

Anyone see any errors in what I just said, have different opinions, or further thoughts?

In T328166#8681950, @BPirkle wrote:

Coming to the performance comparison bit , should we do any benchmarking or is there any other suggestion for evaluating performance?

[ ... ]

If we were selecting a router/framework for a single project with a focused, well-defined need, evaluation would be easier - we compare the needs of our project to our options. But we aren't doing that. We're selecting a router/framework for our scaffolding/servicelib, to be used across WMF for as-yet-undefined projects. This increases the importance of making a selection that can handle not only our current needs, but has the highest likelihood of handling unknown future needs.

I think you may have (better) articulated something here that I've glossed over in my previous responses. This is why I've tended to advocate for a more minimalist approach, because it is easier to add higher level abstractions later (or perhaps even on an as-needed basis), than it is to front-load them early on, and hope those decisions hold up as we gain more experience in our environment, and/or new requirements emerge.

Stated the other way around, it seems important that we pick something unlikely to be a blocker for future projects whose requirements are not yet known.

Anyone see any errors in what I just said, have different opinions, or further thoughts?

I see no errors.

We are currently nearing staging deployment of our first AQS 2.0 service and will soon be prepping the second for deployment. So while I don't want to rush a decision, it'd also be good to make it soon as consequences are about to start accumulating.

In the interest of moving the discussion forward, and at risk of applying Cunningham's Law, my personal preference right now is gorilla/mux. It seems least likely to be a blocker for future projects. However, I confess that I don't have deep knowledge of any of these, so if you disagree please post your thoughts.

To try making this more real, I played around with all four of these in a personal github repo. I had the idea of trying to do rough benchmarks, but getting my hands dirty with each is a benefit of its own, even if the benchmarking doesn't work out. The repo is still a mess, but it is getting late here so I'm stopping for the night. Will try to pick it up again in the next day or two.

I finished up the 4-way comparison and ran some benchmarks. Here are the numbers, analysis and thoughts follow.

Echo:

./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.19ms    1.25ms  48.54ms   85.10%
    Req/Sec     9.22k     2.02k   32.04k    81.15%
  7340152 requests in 1.67m, 840.01MB read
  Socket errors: connect 781, read 109, write 0, timeout 0
Requests/sec:  73348.00
Transfer/sec:      8.39MB

Gin:

./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.22ms    1.20ms  44.12ms   86.80%
    Req/Sec     9.14k     2.00k   40.29k    82.67%
  7275080 requests in 1.67m, 825.63MB read
  Socket errors: connect 781, read 111, write 0, timeout 0
Requests/sec:  72694.76
Transfer/sec:      8.25MB

Mux:

./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.05ms    1.48ms  39.00ms   80.53%
    Req/Sec     9.59k     2.25k   34.83k    82.77%
  7636416 requests in 1.67m, 866.64MB read
  Socket errors: connect 781, read 108, write 0, timeout 0
Requests/sec:  76293.65
Transfer/sec:      8.66MB

Fasthttp:

./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.88ms    1.05ms  45.84ms   92.85%
    Req/Sec    15.46k     3.59k   52.41k    79.53%
  12310574 requests in 1.67m, 1.58GB read
  Socket errors: connect 781, read 109, write 0, timeout 0
Requests/sec: 122989.37
Transfer/sec:     16.19MB

As a baseline, I also did a version using just net/http:

./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.16ms    1.29ms  36.61ms   84.42%
    Req/Sec     9.28k     2.24k   40.95k    87.67%
  7387193 requests in 1.67m, 845.40MB read
  Socket errors: connect 781, read 109, write 0, timeout 0
Requests/sec:  73820.86
Transfer/sec:      8.45MB

Putting that into table form:

Disclaimer: this is a TERRIBLE way to do benchmarking. My laptop is absolutely not a reliable benchmarking platform. I ignored the Max and StdDev values, because I can't imagine they're representative of server behavior. Benchmarking an index endpoint with no middleware that returns a hard-coded string could miss all sorts of relevant differences between the packages. These benchmarks should be used only to draw the roughest of comparisons.

With that said, these results show that fasthttp is indeed fast by comparison, with all other options being virtually indistinguishable on speed. However, that difference is less than 2x, not 10x or 100x.
Do we care? We already feel like Kask is Fast Enough (TM), so I'm guessing the answer is "no". But it is still worth noting.

Also worth noting is that both echo and gin require wrapping handlers to use the standard interface. In other words, echo would prefer I did:

func Index(c echo.Context) error {
	return c.String(http.StatusOK, "echo")
}

instead of the standard interface version:

func Index(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	fmt.Fprintf(w, "echo")
}

To use the standard interface, I had to do this:

router.GET("/", echo.WrapHandler(http.HandlerFunc(Index)))

WrapHandler basically just pulls the response writer and the request out of the context and passes them to a handler of the standard signature. In so doing, unless I'm missing something, it discards much of the framework-ey goodness that would make someone choose echo in the first place. In other words, by forcing echo to use standard handlers, we'd be bypassing much of echo's value. If we indeed prioritize the standard (a position I do not disagree with), we have to be careful about which of echo's features we consider in our comparison. Some of its benefits may not apply.

Gin is the same as Echo in this regard, the wrapping syntax is just a bit different:

router.GET("/", gin.WrapF(Index))

I confess that I'm a novice to much of this, but I don't see any meaningful difference between these wrappers and the fasthttp adaptor. Gin, Echo, and Fasthttp all seem the same in this regard: use custom handlers and get the full benefit of the framework, or use standard handlers and get less framework benefit (plus a probably-negligible amount of overhead).

All this leaves me still in support of gorilla/mux. But I'm still happy to hear other arguments.

I decided to also benchmark the fasthttpadaptor approach. And I rebenchmarked fasthttp immediately afterward, to make sure there were no substantial differences in the local env I'm running all this in since I ran the previous benchmarks.

~/go/src/go-router-benchmarks-2023 % ./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.87ms    0.88ms  39.37ms   88.23%
    Req/Sec    15.35k     6.74k   56.73k    70.40%
  12220020 requests in 1.67m, 1.56GB read
  Socket errors: connect 781, read 103, write 0, timeout 0
Requests/sec: 122101.34
Transfer/sec:     15.95MB

~/go/src/go-router-benchmarks-2023 % ./wrk -d 100s -c 1024 -t 8 http://localhost:8080/
Running 2m test @ http://localhost:8080/
  8 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.89ms    1.11ms  45.71ms   93.68%
    Req/Sec    15.40k     3.34k   69.22k    84.17%
  12262961 requests in 1.67m, 1.58GB read
  Socket errors: connect 781, read 111, write 0, timeout 0
Requests/sec: 122531.32
Transfer/sec:     16.13MB

As expected, the adaptor adds essentially no overhead. Which raises the possibility that, given that fasthttp appears to indeed be faster, at least in these possibly-misleading measurements, we could use it with the adaptor. That'd give us better performance and still give us the benefit of standards-compliant handlers. It would require us to maintain the discipline of choosing to NOT use the "extras" that fasthttp provides, but that seems no different than with echo or gin.

The advantage of mux still seems to be that we don't have option of doing things that tie us to it in an undesirable way, because it just doesn't offer nonstandard handlers. So using mux means that teams can't mess that up through accident or lack of awareness of the consequences.

Of course, the disclaimer from my previous comment still applies. It may be that fasthttp's apparent benefit is not as large as it seems when executed in a real situation . Or it might be larger.

Thoughts welcome.

So we are ultimately down with two choices Mux and fasthttp . And it really depends what our use case is .
While Mux provides a lot of flexibility in terms of HTTP routes and handlers I feel its more suited for building large and complicated applications . Fasthttp on the other hand is lightweight and is more suitable for large volumes of requests with low latency.
@BPirkle @Eevans Any preferences looking at WMF scale and applications?

In T328166#8733945, @SGupta-WMF wrote:

So we are ultimately down with two choices Mux and fasthttp . And it really depends what our use case is .
While Mux provides a lot of flexibility in terms of HTTP routes and handlers I feel its more suited for building large and complicated applications . Fasthttp on the other hand is lightweight and is more suitable for large volumes of requests with low latency.
@BPirkle @Eevans Any preferences looking at WMF scale and applications?

I really think that with respect to being "lightweight" and/or high performance, we're deep into premature optimization territory (and not to belabor this point, but the authors of fasthttp would seem to agree; our use of fasthttp would seem to go against their recommendations). My preference would be to practice JIT engineering, solving the problems in front of us, and delaying the others until the last possible moment, when more informed decisions can be made. Mux (AFAICT), solves the problem in front of us (mux/routing), and does so in a modular fashion that preserves flexibility for those future decisions.

That said, my primary concern was to avoid using interfaces that would unnecessarily limit our choices in the future. Handlers is where the bulk of the important code in an HTTP service lies, and tightly coupling them to a specific framework seemed like a bad idea. If we have a plan to avoid this, then I reckon I'm less passionate about the remaining concerns.

Our biggest risk is locking ourselves into a solution that later proves to be a major blocker. Using standard handlers avoids that.

Our second biggest risk is picking a solution that we're not happy with in the long run, and having to switch. Using standard handlers doesn't make switching to a different solution trivial - it might still be significant work. At a minimum it would involve revalidating everything affected.

We already have a traffic-intensive service (Kask) in production using a router with performance characteristics that appear comparable to mux. And by all accounts we're happy with it. The only reason we're even having this conversation right now is that we found a functionality limitation in the router we were previously using for AQS 2.0. History therefore suggests that functionality is more likely to be a blocker than performance, not because we don't care about performance but because all our options are sufficiently fast.

We do tend to build large and complicated applications around here. While I'd love to say that we won't do that anymore, I suspect we probably will. And even in smaller applications, we as an organization tend to push the functionality limits of whatever technology we use.

I vote mux.

So I went ahead and checked how the compatible mux with AQS 2.0 . I has support for routing and encoding needed in AQS 2.0 . Mux also supports middleware ,so the current middlewares can also be retained .
The conversion from fasthttp to mux is also starightforward but does require change in lot of places including handlers, logic , tests. So that would require some effort.

@BPirkle Since , we have a consensus for mux , should we go ahead and start implemetation in our AQS services in the next sprint.

@BPirkle Since , we have a consensus for mux , should we go ahead and start implemetation in our AQS services in the next sprint.

Yep. I've created tasks for the four Cassandra-based services. The Druid-based ones are still pretty formative, so I'm assuming that when we get to them, we'll basically start over with a copy of Device Analytics and make necessary adjustments, and they therefore won't need a conversion tasks. Feel free to make any adjustments to the tasks that you see fit.