Important (performance) factors to keep in mind when creating HTTP web services (with or without libSXE):
1. Accept()ing new sockets is an expensive and generally single-threaded kernel operation. Therefore it’s important to consider having as many long-lived ‘keep-alive’ HTTP 1.1 connections as possible. For example, your multi-threaded HTTP server might be able to have 500,000 simultaneous HTTP connections and perform at 250,000 keep-alive requests per second, but the server will still only be able to accept() new connections at a rate of 30,000 per second.
3. The general rule of thumb is that the HTTP server nearly never has to close() a socket. The close() function is not as slow at the accept() function but it’s not for free either. Using close() might just lead to a malicious peer to connect again causing an expensive accept(). So assuming your server has a lot of memory then it’s probably a better trade off to just keep a badly behaving HTTP peer connection, and maybe even ‘tar pit’ it etc. The idea is that a little bit of memory for the peer buffers is a lesser evil than the potentially expensive close()/accept() bottleneck.
4. Another reason not to close() is that on very heavily loaded networks then there is no guarantee that the TCP packets generated by a close() will even reach a peer. In fact it doesn’t even have to be a heavily loaded network. For example, a client which suddenly powers off and had an idle keep-alive connection to a server will not inform the server of the close; therefore the server will continue to believe that the TCP connection is still alive unless it tries to send data to the client. So a good rule of thumb is not to rely on the close event — generated when a peer close()s — for cleaning up the resources / states / etc belonging to a particular HTTP connection. So how else to clean up a connection if the server never close()s and we cannot 100% rely on the close event? The only answer which makes sense is to timeout the particular connection. But there is an even simpler way; just close and reuse the oldest TCP connection. For example, the server is configured to handle a maximum of 500,000 simultaneous connections. When we want to accept() the 500,001 connection then we simply close() the oldest connection in the connection pool. Chances are that we’re just reaping one of those ‘zombie’ connections.