Understanding SO_REUSEPORT and TCP_CORK Through a Coffee Shop Analogy
by 
gg582 · 2026-09-13 16:50:43 · 9 views
Table of contents
This post is not generated by AI. And I am not a native speaker of English. I haven't learned professional English education like many Korean developers do; I just live in a small South Korean city. Therefore, there are many grammar mistakes. I only used a grammar checker. Some sentences are unnatural.
How can we assume the problem?
Assume you should make eight cups of coffee.
You have single-serving Hario V60 drippers and 8-serving Hario V60 drippers.
When you try single-serving V60 drippers with eight baristas, it takes less than giving a dripper to a senior barista.
But each barista should prepare for each extraction when many orders are waiting per person since a dripper is small.
When 16 cups of coffee should be prepared, it can't be done so fast like we've already done.
When you use a small dripper eight times, you need more time. But when you use a big dripper, huge amounts are extracted for just one time.
There's an important point. When each barista just uses a bigger dripper, they can extract much more than using a small one.
But, when a dripper is too big, each customer should wait more to finish that extraction.
Hiring more baristas means SO_REUSEPORT. Many 'sockets' can use the same IP and a port number.
Giving a bigger dripper can extract more amounts at once. Basically more water and more bean powders are collected. This means TCP_CORK.
Using TCP_CORK to a broad range without SO_REUSEPORT can worsen average RTT. Also, using too many sockets after applying SO_REUSEPORT can drain Unix sockets. Simply enabling both increases two metrics at once: RPS and network transfer rate. When each barista is using a big dripper, if somebody's extraction ends up early, eight customers can get their cups. Naturally, that barista can prepare for other orders right after those cups.
How can we experiment with this with a real code? Let's dive into this.
Ran wrk -t10 -c100 -d30s http://localhost:8080/.
Source
#define _GNU_SOURCE 1
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
#include <unistd.h>
#include <signal.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <netinet/tcp.h>
#define WORKER_POOL 4
#define TCP_PORT 8080
/// set 1 to turn on each options.
/// set 0 to turn off.
typedef struct Options {
int so_reuseport;
int tcp_cork;
} Opt_t;
/**
* @brief check errno before writing to a client socket.
*/
static inline bool writeCheck(int c_fd, int n) {
switch((int)(n < 0)) {
case 0:
return true;
break;
case 1:
switch(errno) {
case EPIPE:
case ECONNRESET:
fprintf(stderr, "client disconnected a connection: errno(%d)\r\n", errno);
return false;
case EINTR:
return true;
default:
fprintf(stderr, "crash during write call: errno(%d)", errno);
return false;
}
}
}
/**
* @brief implement a client loop.
* @param s_fd server-side file descriptor
* @param opt options including TCP_CORK and SO_REUSEPORT
*/
void clientLoop(int s_fd, Opt_t opt) {
while(true) {
int c_fd = accept(s_fd, NULL, NULL);
if(c_fd < 0) continue;
// serve requests on this connection until the client hangs up.
while(true) {
char req[4096];
ssize_t r = read(c_fd, req, sizeof(req));
if(r <= 0) break;
switch((int)(opt.tcp_cork)) {
case 0:
// writes small chunks
int n = write(c_fd, "HTTP/1.1 200 OK\r\n", 17);
if(!writeCheck(c_fd, n)) goto done;
n = write(c_fd, "Content-Type: text/plain\r\nContent-Length: 42\r\n\r\n", 49);
if(!writeCheck(c_fd, n)) goto done;
n = write(c_fd, "Data payload chunk 1\n", 21);
if(!writeCheck(c_fd, n)) goto done;
n = write(c_fd, "Data payload chunk 2\n", 21);
if(!writeCheck(c_fd, n)) goto done;
break;
case 1:
int cork = 1;
// wait until cork == 0
setsockopt(c_fd, IPPROTO_TCP, TCP_CORK, &cork, sizeof(cork));
write(c_fd, "HTTP/1.1 200 OK\r\n", 17);
write(c_fd, "Content-Type: text/plain\r\nContent-Length: 42\r\n\r\n", 49);
write(c_fd, "Data payload chunk 1\n", 21);
write(c_fd, "Data payload chunk 2\n", 21);
cork = !cork;
setsockopt(c_fd, IPPROTO_TCP, TCP_CORK, &cork, sizeof(cork));
break;
default:
break;
}
}
done:
// close client fd
close(c_fd);
}
}
/**
* @brief run worker loop.
* @param w_id worker-side index
* @param opt options including TCP_CORK and SO_REUSEPORT
*/
void runWorker(int w_id, Opt_t opt) {
// A TCP socket to bind
printf("worker %d has started\r\n", w_id);
int s_fd = socket(AF_INET, SOCK_STREAM, 0);
int one = 1;
setsockopt(s_fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
switch(opt.so_reuseport) {
case 1:
setsockopt(s_fd, SOL_SOCKET, SO_REUSEPORT, &opt.so_reuseport, sizeof(int));
break;
default:
break;
}
struct sockaddr_in addr;
// This experiment uses IPv4 Address. AF_INET6 is disabled in here.
addr.sin_family = AF_INET;
// Allows all NIC address.
// Selects any NIC device available on a machine.
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(TCP_PORT);
bind(s_fd, (struct sockaddr *)&addr, sizeof(addr));
signal(SIGPIPE, SIG_IGN);
// queue len == 128
listen(s_fd, 128);
char ip_str[INET_ADDRSTRLEN];
inet_ntop(AF_INET, &addr.sin_addr, (char *)ip_str, sizeof(ip_str));
printf("listen %s:%d\r\n", ip_str, TCP_PORT);
clientLoop(s_fd, opt);
}
/// main function starts.
int main(int argc, char **argv) {
Opt_t opt;
memset(&opt, 0, sizeof(opt));
/* these two options are adjusted for each benchmark. */
opt.so_reuseport = argc > 1 ? atoi(argv[1]) : 0;
// please look at the switch in clientLoop
opt.tcp_cork = argc > 2 ? atoi(argv[2]) : 0;
// when SO_REUSEPORT is enabled multi-workers can share the same port.
// in this source, http://localhost:8080 address is shared by four workers.
if(opt.so_reuseport) {
// Worker i: w_id i
for(int i = 0; i < WORKER_POOL; ++i) {
if(!fork()) {
runWorker(i, opt);
exit(0);
}
}
}
else
runWorker(0, opt);
while(1) pause();
return 0;
}
Vanilla
wrk -t10 -c100 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 44.49ms 2.20ms 48.07ms 85.80%
Req/Sec 22.46 4.31 30.00 75.42%
676 requests in 30.10s, 71.31KB read
Requests/sec: 22.46
Transfer/sec: 2.37KB
SO_REUSEPORT only (4 workers)
Running 30s test @ http://localhost:8080/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 45.01ms 2.49ms 48.11ms 72.46%
Req/Sec 88.85 12.51 121.00 82.06%
2672 requests in 30.10s, 281.88KB read
Requests/sec: 88.77
Transfer/sec: 9.37KB
TCP_CORK only
Running 30s test @ http://localhost:8080/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.38us 16.81us 4.03ms 99.48%
Req/Sec 44.88k 1.43k 47.22k 83.06%
1343783 requests in 30.10s, 138.41MB read
Requests/sec: 44645.87
Transfer/sec: 4.60MB
TCP_CORK+SO_REUSEPORT
Running 30s test @ http://localhost:8080/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 26.31us 22.58us 4.23ms 99.52%
Req/Sec 72.22k 33.77k 115.72k 56.15%
4323980 requests in 30.10s, 445.36MB read
Requests/sec: 143653.90
Transfer/sec: 14.80MB
Analysis
As you can see, a benchmark with TCP_CORK transfers much more data per second.
| Vanilla | TCP_CORK |
|---|---|
| 2.37 KB/s | 4.60 MB/s |
Also, enabling TCP_CORK increased RPS a lot, from 22.5 to 44645.9. The latency is 44.49ms without it and 21.38us with it.
It's exactly the same as this sentence at a coffee shop:
When you use a small dripper eight times, you need more time. But when you use a big dripper, huge amounts are extracted for just one time.
| Vanilla | SO_REUSEPORT |
|---|---|
| 22.5/s | 88.8/s |
When we enable SO_REUSEPORT, RPS increases: 22.5 -> 88.8.
It's exactly the same as this enhancement at a coffee shop:
When you try single serving V60 drippers with eight baristas, it takes less than giving a dripper to a senior barista.
Now enable both of them. Let's look at whether these sentences are valid:
But each barista should prepare for each extractions when many orders are waiting per person since a dripper is small. Using
TCP_CORKto a broad range withoutSO_REUSEPORTcan worsen Average RTT. Also, using too many sockets after applying SO_REUSEPORT can drain Unix sockets. Simply enabling both increases two metrics at once: RPS and network transfer rate. When each barista is using a big dripper, if somebody's extraction ends up early, eight customers can get their cups. Naturally, that barista can prepare for other orders right after those cups.
Let's see the RPS metric.
| SO_REUSEPORT+TCP_CORK | SO_REUSEPORT |
|---|---|
| 143653.9/s | 88.8/s |
Let's see the network transfer rate too.
| SO_REUSEPORT+TCP_CORK | SO_REUSEPORT |
|---|---|
| 14.80 MB/s | 9.37 KB/s |
Woah. Both are good.
Enabling both accomplished the best effort.
Those two analogies are nice enough to describe how it works.
Knowing what's happening beyond exaggerated 'Blazingly Fast' code
When we use LLMs to refactor a codebase, we sometimes cannot understand why it's faster. Some people just merge it, and some reject it because 'they could not understand.' But the meaning of that source code is often neglected.
When we understand what they are doing, we can imagine what it affects when some accidents happen. When we don't try to understand, we accept some sloppy patches. And sometimes, we reject some requests that are actually better.
Applying TCP_CORK and SO_REUSEPORT is cheap. But understanding their behavior with experiments can easily lead us to be creative on a bottleneck.
Some say coding with our ten fingers is the only way to understand 'the truth of engineering.' But we can't learn anything when we just code without thinking on our own. Reading books like TAOCP may give us inspirations. But knowing what we've done is quite separated from those 'AI ethics and digital monads.'