One at a time is a queue: going multithreaded

Open a second telnet and watch it hang

The daytime server answers in well under a millisecond, so its single-threaded accept loop looks fine. Give it a protocol that takes a second — a file to read off disk, a lookup to run, a client on hotel Wi-Fi — and the flaw is immediate. Point one telnet at it and hold the connection open. Point a second telnet at the same port. The second one connects (the kernel finished the handshake without asking your program) but hears nothing whatever until the first is done, because there is one thread and it is inside the first client's handler.

An iterative server serialises its clients. Its ceiling is one connection per handler-duration no matter how idle the machine is — and nearly all of that duration is a thread blocked on I/O, doing no work at all.

The queue behind accept()

The connections you have not accepted yet are not lost. The operating system completes the TCP handshake on your behalf and parks each connection in a fixed-length FIFO queue attached to the listening port; accept() removes the entry at the front. Java requests a queue of 50 by default, and the OS silently caps that at its own maximum (128 on some systems, and you cannot exceed it by asking for more). Once the queue is full, the host refuses further connections on that port — the client gets a connection refused, not a slow server.

So the queue buys a burst, not a rescue. Empty it faster than clients fill it and all is well; let your average handler outlast your average arrival interval and the queue saturates and the server starts turning people away. Lengthening the queue does not fix that. The fix is to stop doing the handling on the accepting thread.

One thread per connection

Move the loop body into a Runnable that owns exactly one Socket, start a thread on it, and let the accept loop do nothing but accept and hand off. The accepting thread is then back in accept() microseconds later, whatever the handler is doing. A thread is far cheaper than the separate OS process that older Unix servers forked per connection, which is why this design scales to hundreds of clients where forking stalls at a few hundred.

What the handler owns matters as much as what it does. The Socket, its streams, any parse buffer, any per-request counter — all of that is per-connection state and belongs in fields of the handler instance, one instance per connection. Anything genuinely shared must be immutable or thread-safe. A static scratch buffer shared by handlers is a bug that appears only under simultaneous load, which is to say only in marking.

The socket escapes the try block

One trap deserves its own paragraph. In the iterative server the accepted socket was a try-with-resources resource, which closed it at the end of the block — correct there, catastrophic here. The socket now outlives the block: it is handed to another thread that has barely started. Try-with-resources would close it at the bottom of the loop iteration, before the handler wrote a byte, and every client would see an empty response. The accepting thread must not own the socket's lifetime; the handler must, and it closes it in the handler.

Worked example — the daytime server, one thread per client

public final class ThreadedDaytimeServer {

    public static void main(String[] args) throws IOException {
        try (ServerSocket listener = new ServerSocket(1313)) {
            while (true) {
                // deliberately NOT try-with-resources: this socket escapes
                Socket client = listener.accept();
                new Thread(new DaytimeHandler(client)).start();
            }
        }
    }

    private static final class DaytimeHandler implements Runnable {

        private final Socket connection;   // per-connection state, owned by this task

        DaytimeHandler(Socket connection) {
            this.connection = connection;
        }

        @Override
        public void run() {
            try (Socket owned = connection) {
                Writer out = new OutputStreamWriter(
                        owned.getOutputStream(), StandardCharsets.US_ASCII);
                out.write(ZonedDateTime.now() + "\r\n");
                out.flush();
            } catch (IOException ex) {
                System.err.println("dropped: " + ex.getMessage());
            }
        }
    }
}

Two telnet sessions now overlap. And the server has acquired a new weakness: the number of threads is chosen by whoever is connecting, not by you. A flood of simultaneous connections spawns threads until the JVM runs out of memory and dies — a denial of service you built yourself. That is the problem the next lesson bounds.