The receive window
is your asset.

The last row is often left out and is half the reason the split exists. There is no such thing as multicast TCP, and there cannot be: acknowledgements, retransmission and flow control are all defined against a single peer, and none of them generalise to an unknown number of receivers.

Why a market data feed is multicast UDP

• One to many. Every subscriber wants the identical stream. With multicast the exchange sends each packet once and the network replicates it at the switches. With TCP it would need a separate connection per subscriber and would send the same data hundreds of times, which puts the fan-out cost on the source and makes every subscriber's latency depend on how many others there are.
• Latency beats completeness. A retransmitted quote that arrives 40 ms late is not late data, it is worthless data, because the market has moved. TCP would hold back everything behind the gap while it recovered, delaying the fresh data too.
• Loss is detectable and recoverable out of band. Feeds carry a sequence number on every message, so a receiver notices a gap immediately. Recovery is a separate mechanism: a retransmission request channel, or a redundant A and B feed on two paths that you arbitrate between. That is a better design than TCP's because it recovers only what you actually need, on a channel that does not stall the live feed.
• Fairness. Multicast delivers to every subscriber at once, whereas serving hundreds of TCP connections in a loop inevitably favours whoever is early in the loop.

Why order entry is TCP

• It is point to point. One client, one exchange gateway. There is no fan-out for multicast to solve.
• The message must not be lost. A dropped order is not a stale price you can skip, it is an instruction that either did or did not happen, and you cannot tell which.
• Order matters absolutely. A cancel arriving before the order it cancels is incoherent.
• The volume is tiny. A handful of small messages against a firehose of market data, so TCP's overhead is irrelevant here.

The mechanism, step by step

1. The receiving kernel holds a buffer for the connection, of some size, say 64 KB.
2. Arriving data goes into that buffer and stays there until the application calls recv and copies it out.
3. Every acknowledgement the receiver sends includes a window field: the number of bytes of free space left in that buffer right now.
4. The sender tracks its unacknowledged bytes. It may send while that total is below the advertised window, and must stop when it reaches it.
5. When the application reads, the buffer drains, so the next acknowledgement advertises a larger window and the sender may resume.

Notice what this achieves without any explicit negotiation. The receiver never has to say "slow down". It simply advertises the truth about its buffer, and the sender's own correctness rule does the throttling. That is why it holds up under packet loss: there is no message to lose and no protocol to get wrong.

Zero window and window probes

If the application stops reading altogether, the buffer fills and the receiver advertises a window of zero. The sender must stop completely. But the update that would reopen the window travels in an acknowledgement, and if that acknowledgement is lost, the sender waits forever for news that will never come while the receiver waits for data that will never be sent. A deadlock, from a single lost packet.

The fix is the persist timer. A sender facing a zero window periodically sends a one-byte window probe. The receiver must respond with its current window, so the reopening is discovered even if the original update was lost. Being able to explain that this exists because the reopening announcement is not itself retransmitted is a genuinely good detail.

Window scaling

The window field in the header is 16 bits, so it caps at 65,535 bytes. Combine that with the fundamental throughput limit:

max throughput = window / round-trip time

64 KB over a 100 ms round trip is about 5 Mbit per second, no matter how fast the link is. The pipe is fast but you are only allowed 64 KB in flight, so most of the time is spent waiting for acknowledgements to come back. That is the bandwidth-delay product problem.

The window scale option, negotiated once in the handshake, applies a left shift of up to 14 to the field, raising the ceiling to about 1 GB. This is why it exists, why it can only be set at connection setup, and why a long-distance transfer that will not go fast is usually either a scaling problem or a buffer sizing problem rather than a bandwidth problem.

Sliding window animator

A sender filling a receiver's buffer while the application drains it. Slow the reader and watch the buffer fill, the advertised window shrink to zero, the sender stall, and the persist timer start probing.

Packets take different paths and can arrive out of order. TCP handles this: segments that arrive early are held in the out-of-order queue, indexed by sequence number, waiting for the gap ahead of them to fill.

The data is received. It is sitting in kernel memory, intact and verified. What it is not, is deliverable.

The consequence is that a single lost packet delays every byte behind it by at least one round trip, no matter how much of that data has already arrived safely. On a 100 ms link, one dropped packet stalls the application for 100 ms while megabytes of perfectly good data wait in the kernel.

This is the precise reason a market data feed does not want TCP. A lost quote would hold back every fresher quote behind it, so a single drop turns into a stall of the whole feed. Multicast UDP delivers each datagram the moment it arrives, and the application decides for itself whether a gap matters, which is the right place for that decision.

What the industry did about it

HTTP/2 multiplexed many requests onto one TCP connection to avoid opening several. That removed application-level head-of-line blocking and left the transport-level version in place, so one lost packet stalled every multiplexed stream at once, which was sometimes worse than the problem it solved.

QUIC, which is what HTTP/3 runs on, fixed it by building an ordered-stream abstraction on top of UDP, with independent sequence spaces per stream. A loss in one stream blocks that stream and nothing else. Note the shape of the fix: keep the ordering guarantee, but stop applying it to unrelated data. Head-of-line blocking is not a TCP bug, it is the unavoidable price of a single total order, so the only real fix is to have more than one order.

Head-of-line blocking demo

Ten segments in flight. Drop one and watch everything behind it arrive, sit in the buffer, and stay undeliverable until the retransmission fills the gap.

Cumulative acknowledgements

An acknowledgement number means "I have everything up to here, send me this next". It is cumulative, so ACK 5000 confirms every byte below 5000 at once and a lost acknowledgement is harmlessly superseded by the next one. It also means an acknowledgement cannot express "I got 5000 to 6000 but not 4000 to 5000", which is why the selective acknowledgement option exists: SACK adds explicit ranges of received-but-not-contiguous data so the sender retransmits only the actual hole rather than everything from the gap onward.

Two ways to notice a loss

Fast retransmit is the quick path. Every out-of-order segment makes the receiver re-send the same acknowledgement, pointing at the gap. Three duplicate acknowledgements are taken as strong evidence that one segment was lost while later ones are getting through, so the sender retransmits immediately without waiting for a timer. Three rather than one, because a little reordering is normal and one or two duplicates might just be packets arriving out of order.

The retransmission timeout is the fallback for when nothing at all is coming back, which is what you see when the tail of a transfer is lost or the path is badly broken. Choosing that timeout is the interesting part:

SRTT  = (1 - a)*SRTT   + a*sample          # smoothed round-trip estimate
RTTVAR = (1 - b)*RTTVAR + b*|SRTT - sample|   # its variability
RTO   = SRTT + 4*RTTVAR                       # with a floor, commonly 200 ms

The variance term is the point. A path with a stable 50 ms round trip can use a tight timeout, while a path that swings between 50 and 500 ms needs a loose one or it will retransmit data that was merely late. Including four times the deviation adapts the timeout to how predictable the path actually is, rather than picking one number for all networks. On each successive timeout the value doubles, which is exponential backoff, so a genuinely broken path is probed less and less often instead of being flooded.

The three-way handshake

client                                          server
  |  SYN,  seq=x                                   |    "here is my starting sequence number"
  | ---------------------------------------------> |
  |  SYN-ACK, seq=y, ack=x+1                       |    "got it, and here is mine"
  | <--------------------------------------------- |
  |  ACK,  ack=y+1                                 |    "got yours"
  | ---------------------------------------------> |
  |  connection established, data may flow          |

Three messages rather than two because both directions need their own sequence number and both need confirming, and a two-way exchange would leave one side unsure whether its own number arrived. The initial numbers are randomised, which prevents an off-path attacker from guessing them and injecting data into the connection. The cost is one full round trip before any data moves, which is why TLS 1.3 cut its own handshake to one round trip and why QUIC merges the transport and cryptographic handshakes to get to zero for a resumed connection.

Closing takes four messages, because each direction is shut independently: a FIN and an ACK each way. That is what a half-open connection is, one side done sending while the other continues. The side that closes first then sits in TIME_WAIT for twice the maximum segment lifetime, so that a delayed straggler from this connection cannot be mistaken for data belonging to a new connection reusing the same port pair. A server with thousands of sockets stuck in TIME_WAIT is showing you this rule doing its job.

Sockets, ports and the four-tuple

a connection is identified by:  (src IP, src port, dst IP, dst port)

That tuple is why a server on port 443 can hold a hundred thousand simultaneous connections: the destination address and port are identical for all of them, but each client contributes a different source address or port, so every tuple is distinct. It is also why a client machine is limited to roughly 65,000 outbound connections to the same destination, since only the source port varies, and why connection pooling matters.

Ports are a 16-bit demultiplexing key, nothing more. Below 1024 they conventionally require privilege. The listening socket and the connected socket are different objects: accept returns a new socket for each connection while the listener keeps listening.

What happens when you type a URL

1. Name resolution. Browser cache, then OS cache, then the configured resolver, which walks root, then top-level domain, then authoritative servers unless something along the way has it cached. Usually UDP, falling back to TCP for large responses.
2. Connect. TCP three-way handshake to the resolved address on port 443. One round trip.
3. TLS. Certificate presented and verified against a trusted root, keys agreed. One more round trip under TLS 1.3, and zero for a resumed session.
4. Request. An HTTP request goes out, the server responds. If it is HTTP/2 or HTTP/3 many requests share one connection.
5. Render. Parse the HTML, discover subresources, fetch them, build the DOM and the layout, paint.

The useful observation is that on a fresh connection you have spent two or three round trips before a single byte of content has been requested. On a 100 ms path that is 300 ms of nothing but setup, which is why connection reuse, session resumption and putting servers physically closer to users all matter more than raw bandwidth for page load time.