Tangled in the Threads

Jon Udell, November 16, 2000

How do peers connect?

Napster and Scour aren't fully NAT-permeable

The newsgroup mulls over whether TCP or UDP connection-splicing could assure bidirectional communication between any two peers, even when both are firewalled.

Back in September, I posed a simple question that turned out to be not so simple: How do Napster peers connect? My PC, connected to the Net by way of a DSL line and a Cisco 675, allows no inbound connections. Yet it works as both a Napster server and client. How?

Suppose you find something in my Napster library that you want, and your device does accept connections. Napster, the connection broker, just needs to arrange so that I always contact you, even in this case where you are nominally the client and I am nominally the server. This part's straightforward, though clever.

But suppose your device also refuses inbound connections? Napster would then have to either:

  1. Relay all the traffic between us, rather than just brokering the connection.

  2. Broker the connection in some way that magically splices together two client-initiated TCP sessions.

I was sure Napster didn't do relaying, that would require massive bandwith. I wasn't sure about the connection-splicing, though. Was this even possible?

As it turns out, Napster just plain doesn't work when both peers refuse inbound connections. This problem will loom larger as more peer-to-peer applications emerge. How do you connect arbitrary endpoints efficiently, in the presence of firewalls and NATs? It's a fundamental question, and answering it could end up changing how NATs, or TCP, or both, work. In his online essay What is P2P?, Clay Shirky suggests how big the stakes already are:

Whois counts 23 million domain names, built up in the 16 years since the inception of IP addresses, in 1984. Napster alone has created more than 23 million non-DNS addresses in 16 months, and when you add in all the non-DNS Instant Messaging addresses, the number of p2p addresses designed to reach dynamic IPs tops 200 million. Even if you assume that the average DNS host has 10 additional addresses of the form foo.host.com, the total number of p2p addresses now equals the total number of DNS addresses after only 4 years, and is growing faster than the DNS universe today.

How will these P2P addresses interconnect efficiently? The problem's not insurmountable for dialup users. Many don't bother with software firewalls. Those who do can, fairly easily, reconfigure them to accept connections. For the growing ranks of cable-modem and DSL users, with NAT-enabled devices from the likes Cisco and SonicWALL, it's much harder. These devices are not so easy to reconfigure. Arguably, home users just shouldn't accept unsolicited contact, period.

Groove, which I wrote about in a recent column, accepts the reality that NATs will, for the foreseeable future, obstruct the flow of peer-to-peer communication. The solution, in Groove, is to do what Napster doesn't -- relay data streams between pairs of endpoints that would otherwise find themselves incommunicado.

Is TCP connection-splicing possible?

Doing such relaying in a general way, rather than specifically for Groove peers, may emerge as one of the business opportunities created by the P2P movement. And yet it's hard to stop thinking about how to cut out the middleman and splice together a pair of client-initiated sessions. Is it even theoretically possible? Maybe not.

Peter Hess:

Even if all the information about an established connection is provided to a client, it will still need to negotiate its own connection with the opposite endpoint. This is a property of TCP, not NAT, and to not have this property is to no longer follow the rules of the TCP protocol.

Randy Switt:

Where is the TCP handshaking broken? All Syn, Fin, and Ack flags are passed on unmodified, sequence numbers are untouched, only the IP and port fields are modified per the entry in the NAT table. All connection negotiation is done and established by the internal client and the remote server, the NAT device is only masking the IP and port fields.

Peter Hess:

I still believe that the NAT box must track the source and destination addresses as well as the ports. That socket information (Address:Port) defines the TCP connection and indeed is shown in the NAT table.

Randy Switt:

Is it not possible for two connections to use one NAT table entry? I know that this is NOT generally the way it is done, but I'm not sure it's not possible. Basically, if we stipulate that BOTH connections MUST use the same incoming port to the client, we can remove NAT from the whole picture.

The way I understand it, a client behind a NAT cannot act as a server because an incoming connection will not be directed to the interior client. But standard web calls and such will work, because all sessions instantiated by the NATed client will get a port-client mapping in the router's NAT table. The way I think it may be done is that a Scour client sets up a persistant NAT table entry, by maintaining a continuous connection to the server. While this connection is maintained, all traffic sent to that port on the router will be redirected to the Scour client on the internal client. As long as the connection is up, other clients can transfer files over this Port-Client mapping, as long as they know which port to transfer to (which is available from the server). As such, it is a MAJOR security hole, since it sets up persistant connection to an interior machine, which is normally difficult to do with NAT.

Bruce Elrick:

No, A TCP connection is defined by 4 things: source IP, source port, destination IP, destination port. Thus the NAT will consider an incoming _packet_ from the other client (thus different source IP than incoming packets from the server) to the port being used by the server for its NATed connection as a spurious packet, not one that is part of the TCP connection it is doing NATing for between the first client and the server. An incoming TCP handshake to that port would be treated as a connection attempt directly to the firewall/NAT machine; since there would not be a process listening on that port, it would be refused.

During this discussion, Randy had written to Scour, asking how they handled the two-NATed-client case. Here's the reply he got: 

Subject: Re: [Dev] Peer to peer connections behind firewalls

Scour does not support transfers between 2 firewalled clients, nor
does anybody else that I'm aware of.  While it's theoretically
possible, it would be quite difficult to pull off (I'd think blind
spoofing would be needed to establish the connection).  It would make
a good PhD thesis!

Is UDP the answer?

Meanwhile, the same subject came up on the decentralization list on eGroups. A message was posted there with a reference to a UDP-based hack that apparently solves the problem, at least for some gaming applications.

Peter Hess' analysis of this UDP-based technique:

NAT needs to see an outbound connection before it will allow inbound packets to the client. Apparently, by having all clients send outbound packets first (and simultaneously), the outbound hole is opened for each sending client. This allows the inbound packets from all the other clients to come through because they now look like responses to the outbound packets. This establishes the bidirectional connection. The relay server is necessary only to make sure that everybody is on the same port and that they all have the correct destination addresses. It serves only as a directory

The novel thing is that the initial outbound packets probably won't ever reach their destinations. They are sacrificed to open the outbound connections, which are then exploited by the additional outbound packets following behind. It works sort of like a suicide bomber blowing a hole in the wall from inside so that his comrades can enter from the outside.

This procedure seems to be specific to UDP. I don't think it will work with TCP because of the handshaking necessary (unless the NAT box allows inbound connections, in which case this whole procedure isn't really necessary). Because UDP doesn't handshake first, an open outbound connection through the NAT box will be sufficient to allow inbound UDP connections to that socket, providing they arrive on the correct port (which is why the peers need to have a common understanding on this point).

Should P2P apps jump on the UDP bandwagon, then? Bjørn Borud doesn't think so:

TCP is not so much optimized for providing responsive reliable connections as it is optimized not to be anti-social. A very important part of the TCP design and implementation is to reduce the chances of really bad network congestion.

A few years ago a friend of mine actually implemented his own TCP clone over UDP. It had none of the features that make TCP nice to the network, but it made interactive sessions over unreliable networks a *lot* more responsive. And indeed it was pretty anti-social. It would waste a lot of bandwidth retransmitting data.

If everyone had been using that protocol in place of TCP you would soon experience horrible congestion and people would start blocking a lot of UDP traffic.

Are firewalls useful anymore?

In another (offline) discussion of these issues, an acquaintance remarked that firewalls without application-level gateways have become pointless, given that we tunnel so much stuff through port 80. In short: the emperor has no clothes, and we're all pretending not to see.

Still, for me as a home DSL user, the idea of firewalling doesn't seem quite so pointless. When I first got my Cisco 675, I figured I'd immediately open up at least port 80, because offering web service was a longstanding habit. Then I reconsidered. Why, exactly, should I let in any uninvited sessions in to my personal machine? So, I don't. I offer web service elsewhere.

This isn't perfect (what is?) but it is typical, I'd argue, of the vast majority of cable/DSL users. For this kind of user, there isn't a legacy of:

I hope P2P doesn't repeat that history on a different port number. In general, I like the "let nothing uninvited in" policy. Can it work? That depends, I suppose, on how we define, and implement, the notion of a P2P rendezvous.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He's now an independent Web/Internet consultant, and is the author of Practical Internet Groupware, from O'Reilly and Associates. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

Creative Commons License
This work is licensed under a Creative Commons License.