Jon Udell: SSL Proxying

Tangled in the Threads
Jon Udell, February 14, 2001
SSL Proxying

Opening a window onto secure client/server conversations

In the second of a two-parter, Jon dives into the details of SSL proxying

Last week I mentioned that the new beta version of Proxomitron, a local web proxy, supports SSL. Let's look at how this works. Normally, Proxomitron works only with unencrypted traffic. It listens (by default) on port 8080. To relay traffic through it, set your browser's HTTP proxy to localhost:8080. In Netscape, that's Edit->Preferences->Advanced->Proxies->Manual Configuration->View->HTTP. In MSIE, it's Tools->Internet Options->Connections->LAN Settings->Proxy Server->Use a Proxy Server.

Now you can watch the conversation between your browser and a web server in Proxomitron's log window. Here's the browser on my Windows machine talking to the server on my Linux machine, as seen in the log window:

A browser requests a script, with parameter foo=bar:
GET /cgi-bin/test?foo=bar HTTP/1.0
Connection: keep-alive
User-Agent: Mozilla/4.5 [en] (WinNT; I)
Host: udell.test
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
The server responds like this:
HTTP/1.1 200 OK
Date: Wed, 14 Feb 2001 18:43:20 GMT
Server: Apache/1.3.12 (Unix)  (Red Hat/Linux) mod_ssl/2.6.6 OpenSSL/0.9.5a mod_perl/1.24
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
Connection: keep-alive
If-Modified-Since: Wed, 23 Aug 2000 19:45:13 GMT; length=2890
User-Agent: Mozilla/4.5 [en] (WinNT; I)
Pragma: no-cache
Host: udell.test
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
In this case the script, /cgi-bin/test, just echoes the name/value pairs it received.

This is useful because, as I mentioned last time, quite a few website APIs are made of nothing more than HTTP headers and GET requests, which you can analyze in Proxomitron's log window.

What about forms that use POST rather than GET requests? The new beta version of Proxomitron lets you watch these too. In the log window, do Edit->View Posted Data. Now, if you access /cgi-bin/test from a form by means of a POST request, rather than from the browser's "command line" by means of a GET, you'll see this:
POST /cgi-bin/test HTTP/1.0
Connection: keep-alive
Referer: http://udell.test/test.html
User-Agent: Mozilla/4.5 [en] (WinNT; I)
Host: udell.test
Content-type: application/x-www-form-urlencoded
Content-length: 7
Posting 7 bytes...
foo=bar
These are the basic investigative tools you need to discover a website's API, so that you can automate it. Of course this whole approach is, as I've said, rather lame. Websites should offer documented XML APIs. Website-building tools should ensure that the creation of such APIs is a natural by-product of the development process, not a special procedure requiring extra thought and effort. The Microsoft .NET beta gives a tantalizing view of a world in which rational web APIs are routine and automatic. But the fact remains that today, hardly any websites that you might want to automate offer official APIs, and there's a lot of mileage still to be gotten out of discovering and using their unofficial APIs.

It's also true that many of the sites you might want to automate are secure sites. And now, thanks to the SSL-ized version of Proxomitron, you can.

A window onto encrypted client/server conversations

To watch SSL traffic in Proxomitron, configure your browser to also proxy secure traffic. In Netscape, that's Edit->Preferences->Advanced->Proxies->Manual Configuration->View->Security. In MSIE, if you established 8080 for normal web traffic, you're already proxying secure traffic as well (and, indeed, you have to use Tools->Internet Options->Connections->LAN Settings->Proxy Server->Use a Proxy Server->Advanced to disable SSL proxying).

Here's that same POST request, on the secure side of my Apache server, with the only difference being the bolded SSL cipher signature:
SSL cipher SSLv3 EXP-RC4-MD5 (128 bits)
POST /cgi-bin/test HTTP/1.0
Connection: keep-alive
Referer: http://udell.test/test.html
User-Agent: Mozilla/4.5 [en] (WinNT; I)
Host: udell.test
Content-type: application/x-www-form-urlencoded
Content-length: 7
Posting 7 bytes...
foo=bar
That's all there is to it! Now, the browser's secure traffic flows to Proxomitron. It decrypts that traffic, so you can see it in the log window, and then re-encrypts it to the destination server. Coming back the other way, it decrypts the server's responses, so you can see them in the log window, then re-encrypts them to complete the secure loop back to the browser.

It's really quite amazing, and amazingly useful. Automation tasks that used to look like more trouble than they were worth -- like, for example, driving a HotMail or E*Trade account from a script -- suddenly look easy.

Does this make a mockery of the notion of a secure channel? I don't think so. Your browser only connects to an SSL proxy if you tell it to. And when it does, the process is not seamless -- nor should it be. If you use the default certificate that comes with Proxomitron, your browser will challenge you for two reasons. First, because that certificate is self-signed, not signed by one of the certification authorities (e.g. VeriSign, Thawte) your browser is hardwired to trust. Second, because the name in that certificate won't match the name of the secure site you're going through Proxomitron to get to. So it's pretty clear that you're connecting to Proxomitron.

It's useful to think, for a moment, about how an SSL proxy normally works. Normally, it's just a relay used to move encrypted traffic through a firewall. In principle this relaying is quite secure. No decryption happens or, indeed, could happen without your knowledget -- or at least, without your ability to discover it by checking the certificate. So while Proxomitron is quite cleverly inserting itself into this secure channel in order to snoop, it's playing by the rules.

Proxomitron can, however, bend those rules slightly. I wrote to Proxomitron's author, Scott Lemmon, and he reported this interesting workaround:

Since real certificates include the site's hostname encoded in them, you'd have to create new server certificates on the fly to get around that. This is why Proxomitron also accepts URLs like so...

http://https..securesite.com/

This lets the browser to make a non-encrypted connection to Proxomitron which then goes on to make an encrypted connection to the server. The trick can be made a bit more transparent with a web filter to replace "https://" with " http://https.." in pages. Also, I guess to ease my perverse sense of completeness, "https://http..normalsite.com/" also works. ;-) It's really pretty useless, but you can have any mix of SSL/non-SSL connections between Proxomitron, your browser, and a website.

Perverse? Definitely. Useless? I wouldn't say so. I think it's really useful, though in delightfully perverse way I'll admit.

Do-it-yourself SSL proxying?

The SSL-aware version of Proxomitron solved my problem -- API discovery on secure sites -- beautifully. But I'm still left wondering what it would take to do something similar in a scripting language, such as Perl, and thus gain the ability to influence as well as monitor the traffic.

The SSL-aware version of Proxomitron solved my problem -- API discovery on secure sites -- beautifully. But I'm still left wondering what it would take to do something similar in a scripting language, such as Perl. Why would you want to? It's true that Proxomitron's regular expression can be used to alter the web pages flowing through it, but in some circumstances you might want to bring the full power of Perl and its supporting modules to bear.

A basic proxy server is a very easy thing in Perl. Here's one based on Perl's HTTP::Daemon and LWP modules:
#! /usr/bin/perl -w
use strict;

use HTTP::Daemon;
use LWP;

my $d = new HTTP::Daemon LocalPort=>$ARGV[0];
print "Please contact me at: url, ">\n";

my $ua = new LWP::UserAgent;

while ( my $c = $d->accept )
  {
  while ( my $request = $c->get_request )
    {

    # have your way with $request here...

    my $response = $ua->request($request);

    # have your way with $response here...

    $c->send_response($response);
    }
  $c->close;
  undef($c);
  };
If you run this at 8080, and point your HTTP proxy setting at it, you've got a tiny proxy server. It does exactly nothing, but is in a position to do all sorts of magic by altering either the request it receives from the browser, or the response it receives from the destination server, or both.

What happens if you point your secure proxy setting at this same Perl-based proxy? It won't work, not surprisingly, but it does receive an initial cleartext request. If you print its contents (using Perl's invaluable Data::Dumper module) you'll see that it is neither a GET nor a POST, but rather, a CONNECT.

In newsgroup discussion, Peter Hess pointed to a couple of documents that describe the CONNECT method. Notes RFC 2616 (the HTTP 1.1 spec):
9.9 CONNECT

   This specification reserves the method name CONNECT for use with a
   proxy that can dynamically switch to being a tunnel (e.g. SSL
   tunneling [44]).
According to Netscape's document on SSL proxying:
Internally, SSL proxying uses the CONNECT method with the destination
hostname and port number as a parameter followed by an empty line: 

CONNECT energy.netscape.com:443 HTTP/1.0 

A successful response from the proxy server is

HTTP/1.0 200 Connection established
Proxy-agent: Netscape-Proxy/1.1 

followed by an empty line. The connection is then set up between
the client and the remote server, and they can transfer data in both
directions until either closes the connection. 
Scott Lemmon:
The "secure" proxy option works a bit differently from a normal HTTP proxy connection. It's arranged so that the proxy doesn't need to have any idea what SSL even is. There's a clear-text handshake that goes on before any encrypted data is send down the wire....

browser sends : CONNECT somehost:port HTTP/1.0\r\n (any other headers) \r\n proxy responds: HTTP/1.0 200 Connection established\r\n \r\n

This sets the proxy up as a relay between the browser and the secure website. In theory, you can then send pretty much anything down it - even non-HTTP protocols!

Anyway, once we've established the connection with the browser, Proxomitron can just give the socket to SSLeay and pretend to be the remote site. One little caveat here - the browser sends the SSL request as if connected directly to the host, so the "GET" request only has the path portion of the URL ("GET /foobar.html HTTP/1.0"). This isn't too big a deal since we can re-create the full URL from the hostname in the original CONNECT request. After that, it's fairly straightforward - Proxomitron just makes a SSL connection to the remote site. It only has to do a CONNECT itself if it's going through an additional proxy server on the way.
Armed with this information, it might not be too hard to make an SSL-capable version of the simple Perl proxy shown above. If you're inclined to try, have a look at IO::Socket::SSL. It seems that a version of HTTP::Daemon modified to use this, rather than IO::Socket::INET, should be in a position to do the necessary magic. If someone's already done this and wants to show us how, please do!

In any case, it's clear that we're finally entering an era in which SSL-aware applications and tools are common and widely available. Does the expiration of RSA's patent make it legal to do this kind of stuff now, using OpenSSL? Honestly, I don't know. But after spending years fretting about this issue, I've decided to stop worrying and start enjoying some of the many OpenSSL-related apps that are popping up like weeds.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He's now an independent Web/Internet consultant, and is the author of Practical Internet Groupware, from O'Reilly and Associates. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

This work is licensed under a Creative Commons License.