Tuesday, December 27, 2005
This is going slower than I would have liked, but I have a start on a paper proposing a sockets API. Right now, the paper simply analyzes the current sockets APIs of a few CL implementations. There is a "to be written" section where the eventual proposed API will go. I have a start on that, too, but I held it back because I wrote a lot of it half-asleep on the plane back from NYC a couple weeks ago, and I know there are a bunch of blunders in it. The rest of it seems sane enough to start spreading around, however. If anybody has any feedback, please shoot it to me at dave at findinglisp.
Thanks to Thom Goodsell and Iban Hatchondo for their comments.
Thanks for the well-written and thorough summary of CL socket issues. You seem to have covered everything, including the importance of looking beyond blocking threads. I'm looking forward to API.
What a nice little Christmas present. Strictly speaking, SSL is beyond the scope of this document, but it is an important protocol, and the Java folks kind of stumbled with the initial implementations -- until recently, you could use SSL or java.nio, but not both. Any thoughts on how SSL would work with this framework? Would SSL have to be pure lisp, or use FFI?
Important work, thanks for doing that. A couple of immediate comments:
You can do non-blocking sockets in SBCL; see this post.
There's a couple of links to posts by Tridge here that give some detail on common misconceptions about threads (conclusion: threads suck for virtually everything).
Threads only suck if you have alternatives for multiprocessing, but lisp doesn't have an equivalent to the 'select()' feature that C provides for changing contexts, so I don't really see any other way for lisp to walk and chew bubblegum.
fred: Lisp doesn't have an equivalent for select(), true, but nor does it have an equivalent for connect() or accept(), which I suggest is kind of the point of the exercise ;-)
Incidentally, SBCL's choice to represent IP addresses as octet vectors is probably the primary reason it has no ipv6 support. At least, as original author of SB-BSD-SOCKETS, the choice of an address representation was always the first thing I got hung up on when trying to add IPv6.
Well, that plus "there are no interesting services that require IPv6", but that's just a chicken/egg thing and shouldn't be a bar to doing it for fun.
Dan, agreed on IPv6 and address representations, but I really like the vector (or list) of decimal values for IPv4. That makes it so much easier to deal with at the REPL. Most of the other implementations use a single 32-bit integer which makes it more difficult to figure out what's going on.
Is it natural that I have no idea what any of this means? Yes, it's just a reminder that there are still a ton of things that I have to learn.
API, the essence, the meaning, the superfluous mind warp that takes us all for the pc whores that we are not. Delight me, oh tease my buttons, but never accuse me of being the ludite!
Miles, it is the natural order of things,,,,,you have obviously been out a little more than some of the contributors. Sockets with interger combos, what next? Socket for rockets perchance? An ingenue of the highest order my son,,,,let it flow...
Fred: fork() is quite popular in certain circles :)
Another problem with threads is when you need communication between the threads themselves, not just the outside world. This requires some sort of multiplexing (the thread waiting on the socket and a pipe/mutex/whatever). If each thread is largely autonomous then why not use fork()? The only explanation I can think of is "memory", but that's a very poor trade-off on modern computers.
I don't think fork() is viable for Lisp's like SBCL and CMU that over commit memory (top reports SBCL's using 800MB right now). Maybe for clisp with it's smaller memory footprint... But that sort of makes it impractical as a cross-implementation strategy.
Fred: To quote the Moaning Goat Meter FAQ, "Here's a nickel. Go buy more memory." ;)
Actually, it's not as bad as it looks. Overcommitted memory is virtual, the resident is more important; I get about ~25M. However, fork will use copy-on-write pages, so most of this memory *should* be shared between the processes ('top' suggests about 14M is shared immediately after fork). This will degrade over time, but if your forked process is short-lived then this doesn't matter.
It will matter for longer-lived processes though, so yes, a lighter-weight runtime is useful in those cases. ECL gives me an RSS of ~9M, with a shared component of 4-6M, and it has a much more permissive license that Clisp. I personally think ECL doesn't get enough attention in the lisp world.
Post a Comment
Links to this post: