Friday, November 30, 2007

Experimenting with IronRuby

In an ideal world, the last 2 months would not have blown me away in a hurricane of work and study, and I would have had some time to play with IronRuby. Unfortunately as you by now know, I don't live in an ideal world. :(

Anyway, enough feeling sorry for myself and on with the experimentation. I resumed tonight (after updating my working copy from the server) in my efforts to get some simple http programming going in IronRuby. I've spent the entire week in Juval Lowy's WCF Masterclass and although http is just the tip of the iceberg in terms of WCF, watching an expert in action has inspired me somewhat. A large part of this inspiration stems from the absolutely phenomenal amount of extremely high-quality material (including free source code!!) that Juval makes available not only to attendees, but to the public in general at IDesign.

So, first things first, I took a quick look in the file rss.rb in my oddly named "projects\rss" folder to try and remember what I'd been working on the last time. Inside I found the following:

require 'net/http'

h ='', 80)
resp, data = h.get('/index.html', nil)
if resp.message == "OK"
    data.scan(/<img src="(.*?)"/) { |x| puts x }

Oh yes, that's right! I'd pulled this piece of code almost directly (maybe directly?) out of Ruby's Pickaxe book - the .chm version which installs with MRI. This was supposed to be my glorious start on the road to understand the inner workings of RSS - hence the (for now) irrelevant folder and file names.

I remember I was trying to get it to run under IronRuby and I also remember that the IronRuby parser had some trouble with parsing the net/http file. At the time, I spent about 3 hours tracing through the debugger just trying to figure out which line in net/http the problem was on and eventually I narrowed it down to a couple of methods with the (for me) unusual syntax of:

variable = dictionary_lookup["key"] or return nil

It turned out that at the time, the parser couldn't handle this (the or return nil part), so I dutifully wrote a test case which reproduced the problem without requiring all 2300 lines of net/http.rb:

def fun(input)
    a = input == 'test' or return nil
    return a
puts fun('not_test')

Unfortunately, at this point I had no idea where to even begin to look to sort this bug out, so I simply submitted the test case to the mailing list, got snowed under at work and forgot about the bug.

In between, a few months have passed and I remember seeing an email on the list a few weeks ago saying that this problem had been resolved, so tonight I decided to try my luck again. After spending half an hour or so trying to puzzle my way through the solution configuration (rake compile config=release is no good if you want to step through the debugger) I eventually managed to get the interactive console running in the debugger. A quick require 'net/http' later and I was once again up to my elbows in exceptions:

System.NotImplementedException: Statement transformation not implemented: WhileLoopStatement
  at Ruby.Compiler.Ast.Statement.Transform(AstGenerator gen) in D:\Projects\IronRuby\trunk\src\ironruby\Compiler\Ast\Statements\Statement.cs:line 32

Unfortunately, I'd rolled back all the code that I'd put in to print the lines of ruby code as they were being parsed to track down the parser errors (so that I didn't get conflicts while updating from the repository), so again it took a little fiddling with the debugger to puzzle out exactly where the problem was. Apparently my previous bug has been sorted out, but that's just allowed the compiler to progress a little further through the file before stumbling on something else. It looks like the code for while loop statements (not expressions) hasn't been written yet, and that's causing the compiler to throw a NotImplementedException while trying to understand the request method on class HTTP which has the following code:

def request(...)
        res = HTTPResponse.read_new(@socket)
    end while res.kind_of?(HTTPContinue)

It looks like the code is throwing an exception in the base class (there's no overriding implementation of Transform in WhileLoopStatement yet), so my first shot at getting past this problem was just to override the method and return null. That seemed to help a little, because I no longer saw the same exception, but it was just the illusion of progress because a short while later I got an assertion about an object that cannot be null.

A little bit of searching through the solution led me to the WhileLoopExpression class in Ruby/Compiler/Ast/Expressions. This class already has an implementation in it, so I figured I'd try and just copy/paste (shudder) it into the WhileLoopStatement class and see what happened. After resolving a few compiler issues (I had to instantiate a List<Statement> object and put the single Statement object that I had into the list before passing this list through to gen.TransformStatements) I was able to compile and start the interactive console up again. Another quick require 'net/http' and I seem to be past the issue, but now I get:

System.NotImplementedException: TODO
  at Ruby.Compiler.Ast.SuperCall.TransformRead(AstGenerator gen) in D:\Projects\IronRuby\trunk\src\ironruby\Compiler\Ast\Expressions\SuperCall.cs:line 42

Hmm, it seems something in the file is trying to call the superclass's implementation of whatever it's overriding. Two minutes looking into SuperCall.cs and I realize that I'm not going to get so lucky with this one. I don't know enough about the internal workings of the compiler (or ruby itself for that matter), so I try a different tack. I open http.rb and I search through it for the word "super". It turns out that it's only called from 2 places in the file, and it doesn't look like the compiler will break if I just comment those two lines out. (Note: At this point I've made a copy of net/http.rb in case I ever actually want to use it for anything - the copy sits in a net folder inside of where Visual Studio builds my interactive console to).

Back to rbx and I type my single line of ruby once again and... Success! Well, sort of. It looks like the compiler is done with net/http - it's managed to get all the way through a 2300 line ruby library - not bad for pre-alpha software. Unfortunately, net/http doesn't live in a bubble - it requires net/protocol, which I haven't yet copied to my working directory. This causes rbx to throw an exception:

Ruby.Runtime.LoadError: Could not load file or assembly 'net\/protocol' or one of its dependencies. The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) ---> System.IO.FileLoadException: Could not load file or assembly 'net\/protocol' or one of its dependencies. The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
File name: 'net\/protocol'

Apart from the HRESULT (COM -> *shudder*) this seems like a fairly easy problem to solve. I copy protocol.rb from the same place I got http.rb and try again, and now I'm met with the exception:

Ruby.Runtime.LoadError: Could not load file or assembly 'socket' or one of its dependencies. The system cannot find the file specified. ---> System.IO.FileNotFoundException: Could not load file or assembly 'socket' or one of its dependencies. The system cannot find the file specified.
File name: 'socket'

"This is going to be easy" I think to myself as I alt-tab back to the ruby libraries folder. "What's taking Microsoft so long?" I think to myself as I start to search for socket.rb. Unfortunately this thought is cut short when I realize that there doesn't seem to be a socket.rb file anywhere in my ruby libraries folder. Fearing the worst, and already regretting my hasty MS-bashing, I widen my search to just look for files with socket in them. This turns up socket.c => uh-oh. It looks like the socket library hasn't been implemented in ruby itself but rather in C (for reasons of speed, cross-platform...ity and the convenience of working closer to the hardware I presume). "So this is what they've been talking about on the list - re-implementing the native C libraries" I mutter.

After giving it some thought while writing this article, it looks like my RSS reader is going to have to wait for a while - at least until somebody adds enough socket support to IronRuby to allow me to do some http communication. Unfortunately, the only code I've been able to add so far is the copy/pasting I did in WhileLoopStatement, and from what I understand, Microsoft is not taking community contributions to the compiler - only to the libraries.

I guess I'll fire off an email to the list anyway with the code in it, in case it is actually correct and they do actually accept it, and even if they don't perhaps I can get some feedback as to what I should have done differently so that I'll know better for next time. In the mean time, I guess I can get started implementing some of the socket functionality - it doesn't look like anybody else has offered to tackle this yet. At least there, I've got more chance of having my work accepted, and it also seems like a smaller chunk to bite off than an entire http library.

Wish me luck!