Hackweeek

A couple of weeks ago we had another hackweek at Novell, and I continued working on getting vbnc to use cecil, as I did last time. With a big difference: I finished.

Yep, vbnc now uses Cecil to emit assemblies, the System.Reflection.Emit API is not used anymore. This makes it possible to fix a few long standing bugs, but most importantly (to me at least, the bugs are quite corner-case), a lot of unnecessary code has been deleted (~10k fewer lines of code in the compiler).

Another advantage is that it's now trivial to add support for compiling to different runtime versions - vbnc won't ever suffer from multiple personalities like the mono's C# compiler does: mcs/gmcs/smcs/dmcs...

Performance is still somewhat lower (bootstrapping the compiler itself is ~20% slower), but I haven't even looked at why (almost 3 years of working on a branch with fairly big changes has probably introduced some non-optimal code), so this is likely to change. Speed is also helped by using the new cecil/light which jbevain released recently.

And since I was done a day early and wanted to have a break from what I've been doing for the last 3 years with vbnc, I implemented the first VB 9 feature: ternary ifs!

ASX playlists

If you tried watching Nasa TV in my last post with Silverlight, you'd find that it wouldn't work. And once again the problem is with the ASX file - to be more specific, the exact same problem I had to find a quick fix for last week for Moonlight.

The ASX file I linked to links to another ASX file. This second ASX file has a copyright symbol in it (©), and it stores it in the CP1252 code page (i.e. the binary value 0xA9 - this is an illegal value in utf8). Now it seems like Silverlight expects the file to be in utf8 [1], and it errors out if it isn't [2].

Unfortunately I can't link to the real mms link, Yahoo seems to uses some sort of authentication ensuring that you got the link from the ASX playlist.

I notified Yahoo using their contact form about this, asking them to fix their ASX output, now let's see what happens...

[1] When there are no byte-order markers at least
[2] Windows Media Player plays the ASX file just fine.

Nasa TV

Interested by the NASA's crash landing on the moon, I wanted to watch the event live on my Linux machine. NASA has a few live TV channels, but none seemed to work on Linux. Until I discovered that they use ASX files to view the channels in Windows Media Player - turns out Silverlight (and by consequence Moonlight) also support ASX files. It was just a matter of creating a simple silverlight application referencing the ASX links. And that's what I did!

The bad thing is that our ASX support was a bit broken [1], and I had to fix it. This means that the currently released beta is not able to view the streams, you need to download brand new xpis from here: x86 / x64 (fair warning: these xpis are not tested at all, download at your own risk). Download, restart firefox and you should be able to watch the event live here.

(video is scaled 200%, quality is kinda bad, but that's how it's transmitted from NASA)

[1] IMHO ASX files are broken by themselves - they look like xml files but aren't (they don't escape special characters that xml files have to), and they're encoded in the current encoding of the machine generating the ASX file, and in order to correctly parse the file you have to know which encoding that is (the file itself doesn't know).

Git & make

One of the most annoying issues with git (which is actually a side-product of the fact that it's very easy to create/administer branches), is that switching branches will cause a lot of recompiles.

ccache helps some, but the real life-saver here is a script which doesn't come bundled with your installed git, but which you'll have to download the source code to get: git-new-workdir. It clones your repository to a different directory, with some symlink magic so that everything is shared except the index. It's the git equivalent a checkout of a svn repository to different directories, except that anything you do in any of the directories is available in all directories.

Git & Whitespace

Recently I've been slowly transitioning from svn to git, though one of the mildly annoying differences is that git likes to colorize trailing whitespace in diffs with a very distracting color:



First try at fixing it: a tiny script to fix the file before committing.

Oops, that introduced a huge number of changes and would mess up history quite a bit, not the way to go.

So I added this to my ~/.gitconfig:

[core]
 whitespace = -trailing-space

and now I can see the important changes in the diffs again.

Cecil Reloaded

For our hack week here at Novell I continued hacking on making vbnc work with Cecil, and yesterday I reached the point where vbnc is able to compile itself using only Cecil (in my previous post I also said vbnc was able to bootstrap, but back then vbnc was still using SRE to do all the hard work, and Cecil just to write the final assembly to disk).

The initial performance problems are mostly gone, it's still not quite as fast as before, but there are also quite a few low-hanging fruit yet in that area. The first successful bootstrap yesterday took 40 seconds (compared to 12 seconds for the normal vbnc), and after a few optimizations I'm now down at 24 seconds. Almost all of that time was gained by allocating less strings, the first non-optimized version allocated ~1.4 GB of strings, and it's now down to 119 MB.

I'm also using a delay-loading mechanism to load data from referenced assemblies on-demand, this is not that visible when you compile a lot of code in one go, but it sure make a difference when compiling thousands of 15-20 line tests one by one.

I'm pretty confident I can get compilation time down to the 12 seconds I had before, especially given that Cecil hasn't been much optimized [1] (not actually used by compilers) before, while Mono's SRE implementation is known to be quite fast.

As a sidenote the actual amount of code in the compiler has decreased, from ~75k lines with the normal vbnc to ~68k for the cecil version.

[1] Sébastien Pouliot corrected me, cecil is used by some know compilers, and probably some unknown too. Some parts of Cecil has also been optimized already.

Cecil

When I started writing vbnc, there weren't many options when it came to deciding which library to use to write the assemblies, only System.Reflection.Emit (SRE) was a real option.

SRE is very powerful, but unfortunately it has a few known and unknown limitations, mostly because it was never designed to be used by a full-fledged compiler.

Since I heard about Cecil, I've wanted to switch, and for the last months I've slowly added support for emitting assemblies with Cecil. Yesterday I reached a very important milestone: vbnc is able to bootstrap itself when using Cecil!

And I have to say that Cecil is A LOT easier to use than SRE, especially with generics.

There are still some problems for switching right away to only using Cecil, one of the biggest being that it's quite slow if you have many referenced assemblies (since Cecil loads everything from an assembly when loading it), though there is work in progress here.

Hopefully these issues will be solved shortly, and I can finally remove everything SRE-related, which has caused me quite a few head-aches!