Buyaa, I'm Microsoft MVP once again in the XNA/DirectX category for one more year (2009/2010). I have been an XNA MVP since 2006 and I'm still very proud of it :)

I was pretty busy this year with our current game project "Fireburst" at exDream (a racing game for Xbox 360, PS3 and PC using the Unreal3 engine, more about it soon) and theirfore I did not do many other things (except writing some iPhone games, starting to develop my own dynamic language and some tools). But once that project is done in 1-2 months, I will do lots more XNA fun stuff and hopefully XNA Community Games (now called Indie Games on the Xbox 360, I really hate that term) will be available in Germany so I can finally review and submit some games myself. I'm still pretty fit in XNA 3.1 and DirectX 11 (played around with it a lot in March), but for XNA I'm still waiting for availablilty in Germany and for DirectX 11 I'm really waiting for some cool hardware at the end of this year.

At exDream we recently also had some interesting discussions about using .NET for PS3 (recently possible thanks to Novell), Xbox 360 (hello XNA) and PC or even more dynamic script languages for upcoming projects. Maybe also including some other promising platforms such as iPhone (also .NET able thanks to Unity), Android, PSP Go, WII, whatever, but that all should depend more on the game and if we are able to manage so many platforms. Working with XNA was almost no extra work to make a game run on the Xbox 360 plus the PC, the game just has to fit and you obviously should allow control with the Xbox 360 controller. Our multiplatform game Fireburst is also not that bad to develop for because most issues are handled by the great Unreal3 engine (except the fact of course that it is all ugly C++ and UnrealScript code, which just looks like C++ anyway and even has to be compiled), but it is still quite a lot more work than just doing a PC only game, especially because of optimizations and testing required for all platforms.

Enough rambling, I'm going to celebrate this day by installing Windows 2008 server (omg) because our pre-release version we had on there just ran out (warg) .. stupid thing has to be completely reinstalled, no upgrade option ..

For some time I had problems with StarCraft in Windows 7, after playing for 20-30min Windows 7 completely freezes keyboard, mouse and video (background programs and even Skype still work, but a hard reset was always the only way to get out of it). Since I was not playing much (just a few games a week) and it did not happen to me that much, I did not worry at first. But after starting to play ICCUP a few weeks ago with some friends, it just got annoying when I was crashing every 2-3 games. At first I was thinking maybe I overclocked way too much with my 4 Ghz, so I tried playing with normal 2.6 Ghz, but that did not make any difference. I also removed one of my graphic cards and tried playing on my primary screen instead of my third one. Nothing did help. It seems when closing all programs and making sure StarCraft only ran on one core (using the processor affinity setting in the task manager) the crashes occured less often, but I could not really fix this issue.

In Windows 7 Beta I also had some crashes (but usually not blue screens or whole computer freezes) with some games and applications, e.g. some Half-Life2 games like Garry's Mod or Left 4 Dead crashed a few times for me, but with WIndows 7 RC this did not happen anymore. In fact other than testing my overclocking settings I never had any crash or bluescreen in Windows 7 RC except for playing StarCraft.

After reading a bit on some forums (TeamLiquid, Blizzard, random Boards on the Internet) I saw many other people having the exact same problem: Windows 7 RC x64 + StarCraft does not work well together. In fact any other 2D DirectX from that time (like Age Of Empires 2, Worms Armageddon, etc.) seems also to cause random crashes in Windows 7 RC, especially on a quad core system. Some people also had another issue with the 256 colors, the trick here was to keep the Screen Resolution dialog open in Windows and then start the game.

Well, the solution is kinda retarded, but it seems to work (many people reported it and I did not have any problems since yesterday when I started using this trick): Kill the Explorer.exe process before starting StarCraft.exe and restart it only after closing StarCraft.exe again.

Since I also wanted to use the ICCUP Launcher, I wrote a little cmd script for this job. I also had to disable some of my programs like WallRotate to prevent the 256 colors messing up in StarCraft every time my desktop changes. Here is the .cmd file I use for starting StarCraft via the ICCUP  Launcher.exe plus all the additional tricks like waiting 2 seconds for the update check of Launcher.exe, killing WallRotate.exe (I commented it out, replace it with any other program you might want to close too) and Explorer.exe and restarting everything after closing StarCraft again.

rem First start the ICCUP launcher (it won't work without Explorer.exe)
cmd.exe /C start C:\Games\Starcraft\Launcher.exe

rem Wait a bit for ICCUP Launchers update check! Use ping trick to wait 2 sec!
ping 127.0.0.1 -n 2

rem First kill Explorer.exe, which messes up our colors in StarCraft
taskkill /f /IM explorer.exe

rem Also kill our desktop changer, which messes up the colors in game
rem taskkill /f /IM WallRotate.exe

rem Change to the StarCraft directory to make sure we run normally!
cd C:\Games\Starcraft\

rem And launch StarCraft ourselfs since we can't see the ICCUP launcher without Explorer.exe
rem Please note that /affinity 1 makes sure we only use our first core
rem Using all cores for StarCraft.exe can lead to crashes in Windows 7 :(
cmd.exe /C start /affinity 1 C:\Games\Starcraft\StarCraft.exe

rem Some early tests:
rem C:\Games\Starcraft\StarCraft.exe
rem C:\Games\Starcraft\Launcher.exe

rem Wait for the game to quit, press Enter to continue
pause

rem Restart Explorer.exe and WallRotate, that's it!
start explorer.exe
rem start C:\code\WallRotate\bin\Release\WallRotate.exe
exit

I want to quote from this great article I just read: Singletasking: The Next Trend in Web Working?

"Singletasking is just what it sounds like: approaching and tackling one task at a time, sequentially, instead of trying to do a whole bunch of things at once, as has become de rigeur in our modern multitasking age. If you’re like me, the thought is probably at least a little refreshing, and maybe more than a little appealing right off the bat.

The principle is sound. Take on one task at a time, and don’t begin another until the one you’ve already started is complete. It sounds simple, but you know as well as I do that actually implementing that kind of thing in real life will take a lot more effort than you might first think. For one, it means ignoring any urge to procrastinate, and making sure that you prioritize very carefully in advance, lest you realize too late that what you thought was most urgent actually could’ve taken a back seat to something else."

The article goes on with tools used to track work tasks and even ideas on only use one Tab in Firefox and only using one monitor. I can hardly agree with that, I currently have 10 tabs open (after reading all emails and stuff on the web) because I don't have time right now to finish reading those sites or because some things are still pending (waiting for a response and keeping the tab open as a reminder). I also currently look at 3 monitors. Well not really, I only write this on my right monitor, but I have my main task (some optimization work) on my main screen in the center in Visual Studio. I really hate putting anything above Visual Studio because then I won't focus on writing code, but do other stuff instead. I only do it if I have to write some document, have to do some web research or reading some documentation.

While I totally agree with doing (and finishing) one task at a time (that's actually how my TODO list works that I use in every single project for the last 6+ years), I just can't agree with just having one monitor (or just using one at a time like the writer of the article proposes). For example when opening an editor like for our Unreal3 game, you absolutely need 2+ monitors because else you just have 5 overlapping windows and spend half of your time moving them around trying to see whats behind them.

In Visual Studio I only have the Solution Explorer sometimes and some Find Results or Console Windows on my secondary screen, but all the screens are filled with useful stuff anyway. I also have basically 2 screens at once at my primary screen anyway because I always look at 2 concurrent tabs in Visual Studio (really can't live without it). That does not mean that I really multitask. I ALWAYS only work on one tab, but I can quickly look over, see all the variables and methods that I need and I can code much faster that way.

If you want to have some distractions when working I suggest only checking them out after completing a task. I usually do it as a reward kinda thing, like playing a game I wanted to test out only after I finished my current task. And then continue with the next task.

As Jamie Cansdale posted in my comments ^^:

There's now a version of TestDriven.Net what works with VS 2010:
http://weblogs.asp.net/nunitaddin/archive/2009/06/03/testdriven-net-2-22-support-for-visual-studio-2010-beta-1.aspx

Finally time to use VS2010 a little bit more, without TestDriven.Net it was just unusable before.

I worked all weekend on AI and game logic scripts in Lua and UnrealScript .. would like to talk about it, but since our game project is still not announced yet, I'm not allowed to .. grr ..

Also reading (or hearing) right now: http://www.brainrules.net/
And working through the great new release of the DLR and the new example script language Sympl: http://www.codeplex.com/dlr

Download link for CR_RandomBackgroundChanger now at CodePlex.

I used several tools over the years to change my background image automatically every few minutes. I even collected over 50k desktop images over the years ^^ While this is great and always fun to watch at changing backgrounds every few minutes, most of the time I do not even notice because all my screens are completely covered by windows.

As a programmer I have Visual Studio open most of the time, especially at work, where distractions like changing background images would have the biggest impact. Luckily Visual Studio has a boring white (or black for me) background and will not distract us that much. It would be terrible if Visual Studio would have changing background images that even make it harder to look at code and could be distracting having a background image changing every 5 minutes. Let's explore that!

First of all: How to even change or set a background image directly in the Visual Studio code window?
There is a great collection of gadgets from the SlickEdit guys, called Free SlickEdit Gadgets, which are completely free to use and include a feature to set a background image in the code editor. Searching for this on a search engine like Google isn't that easy, it is easier to find if you search for "Visual Studio Dancing Banana" :)

After you have installed the SlickEdit Gadgets (which currently only work on VS2005/VS2008, not on VS2010 yet), you should see a SlickEdit menu item. From there you can select Editor Gadgets, which just opens the Visual Studio Options dialog. There you can go to the Editor graphic tab and select a background image to be displayed in the editor. I use the following settings:

• Display an image: Checked
• Image textbox: The selected image
• Tile Radiobutton checked (use bottom left if you just want a small transparent image there, IMO not distracting enough!)
• Lock the image location checked
• Transparency checked and set to 75% (which means 25% visibility, 75% of your background color remains).

Now your code window might look like the following:


While this is kinda fun, it is not distracting enough yet. We need changing background images and preferably a way to quickly change the editor background image with a click at anytime in case it doesn't fit. First of all we need to find out where SlickEdit saves this background image. After searching for some SlickEdit setting file and not able to find any, I checked the registry and found this key: HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\SlickEdit\EditorGadgets\ImageFilename This is where the image name is saved. Just for fun I changed it to "C:\Users\Public\Pictures\Sample Pictures\Penguins.jpg", but obviously nothing happend because there was no reason for Visual Studio to reload and apply any new settings from the Options dialog. So how to we force that to happen?

Well, after trying to find some solution for that for a while, I gave up. There is no event I can trigger to update this and there is simply no functionality in the SlickEdit Gadgets to get this update working.

Since I would need to write something to set a new background image every few minutes anyway and this won't be easy or even impossible with my little Wallpaper Changer app I decided to write a VS addin myself. I started a new CodeRush plugin (which also runs on the free DXCore framework) and called it CR_RandomBackgroundChanger (a little bit too long for CodePlex, so there it is just called VSBackgroundChanger). Next I used the EditorPaint event and wrote the following code to display an image in the code editor:

 private void BackgroundChangerPlugIn_EditorPaint(EditorPaintEventArgs ea)
 {
     //Log.Write("BackgroundChangerPlugIn_EditorPaint");
     //tst: ea.DrawLine(1, 1, 50, Color.Red);
     if (currentBackgroundImage == null)
         currentBackgroundImage = Bitmap.FromFile(
             @"C:\Users\Public\Pictures\Sample Pictures\Desert.jpg");
     ea.Graphics.DrawImage(backgroundImage, 0, 0);
 } // BackgroundChangerPlugIn_EditorPaint(ea)
 

This produced terrible results. The image got displayed for a fraction of a second and then was replaced by the default background color. Using other events from the StandardPlugIn class in the CodeRush framework like EditorPaintBackground, EditorPaintForeground, EditorPaintLanguageElement, EditorvalidateClipRegion, etc. also did not help much, the background image was usually not displayed and even if it was flickering terribly and disappearing all the time.

So CodeRush was not very helpful in this particular instance, but no reason to give up. Actually I gave up that day after trying out so many things for many frustrating hours. The next day I tried something different: Use the NativeWindow class to intercept any event that is send to the Visual Studio code editor window (called View in CodeRush btw). Basically you derive from NativeWindow and then call the AssignHandle method to set the window handle (I just do it in the constructor). From then on you will get all WndProc events and you can decide whether to pass it on to the original WndProc or do something yourself.

public class ActiveViewBackgroundHelper : NativeWindow
{
    #region Constructor
    /// <summary>
    /// Create active view background helper class, each document view
    /// will get an instance of this class to handle the background drawing!
    /// </summary>
    public ActiveViewBackgroundHelper(IntPtr windowHandle)
    {
        base.AssignHandle(windowHandle);
    } // ActiveViewBackgroundHelper(windowHandle)
    #endregion

    #region WndProc
    /// <summary>
    /// WndProc, we are only interested at the WM_PAINT event!
    /// If a paint happens and we got a valid  we will call DrawBackgroundImage
    /// </summary>
    /// <param name="m"></param>
    protected override void WndProc(ref Message m)
    {
        // Your own code goes here
        base.WndProc(ref m);
    } // WndProc()
    #endregion
} // class ActiveViewBackgroundHelper

Now I was able to intercept all WM_ERASEBKGND and WM_PAINT messages. WM_ERASEBKGND turned out to be non-relevant because all the rendering happens in WM_PAINT. Otherr events like WM_NCPAINT and all the code rush events happened between a single WM_PAINT event anyway. So I had to dig deeper into WM_PAINT. I could not render a background image at the very start of WM_PAINT and if I aborted rendered right there, it would be displayed in the editor. But there was no text anymore making VS just useless. I had to figure out how to render text on top of my image by copying all the VS editor content to a bitmap and then later drawing it on top of my background image. To make all the flickering go away by drawing into a targetBitmap and then displaying that at once to the window (everything else flickers). There is also a lot of other optimizations and tricks in there. We will get back to the code in a little bit.

After working on this for the better part of the last weekend, I finally got it working and put some extra features in this CR_RandomBackgroundChanger addin, which you can see in this nice option screen for it. Note that I have a black background in Visual Studio, if you have a white one the preview image would be white with black text and it would have 25% of the background image mixed in.



You can download the addin and the source code at http://vsbackgroundchanger.codeplex.com/

I'm trying out CodePlex for this addin. I will probably improve it a little in the future and upgrade it to VS2010. I will also put up other projects on CodePlex like CR_Commenter and maybe some of my XNA games. I also plan to work a little bit more on my language and then put it on CodePlex too when I got a alpha version working with all basic features.

In the option screen of CR_RandomBackgroundChanger you can specify 5 settings:

• Enabled: Whether to use this addin at all. This allows you to quickly disable the addin and keeping all your previous settings once you need more distractions again.
• Directory: All images in this directory and all images in all the sub directories will be used for displaying random background images in your Visual Studio code editor. Please note that searching for 100 000+ images can take a few seconds (I tested it with that many), but the subsequent VS startups and every background image change will be very fast because of all the caching involved. The disadvantage of all the caching is that if you add new images or even delete the whole images folder, you need to update this directory to get all those changes reflected by the addin by either deleting the cached ImageFilenames.txt in your %TEMP% directory or by just selecting the same directory again.
• Transparency (default: 25% opacity): This is the most important setting because having colorful background images at 100% opacity can often make the already colorful editor text unreadable. If you know that your background images have mostly the same color as your code background you can use higher settings (for example I used some black only images with some stuff on the right in the beginning, while this was cool at first, it got boring after a while because I'm used to changing background images every few minutes). If you have photos or other images that do not really fit to having text on top of them use even lower opacity settings (e.g. 10%) to make Visual Studio text easier to read. You can mostly still see the background image if your code window is big enough and you have some empty lines in there :)
• Change image every x seconds: Use this setting to allow the addin to change the background images every 5 minutes (or use whatever number of minutes you like). Note that using 0 minutes will result in constantly changing backgrounds while you are typing and scrolling, not really sure if this is useful, but was good for testing ^^
• And finally the last settings is to use different background images for different document windows. If you have 20 documents opened in Visual Studio, every one of them will get a different background image (which can then change every 5 minutes too). This even makes it easier to figure out where you are, I probably need to improve the Tab-Rendering too to make it more useful like the ColorfulTabs addin for Firefox. And this is of course great fun, usually I get annoyed by having too many documents open so I often close them all after I get 50, but now I really like having many different background images open.

The option screen also features the Preview box, which lets you see your selected background images and tweak the transparency setting until it looks right for you. Once you close the Options screen all settings will be applied to your VS code editor. Please note that this addin only works whenever a WM_PAINT event happens. This is usually only in the active document you are working on. Most of the time the background image will stay even if you change to another tab on the other side (if you use 2 tabs at once, else you will never notice this anyway). Sometimes you see some black or white background lines drawn on top of your background image because just some text was updated, but this happens very rarely. I hope you enjoy the addin, I will probably use it for a long time and I will port it to VS2010 once CodeRush runs there or I'm using VS2010 (still waiting for a working TestDriven.net version). Hopefully implementing this addin with the MEF (Managed Extensibility Framework) is not hard.

Let's go back to the code. Last time I stopped talking about the WndProc method, which is most important for catching and handling the WM_PAINT event. From here we can call our DrawBackgroundImage method, which does all the magic. Please note that the source code for the addin is a lot more complex than the code presented here. This is mostly because of heavy optimizations and a lot of caching to make the code run as fast as possible (one of the most important things about this addin, it would be unusable if it makes VS slow).

#region WndProc
/// <summary>
/// WndProc, we are only interested at the WM_PAINT event!
/// If a paint happens and we got a valid  we will call DrawBackgroundImage
/// </summary>
/// <param name="m"></param>
protected override void WndProc(ref Message m)
{
    // Just make sure we mark the erase background message as handled.
    // Will not do anything anyway (all painting done in WM_PAINT).
    if (m.Msg == (int)WindowsMessages.WM_ERASEBKGND)
    {
        // Mark event is already handled
        m.Result = (IntPtr)1;
    } // if

    // Handle the paint event
    if (m.Msg == (int)WindowsMessages.WM_PAINT &&
        // Are we currently drawing our own background? Then make sure we do
        // not handle this and use the default message handling instead!
        drawingBackground == false &&
        // Make sure we got a view, else we don't have a windows handle!
        CodeRush.TextViews.Active != null &&
        // Check for invalid hwnd, then we can't paint backgroundImage!
        (int)CodeRush.TextViews.Active.Handle != 0)
    {
        TextView view = CodeRush.TextViews.Active;
        IntPtr activeWindowHandle = view.Handle;
        Rectangle rect = new Rectangle();
        Win32.GetUpdateRect(activeWindowHandle, ref rect, false);
        // Only proceed if we have a valid rect
        if (Win32.IsRectEmpty(ref rect) == false)
        {
            // Make sure we mark this flag so subsequent calls to WM_PAINT
            // will actually just paint the normal stuff, not just our
            // background rendering!
            drawingBackground = true;
            DrawBackgroundImage(activeWindowHandle, rect);
            drawingBackground = false;

            // Mark event is already handled
            m.Result = (IntPtr)1;

            // Do not call base.WndProc, we don't want to process it here!
            // Instead we invalidate inside DrawBackgroundImage and force
            // a new WM_PAINT event inside there, which will be executed
            // because drawingBackground is still true while in there.
            return;
        } // if (rect)
    } // if (WM_PAINT)

    base.WndProc(ref m);
} // WndProc()
#endregion
    

And finally the DrawBackgroundImage method, which copies the current editor text as an image, applies transparency to it. Then we draw our background image and the now transparent editor text on top into a helper targetBitmap, which is finally displayed on the screen at once (else we would get flickering issues). Please note that in the source code you can download there is a some testing code, e.g. for drawing bitmaps with transparency using ImageAttributes and SetColorMatrix, which works great, but is just too slow. Our approach is to pre-calculate the opacity in the backgroundImage and use that over and over again (3-4 times faster). There is still a performance penalty for all this painting, but any future optimizations will be hard. I recommend using 2 tabs, rendering is twice as fast (because only one side is updated when you type or scroll) and you should have a fast PC :)

#region DrawBackgroundImage
/// <summary>
/// This method draws the background and is called from WndProc whenever
/// it intercepts a WM_PAINT message. Again, some caching and confusing
/// optimized code is in here too, again for getting good performance.
/// </summary>
private void DrawBackgroundImage(IntPtr activeWindowHandle,
    Rectangle rect)
{
    // Create an image for storing the orginal editor screen.
    Bitmap sourceImage = new Bitmap(rect.Width, rect.Height);
    // Always create new graphics object, else we won't have current data
    Graphics sourceImageGraphics = Graphics.FromImage(sourceImage);

    // And grab current editor window content and copy it to it!
    IntPtr hdc = sourceImageGraphics.GetHdc();
    Win32.PrintWindow(activeWindowHandle, hdc, 1);
    sourceImageGraphics.ReleaseHdc(hdc);

    // Next find the color on the bottom right and use it as the
    // transparent color! Note: Antialasing will cause artifacts, make sure
    // the backgroundImage fits to the background color, e.g. by using a
    // lot of alpha transparency (<25% visibility)!
    backgroundColorPixel =
        sourceImage.GetPixel(rect.Width - 1, rect.Height - 1);
    sourceImage.MakeTransparent(backgroundColorPixel);

    // Create target image where we wanna paint to, this is important
    // because drawing directly to the VS window will produce flickering!
    Bitmap targetBitmap = new Bitmap(rect.Width, rect.Height);
    Graphics targetBitmapGraphics = Graphics.FromImage(targetBitmap);

    // Clear background with pixel color
    //not required, we draw solid image now (use this for transparent drawing):
    //targetBitmapGraphics.Clear(backgroundColorPixel);

    // Draw background image (tiled and transparent if specified)
    float transparency = Options.transparency / 100.0f;
    Image backgroundImage = GetBackgroundImage((int)activeWindowHandle,
        backgroundColorPixel, transparency);
    
    // Do the drawing as many times as we need to tile to fill everything!
    for (int y = 0; y < rect.Height; y+=backgroundImage.Height)
        for (int x = 0; x < rect.Width; x += backgroundImage.Width)
        {
            targetBitmapGraphics.DrawImage(backgroundImage, x, y);
        } // for for

    // Now draw source image on top (text and foreground stuff), else we
    // would only see our distracting background image and while that is
    // fun, we sometimes still need to be productive and see the editor
    // text ^^
    targetBitmapGraphics.DrawImage(sourceImage, 0, 0);

    // Finally draw on the VS window, but only do one single draw here
    // to make sure we do not have any flickering!
    Graphics graphics = Graphics.FromHwnd(activeWindowHandle);
    graphics.DrawImage(targetBitmap, 0, 0);

    // This is important: Validate the rect so all this can now be
    // displayed! All WM_PAINT calls during this method will
    // be handled normally (without our background painting), which is
    // used for normal updates because we mess everything up ^^
    Win32.ValidateRect(activeWindowHandle, ref rect);

    // Dispose everything we do not need anymore
    // Note: We always have to create new graphics and dispose them here,
    // else updating sourceImage, targetBitmap, etc. does not work!
    if (graphics != null)
        graphics.Dispose();
    if (sourceImageGraphics != null)
        sourceImageGraphics.Dispose();
    if (targetBitmapGraphics != null)
        targetBitmapGraphics.Dispose();
} // DrawBackgroundImage(activeWindowHandle, rect)
#endregion

For more information check out the source code, the important code for the addin is in ActiveViewBackgroundHelper.cs and the Options and PlugIn classes. The rest of the classes are just helpers for logging, string operations, random methods and Win32 helpers for some calls.

Download this addin at http://vsbackgroundchanger.codeplex.com/

I hope you will enjoy this addin. Here are 2 more screens from using this addin with different background colors and images:

White background theme (25% opacity):


Black background theme (25% opacity):

Last week when I installed VS2010 I played around with it for a few minutes, but I've been very busy at work because our game is at its final stage and we got a lot of heat going on (this is not a hint, no no, no hint from me, I'm not allowed to talk about it. Damn it, this could be a hint).

Currently the most interesting new feature besides the cool VS2010 IDE is the Parallel Extensions for me. Sadly the IDE still unusable for real work IMO because all the addins just don't work, there isn't even a fix for TestDriven.net yet, Jamie Cansdale is probably busy too ^^

VS2010 support for parallel programming goes beyond just adding a few extra classes in .NET 4.0: You also got a great IDE implementation which lots of useful features and new tool debugging and profiling windows for checking out parallel tasks, threads and the scheduling. Next there are native C++ libraries that work with Parallel Extensions too (using lambda functions) and work good together with STL. You can check out all the new features at the official VS2010 page!

Let's take a quick look on how to use these Parallel Extensions. I wrote this in a few minutes last week, I was just too lazy (I mean busy of course) to post it yet, it is obviously very simple stuff, but was still useful to test out some of the new parallel IDE features and check out some new .NET 4.0 classes. First of all let's do a boring foreach loop, which displays numbers from 0 to 9, which get added to expectedTotalNum. This should obviously be 45 because sum(0..9)=45. Later we will do some parallel foreach adding and check if we got the same result. Since it does not matter in which order we add these numbers, it is also a good test to just start a bunch of parallel tasks and let them do their work. Obviously you would never write parallel code just to add some numbers, but this should illustrate the point and it does not hurt your performance much anyway (as long as you have a few lines of code executing that take more than a few instructions).

 // Initialize a list for some parallel testing :)
 List<int> someInts = new List<int>();
 for (int num = 0; num < 10; num++)
   someInts.Add(num);

 // Print out numbers sequentially
 int expectedTotalNum = 0;
 foreach (int num in someInts)
 {
   Console.WriteLine("sequential num=" + num);
   expectedTotalNum += num;
 }

 Console.WriteLine("expectedTotalNum=" + expectedTotalNum);

This outputs the obvious sequential adding of 0 to 9 to expectedTotalNum, which is 45 at the end of the loop:

sequential num=0
sequential num=1
sequential num=2
sequential num=3
sequential num=4
sequential num=5
sequential num=6
sequential num=7
sequential num=8
sequential num=9
expectedTotalNum=45

Now let's do the same in parallel, just by replacing foreach with Parallel.ForEach:

 // And do it with the new Parallel Programming classes in .NET 4
 int totalNum = 0;
 System.Threading.Parallel.ForEach(someInts, num =>
   {
     Console.WriteLine("parallel num=" + num);
     totalNum += num;
   });
 Console.WriteLine("totalNum=" + totalNum);

While the result is still the same because totalNum is 45 at the end of this loop, we get there in a different way. As you can see this is a little bit confusing at first because the adding does not happen sequentially anymore, but in parallel instead. Also note that this output can change when you execute it again and can even be much different on different platforms with different number of CPUs:

parallel num=0
parallel num=2
parallel num=5
parallel num=7
parallel num=8
parallel num=9
parallel num=6
parallel num=3
parallel num=4
parallel num=1
totalNum=45

Okay, good stuff so far, but you might not always have a foreach loop and you might not want to wait for it to complete anyway. Maybe you just have some work tasks that need to be executed, no matter in what order and you might not even care if they complete their work right away or in a little bit. A method could just add some work that has to be done and then return while the work tasks are executed in the background. This is when you would have used the Thread or ThreadPool classes in the past, which are great on their own, but you always have some setup code and it was hard to test and you could not use them all over the place because setting up threads is a costly operation and if your work is just few lines of code, it was always a better idea to execute it right away. But fear no more, now you can just create new tasks with the new Task class in the System.Threading namespace. This is a much easier job and has many advantages because .NET 4.0 handles all the thread creation for you, will reuse threads once they are completed with their task and all this works in a very performant way. Setting up tasks is a little bit more work than just executing Parallel.ForEach, but it allows much greater flexibility and you can add more tasks from whereever you are. Tasks can even have children and you can have a lot of complex code using all these tasks. Testing multi-threaded code is obviously harder than just writing sequential code, but with all the great additions to the IDE, it is now easier than ever with the new Parallel Tasks window and by checking out the Parallel Stacks, which shows you all the running threads and were all the task code is.

The following code will create 10 tasks and do the same thing as above, but with several additions. To be able to wait for all the tasks to finish we will add all 10 tasks to the allTasks list. We also have to make sure that our local foreach variable does not change while we are executing tasks because the foreach loop will quickly create all tasks, but might not execute them right away theirfore using num can cause problems. Instead we just create a local copy of num and use that instead, which can't change because we never increase it like we do with num. Finally we add some boring thread sleeping to make it easier to debug this code and check out whats going on by adding breakpoints. Without the sleep the code still works, but checking out the Parallel Tasks and Parallel Stacks windows will most likely give us no results or only the last few tasks that are still being executed at the end of the foreach loop because the tasks are so simple and executed very quickly. We even wait a little after the foreach to make sure all the tasks have been added and are being executed right now or are scheduled (waiting for execution).

 // And finally some tasks, yeah!
 totalNum = 0;
 var allTasks = new List<Task>();
 foreach (int num in someInts)
 {
   // We need a local variable for num because num itself can change
   // at the end of this loop before the task might even be executed!
   int numToBeAdded = num;
   allTasks.Add(Task.Factory.StartNew(delegate
   {
     totalNum += numToBeAdded;
     Console.WriteLine("Adding " + numToBeAdded + " in task with id=" + Task.Current.Id);
     // Wait a little for checking out the tasks in the new Tasks window in VS2010!
     Thread.Sleep(1000);
   }));
 } // foreach
 Console.WriteLine("Done with foreach loop. Tasks might still be pending");

 // Wait a little for all tasks to start
 Thread.Sleep(100);
 Task.WaitAll(allTasks.ToArray());

 // And finally return the result (45 again if everything worked)
 Console.WriteLine("totalNum=" + totalNum);

First of all the results again, which is 45 again. This only work with numToBeAdded, if you use num instead it will sometimes give you different results because at the time you execute a task num already might have changed, especially if you have more tasks than CPUs used for execution and time consuming code like Console.Writeline or even a Thread.Sleep is in there!

Adding 1 in task with id=1
Adding 7 in task with id=6
Adding 8 in task with id=7
Adding 6 in task with id=5
Adding 4 in task with id=3
Adding 3 in task with id=4
Done with foreach loop. Tasks might still be pending
Adding 9 in task with id=8
Adding 0 in task with id=10
Adding 5 in task with id=2
Adding 2 in task with id=9
totalNum=45

As you can see this even looks more confusing than the way Parallel.ForEach added those numbers because while we might create those tasks sequentially (number 1-10) it does not mean there are also executed in exactly that way. To make it easier to check these things out there is the Parallel Tasks window, which shows the following after starting all 10 tasks by adding a breakpoint after Thread.Sleep(100) just before we wait for all tasks to complete:



Not only do we see all over our 10 tasks here, we also see right away that 8 of them are currently being executed (because I have 8 CPUs with my hyper-threaded i7) and all of them are waiting because of our stupid Thread.Sleep(1000) we added for each of those tasks. A second later those tasks are done and the last 2 are also executed, most likely with 2 thread ids already created earlier. You can click on each task and see where it is currently executing in the source code and you can also check out all the thread information in the normal Threads window. But even more useful than the Parallel Tasks window is the Parallel Stacks, which shows how all this code is related, which task or thread created which new task and so on:



All good stuff. While I already have some ideas how to use this on some of my current projects, I have not ported anything to .NET 4.0 / VS2010 yet because of the many issues I have with the IDE (no addins, color theme not really working, I always have to reset it when starting VS2010, also I don't like that the project and solution formats have changed so I cannot easily switch back to VS2008, etc.). But hopefully more and more addins will work for VS2010 and some of the issues are fixed, then it will be great to use all this new .NET 4.0 stuff (dynamics, parallel extensions, MEF, etc.).

VS2010 beta came out a few hours ago, installing it took way too long (over an hour, VS2008 takes less than 10 min for me) and it seems some addins don't install or just don't work, but other than that everything else seems to work great:

• TestDriven.NET provides an VS2010 option and it will appear in Addins, but the Menu Items will not appear anywhere. Just use keyboard shortcuts and everything still works :) .. at least it worked for a short while (see below). Hopefully Jamie Cansdale fixes this soon because without TestDriven.NET I can't really use VS2010 right now. MS provides its own unit testing framework, which is implemented into VS2010 and even available in the Pro edition now (was only in more advanced versions before), but I still don't like it. It does not output anything to the console except exceptions; it is not possible to just start some Ad-Hoc unit test and for all my functional tests it is also useless. For just "normal" unit test projects VS test integration is quite ok, but I would suggest trying out NUnit or xUnit together with TestDriven.NET, which is just more flexible, easier to use and improves productivity IMO.
  
• Completely unable to install any CodeRush version (neither 2, 3, 9.1, Xpress), so I'm also unable to use my CR_Commenter :( hopefully DevExpress provides a version soon, they always were very quick with early VS2005 and VS2008 support!
• Most other addins I tried also did not install or just do not appear anywhere (only in VS2008)
• One of my own addins I wrote 2 years ago for VS2008 also does not appear, it probably needs to be configured to appear in the VisualStudio/10 reg key!

Most annoying:

• Even though Rico Mariani (the .NET performance god) is helping out the VS team to give VS2010 better performance, it currently lags all over the place. I had 10 second delays in Options, Add References and Renaming dialogs already. Seems to be always an issue with opening up stuff for the first time. After a while everything seems to be fast, but I would not say this is faster than VS2008 yet.
• Only tried this at my home PC till now, but everytime I close VS2010 I cannot open it up again until I restart (or at least log off/log on) again, which is REALLY annoying. I will just keep VS2010 open now all the time and hope it never crashes ^^ The start splash-screen is also messed up and completely black. Maybe this is only on my home PC because I tried to install so many addins, will try this tomorrow at work too.
  Update 2009-05-18: I just found out what was causing this. Some addins tried to load, but failed for some reasons and then the addin loading code locks up or whatever and VS never finishes starting up (the process devenv.exe is still there, but you just don't see any IDE window, they are never created). It seems to be related to the TestDriven.NET-2.21.2448 version, I also tried TestDriven.NET-2.20.2438 - same problem :( (Jamie Cansdale, the creator of TestDriven.NET was also recently blogging about this). Then I used , which also already has VS2010 support and that seems to work fine (no menus, but keyboard shortcuts work fine). I posted more details on Jamie's blog!
  

Every project I have opened, had to be converted to the new VS 10 format, which is kinda stupid because it seems nothing has changed. All the code works just fine and no changes were required. I will start using some .NET 4.0 features and test out stuff in the next days, especially using Parallel Programming and MEF (Managed Extensibility Framework). Probably will play around with the C# 4.0 dynamic keyword a bit also, but can't think of anything yet where I would really need it. Maybe to talk easier to IronPython or my own DLR language ..

Cool features:

• Start page is pretty good, easily customizable, but I will probably never see it as I setup my VS to always load the last solution (like my Firefox ^^)
• It is easy to drag tabs around and move them to different screens. Finally I can use some multi-monitoring.
• Old exported VS2008 settings also work just fine in VS2010, even coloring :) For some strange reason the background color is always white when I start VS2010 for the first time, then after going to options and closing it, it will be the correct color (black for me).
• Overall the IDE is very nice and fits perfectly into my color theme. Its also a lot easier to see whats going on, which tab is selected and positioning tool windows all over the place is better than ever before.

Go check out VS2010 yourself, available on MSDN right now and on MS site on Wednesday for everyone else.

While I'm waiting for VS2010 beta today (refreshing blogs and MSDN every few minutes :D), I just found out that you can have your Visual Studio Tabs be automatically ordered by adding the following registry key in HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0:

• UseMRUDocOrdering: 1 (DWORD)

Now when you select any tab in Visual Studio it will automatically move to the very front thus keeping your recent documents at the start. It is kinda weird at first, but it seems to be a very useful feature. I like it already :)



In case you don't want to create the registry key yourself and want me to help you out modifying your registry (always a safe idea), just execute this file to do the job:

• VS9UseMRUDocOrdering.reg (.26 KB)
• Update: Does not seem to work yet in VS2010, maybe use /10.0_Config key instead?: VS10UseMRUDocOrdering.reg (.26 KB)

PS: Since my brother is running a TOR exit node server Google does not like when I do searches anymore because it thinks I search too much stuff now and might be infected with viruses and trojans (yeah, sure ..). I could use Google Custom Search sites like Blackle.com, which return the exact same results as Google, but for now I'm trying out Yahoo once again (was using it a lot almost 15 years ago, when there was no Google). Sometimes Yahoo is stupid and does not give good answers, e.g. searching for "year yahoo was launched" gives you random answers on Yahoo, but google returns the Yahoo Questions link with the correct answer as the first result, wtf?

However, most of the time you do not get stupid SEO optimized search results and totally messed up product searches you get when using Google or Live Search. As an Microsoft MVP I also get asked a lot why I would not use Live Search and Microsoft is constantly trying to get more people (and MVPs) using Live Search, but the results are very close to Google most of the time anyway and the interface is just strange if you are used to Google for 10 years (the English Live Search is actually nice with a new image every day, but the German version is totally messed up, just white boxes??). Yahoo at least gives different results, which are sometimes better, sometimes worse .. Maybe we all should switch search engines from time to time, its really stupid Google has 95% market share here in Germany almost and it returns so crappy results so often (finding drivers has become 10 times as hard in the last years, searching files or torrents is impossible, because you only get SEO sites, no real results, product searches are totally messed up, Google groups is not used much anymore, etc.). Okay, enough ranting about search engines .. time to refresh MSDN again.

Visual Studio 2010 beta is apparently coming soon. Since I used VS2005 beta and VS2008 beta when they came out, I will be an early adopter once again. I really want to use the cool Parallel APIs in VS2010 and C# 4.0 dynamic stuff :)

Another topic: I try to synchronize several of my servers at different locations every day. This way it does not matter where I download or create a file, I still can use it the next day wherever I am (home, work, different work ^^). I use SyncBackSE for that job, great tool, but the problem with it is that on slow internet connections (e.g. slow DSL at work right now) it takes forever to even scan the directories. Its about 30000 directories that have to be synchronized and this takes a lot of hours, especially if the internet connection is already doing something else. Obviously uploading and downloading a few GB can also take a lot of time, but in 95% of the cases nothing or just a few MB change every day.

Because searching sub directories is so slow I tried to make it faster by removing directories when not necessary (just compressing 100 directories into a zip file, removing empty directories or old files, etc.). But this is obviously a lot of work for this many directories and I can easily overlook that in sub directory 3592 there is 1500 empty folders of some crap from some years ago no one will ever need again. For this reason I quickly wrote this little tool called KillEmptyDirectories, you can see a picture on the right! It only took me an hour to write, so there is no rocket science here and it is very straight forward. I did not even unit test it, I just tested it by executing on several directories. It also can remove hidden and read-only files by changing the file attributes (otherwise File.Delete will always throw UnauthorizedAccessException). I tested it on around 30000 directories and was able to kill around 8000 of those (deleting a lot of old stuff) ^^ with some additional compression of unused older directories I was able to get it down to around 3000 directories (from 30000), so the scanning process is now about 10 times faster.

If you want to try it out, you can download it here. But use with care, you can obviously kill directories with it and by using the "Always kill these directories" options you can even delete directories with files in it. An extra confirmation message will appear if you try to use that feature.

• KillEmptyDirectories.exe (12.5 KB)
• KillEmptyDirectories.zip (11.83 KB)

And this is the main function that does all the directory killing:

/// <summary>
/// Recursively kill directories, will be called as many times as we have
/// sub directories (can be many thousand times). Will always go through
/// all subdirectories first in case we can remove them, which makes it
/// much easier to delete directories with just empty sub directories.
/// </summary>
private int RecursivelyKillDirectories(string directory,
    string[] includeFiles, string[] alwaysKillDirectories,
    bool alwaysKillThisDirectory)
{
    int directoriesKilled = 0;
    string[] subDirectories = Directory.GetDirectories(directory);

    // Only delete this directory if there are no useful files in here!
    // Note: GetFileName will give us the last part of the directory! 
    if (alwaysKillDirectories.Contains(Path.GetFileName(directory)))
        alwaysKillThisDirectory = true;

    // Handle subdirectories always first (maybe we can kill them too)
    foreach (string subDir in subDirectories)
        directoriesKilled += RecursivelyKillDirectories(subDir, includeFiles,
            alwaysKillDirectories, alwaysKillThisDirectory);

    // Get all files here and count how many of those we can ignore
    string[] files = Directory.GetFiles(directory);
    int ignoreFileCount = 0;
    foreach (string file in files)
        if (includeFiles.Contains(Path.GetFileName(file)))
            ignoreFileCount++;

    // Only found ignored files (or no files at all) or do we want to kill
    // this directory anyway?
    if (files.Length == ignoreFileCount ||
        alwaysKillThisDirectory)
    {
        // Check again if we don't have any sub directories left here
        // Maybe the subdirectories above were already killed now!
        subDirectories = Directory.GetDirectories(directory);
        if (subDirectories.Length == 0 ||
            alwaysKillThisDirectory)
        {
            try
            {
                // Kill all files in it (only ignored files anyway)
                foreach (string file in files)
                {
                    // Make sure we can delete hidden and readonly files
                    FileAttributes attributes = File.GetAttributes(file);
                    if ((attributes & FileAttributes.ReadOnly) != 0 ||
                        (attributes & FileAttributes.Hidden) != 0)
                        File.SetAttributes(file, FileAttributes.Normal);
                    File.Delete(file);
                } // foreach
                Directory.Delete(directory);

                // We killed something, yeah
                directoriesKilled++;
            } // try
            catch (Exception ex)
            {
                MessageBox.Show("Failed to delete " + directory + ": " +
                    ex.ToString());
            } // catch
        } // if
    } // if

    // Most of the time nothing was killed
    return directoriesKilled;
} // RecursivelyKillDirectories(directory, includeFiles)

This week I wanted to test using a Ramdisk (Ramdrive using just your main RAM) for compiling Visual Studio projects. Playing games or doing other disk intensive stuff would be great too, but most games are just way too big and smaller older games load pretty quick anyway.

While copying files and benchmarking a Ramdisk is incredibly fast with 4-6 GB/s (theoretically my ram should almost reach 10 GB/s, but well that's already way fast enough). Since I use Windows 7 RC since Friday when it came out on MSDN I had a lot of problems finding Ramdisk programs that actually work and do not crash every 2 seconds. Currently I use RamdiskVE by Cenatek, but the company does not exist anymore because it was bought by Dataram, which provides RamDisk on their own now. I was however unable to run Dataram's Ramdisk on Windows 7, it constantly crashes and has also many other limitations. Here is also a discussion on the Ocz Forum about Ramdisks if you want to check out some of the products for yourself.

The Ramdisk is also useful for Temp files, IE Temp files and other Scratch Disk functionality (e.g. Photoshop), but you should only use those for programs to stupid to use more of your Ram (e.g. because a program is 32bit and you have way more memory in your 64bit system). It also will increase the lifetime of your hard disks or SSDs in case you write a lot of stuff on your Ramdisk because the Ramdisk will only be loaded once at start up and saved once when shutting down your PC. But you should be aware that in case of a crash, you will obviously lose all the changed content on your Ramdisk.

For testing I used my new rig with a i7 920 D0 CPU and 12 GB ram, which I overclocked from 2.66Ghz to ~4.4Ghz. This is pretty fast, for example the For vs Foreach Performance application from the last blog post is twice as fast as my 6600 CPU from last week (all times cut in half basically, which is more than I expected).

I also tested quite a lot of hard disks and SSDs, which was kinda interesting because I learned that it does not matter if you mix totally different hard drives into a Raid 0 as long both are similarly fast. For example I tried using 2 older 160 GB disks in a Raid 0 and was only able to get around 60mb/s, which is slower than a single Raptor hard disk, but putting a new fast Samsung 250 GB and a Raptor 150 GB together into a Raid 0 gave me around 160mb/s. To keep things crazy I also added another Intel x25-M SSD to the one I already had and put them into a Raid 0 too, which is amazingly fast (>450mb/s). For highest speed you should always make sure to Enable Write-Through Cache on the Raid controller (in my case a Intel software raid with the ICH10R controller) AND to enable write caching and finally turn off the Windows write-cache buffer flushing (both can be found in hardware->disk->policies) for your disks. Please note that I do not care about data redundancy since everything I do is saved on a server with a Raid 5 anyway. If one of my Raid 0s would fail it would just be annoying to reinstall everything, but I would not lose any of my work or files.

So lets see how much benefit you actually get from compiling several different projects on a fast PC with those different disk configurations. The compile time is obviously heavily dependent on your CPU speed, but I tried to measure how much the total time changes just by using different disks. The following example shows a full recompile of one of my bigger solutions:



All compiles are full recompiles from scratch (no immediate files yet). For testing the DLR (change set 23173, about 25 MB in 20 C# projects), one of my solutions with 5 C# projects, 1 C++ project (~43 MB) and finally the good old Quake3 v1.32 source code (5 MB of c code). Since I do not compile much C++ at home I was to lazy to test bigger C++ projects, but I would guess most times would just scale and the conclusion would be the same.

4 Tests were executed:

• Loading Visual Studio 2008 and opening the each solution (average time),
• Compiling the DLR and running the ToyConsole sample. This will generate around 25 MB of files (17 MB of those are .pdb), ~60 files.
• Compiling and running my own solution (~43 MB, 5 C# projects, 1 C++ project). Generates ~47 MB (lots of copying, >300 files).
• and finally compiling Quake3 (~500 c files).

Most of the tests were done several times, but I stopped all of them with this cool freeware stopwatch called PC Chrono. Keep that in mind, the results will not be very accurate. Each test was done on the following drives:

• Good old Raptor Hdd (one of the fastest desktop disks you can get, almost empty for better testing)
• Single Ocz Ssd with 150mb/s read/write (remember that I complained about the bad JMicron controller on it way back in February)
• Intel X25-M Ssd Raid 0 Array with 450mb/s read, 140mb/s write (nice ^^)
• And finally the Ramdisk with 4500mb/s read+write (at least)

Disk/Compile times	Loading VS+solution	Compiling DLR+start ToyConsole	Compile&Run own solution	Quake3 Compile
Good old Raptor Hdd	3.75s	9.01s	2.95s	17.57s
Single Ocz Ssd 150mb/s	2.96s	10.47s	3.24s	22.67s
Intel X25-M Sdd Raid 0 Array	1.26s	7.79s	2.88s	17.53s
Ramdisk with 4500mb/s	1.43s	7.70s	2.51s	15.89s

With this data the following fancy graph was build. It shows that there are some improvements in several areas, but as long as you are not limited by your disk speed or IOs (amount of input/output operations you can do per second), you do not get much benefit from way faster drives (the ram disk is at least 45 times faster than the hard disk I used, but the performance improvement is maybe 10 or 20%):



Since it is so much fun using the Intel SSD Raid 0, I will keep using it. It is also big enough to use for all my programs and games. For example launching Left 4 Dead levels is 2-3 times faster than before. But I will probably not continue to use the Ramdisk for compiling. I can just keep everything on the SSD raid, which will not run out of space as soon as my small Ramdisk. Maybe when I do some file-intensive stuff in the future I will try out the Ramdisk idea again.

For now my advise for getting the fastest Visual Studio experience would be getting the fastest CPU you can (i7 920 is pretty nice, overclocking is important too, without it my tests would be 40% slower, 8 hyper-threads might also be useful in the future), enough Ram (4-6 GB), a fast disk like the Intel X25-M and of course using Windows 7 (since everything responses much faster). Then you can really have a lot of fun compiling big and small projects, and playing games, and doing other stuff on your PC.

BTW: My brother also just blogged about Windows 7 RC and using RamdiskVE. He also noticed that the network layer in Windows 7 is way faster than before in Windows Vista for him, in Vista he had 30-60mb/s max, now it is like 110mb/s copying files over the network:

Today I had a discussion with a few other programmers (Hi ViperDK and Timo ^^) about foreach and for loop performance. Everyone had his own ideas and experiences, but we talked mostly about XNA on the Xbox 360 (using the compact framework, that does not like foreach, just use for all the time in tight render loops for your XNA games, foreach will create too much IEnumerator garbage that will not be collected) or other special cases (huge arrays or lists).

Anyway, just guessing around is obviously not as good as actually writing a performance test application and see whats going on (see end of this article for download link). Again, we need a disclaimer: You will probably never write loops with 1 billion iterations, these numbers are similar for less iterations, but you should always try to write clean code and ONLY optimize when it is required after finding performance issues! Lets go directly to the results for 1 billion for or foreach iterations in release mode (will loop through a 10k array or list for 100k times, see below for code):

As you can see there are some major differences between using foreach with an array or using a generic list (boxing/unboxing is not the issue here, you would not be able to archive this performance with an ArrayList). Another thing you can notice right away is that using simple arrays is always faster than using dynamic lists, even converting them from a list to an array via .ToArray() and then executing the foreach is much faster.

It also makes a HUGE difference executing this in debug mode (I execute my code almost 99% in debug mode), everything is 4 times slower for this special performance test. Except for Foreach+List, which is in comparison not that bad in debug mode. What is really bad is For+List, which almost takes 16 seconds to execute 1 billion times (again using my Core 2 Duo 6600 CPU with 4 GB Ram).



Thanks to some helper methods the code for this performance test is pretty short and easy (check end of this article for the source code download link):

 Stopwatch timer = new Stopwatch();
 // Run through all tests
 for (int test = 0; test < 8; test++)
 {
     // Find out which test we wanna make
     bool rememberLength = (test / 4) % 2 == 1;
     bool useForeach = (test / 2) % 2 == 1;
     bool useList = test % 2 == 1;

     // Start timer for test 
     timer.Start();

     // Execute
     int result = RunForOrForeachLoops(rememberLength, useForeach, useList);

     // Stop timer
     timer.Stop();

     // Skip this test if we got no result!
     if (result == 0)
         continue;

     // Report results!
     resultText = ...
     Console.WriteLine(resultText + ": " + timer.ElapsedMilliseconds + "ms");

     // Reset timer for next run
     timer.Reset();
 } // for

The interesting code obviously happens in RunForOrForeachLoops. It basically executes the 1 billion iterations in several different ways (8 ways, as you can guess by the for main test for loop). All of those produce the same result, half of the tests use int[] intArray, the other half uses List<int> intList. And again half of the tests uses just for loops, the other half uses foreach for the iterations. Lets examine the useForeach=true and rememberLength=false cases (btw: Iterations is 100000 and the intList and intArrays have a size of 10000 elements):

if (useForeach)
{
    if (useList)
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intList)
                result += val;
        } // for
    } // if
    else
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intArray)
                result += val;
        } // for
    } // else
} // if
else // useForeach == false
...

To figure out the differences between those few lines of code in RunForOrForeachLoops, that almost look the same, we have go to IL code once again. First lets check out what the fastest code block Foreach+Array is being compiled to in IL:

result = 0;
foreach (int val in intArray)
    result += val;

This becomes in IL:

 L_0007: ldsfld int32[] TestSimpleForeachApp.Program::intArray
 L_000c: stloc.2 
 L_000d: ldc.i4.0 
 L_000e: stloc.3 
 L_000f: br.s L_001d
 L_0011: ldloc.2 
 L_0012: ldloc.3 
 L_0013: ldelem.i4 
 L_0014: stloc.1 
 L_0015: ldloc.0 
 L_0016: ldloc.1 
 L_0017: add 
 L_0018: stloc.0 
 L_0019: ldloc.3 
 L_001a: ldc.i4.1 
 L_001b: add 
 L_001c: stloc.3 
 L_001d: ldloc.3 
 L_001e: ldloc.2 
 L_001f: ldlen 
 L_0020: conv.i4 
 L_0021: blt.s L_0011

In L_0021 we jump up to L_0011 as long as we are not done with the loop yet. Inside all the result += val happens, but we do not see any slow code, there are no calls to any methods, there is no unboxing and this all can probably be executed quickly after being complied to assembler by the JIT (just-in-time) compiler.

On the other hand, if we look at pretty much the same code, just with intList instead of intArray, the whole story changes:

result = 0;
foreach (int val in intList)
    result += val;

becomes much more complex IL code with lots of function calls, the creation of an Enumerator and there is even a dispose at the end (hello GC):

  L_0007: ldsfld class [mscorlib]System.Collections.Generic.List`1<int32> TestSimpleForeachApp.Program::intList
  L_000c: callvirt instance valuetype [mscorlib]System.Collections.Generic.List`1/Enumerator<!0>
[mscorlib]System.Collections.Generic.List`1<int32>::GetEnumerator()
  L_0011: stloc.2 
  L_0012: br.s L_0020
  L_0014: ldloca.s CS$5$0000
  L_0016: call instance !0 [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::get_Current()
  L_001b: stloc.1 
  L_001c: ldloc.0 
  L_001d: ldloc.1 
  L_001e: add 
  L_001f: stloc.0 
  L_0020: ldloca.s CS$5$0000
  L_0022: call instance bool [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::MoveNext()
  L_0027: brtrue.s L_0014
  L_0029: leave.s L_0039
  L_002b: ldloca.s CS$5$0000
  L_002d: constrained [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>
  L_0033: callvirt instance void [mscorlib]System.IDisposable::Dispose()
  L_0038: endfinally 

What even looks worst too me is the code that is executed in MoveNext (get_Current just returns this.current, no extra instructions here). Since the IL code is very long and confusing (31 IL instructions), here is the C# code of MoveNext:

public bool MoveNext()
{
  List<T> list = this.list;
  if ((this.version == list._version) && (this.index < list._size))
  {
    this.current = list._items[this.index];
    this.index++;
    return true;
  }
  return this.MoveNextRare();
}

And MoveNextRare could also be called at the end of MoveNext if something changes for the list (not the case in our example). But we have to keep in mind that this is just IL code, not actual code that will be executed on the CPU. The JIT compiler can still optimize things out, which we won't see here. But something must be going on since Foreach+List is definiately slower than using all the other approaches to get through all the values in intList. I recommend Foreach+Array converted from List because the IL for Foreach+Array is really fast and does not use any extra methods or Enumerators, especially if you know that the array won't change much and if you do not have to call .ToArray every time when you do a foreach.

And here is the sample project for all this performance testing fun:

• TestForEachPerformance.zip (4.56 KB) project (C#)
  

The last few days I was working a little bit with the DLR and got to a point today where I was wondering about the performance of a very simple while loop in the DLR, specifically in the ToyScript DLR sample project and of course IronPython. I am totally aware that the following comparison DOES NOT really provide accurate information about the actual performance of the languages and techniques used. Please keep this in mind when reading this article, it only covers a very certain aspect of all used languages. More specially how they perform when adding numbers in a while loop a lot of times! Some of it reflects that compiled languages are faster than dynamic languages, but you should not compare the absolute values, Python or Lua is not really 100 times slower, in many cases you can reach almost similar performance as long as you implement or call performance-critical code in a clever way (e.g. calling C++ code, using frameworks, etc.).

I was only interested about very ruff comparisons to figure out which ways are fast and which languages or techniques are kinda slow (click image on the right for a quick overview). Some code written in Assembler or direct IL code execution was really fast (duh, no wonder), but all execution times of C++, C#, Assembler or IL are way faster than all other dynamic languages or DLR approaches. The important thing for me was to figure out why dynamic languages like IronPython or ToyScript on top of the DLR were like 100 times slower for this specific code. At the end of writing this article I was able to provide pretty good performance with the DLR, which is just 2-3 times slower than the fastest execution time. That's pretty good and could probably be even improved more.

Basically I used the following code sample, which adds 1.0 to counter 10 million times while decreasing the loops variable until it reaches 0. The following code block is not valid syntax in any of the presented languages, but it is very similar to Python. In languages that use compile time types, counter would be a declared as a double (64bit floating point number), most dynamic languages use double as the run-time type for floating point numbers also.

loops = 10000000
counter = 0.0
while loops > 0
  loops = loops - 1
  counter = counter + 1.0

print counter

Languages and techniques used and tested (all with detailed code examples and explanations):

• C# 3.0
• Native C++
• Assembler
• IronPython 2.6
• ToyScript
• Python 2.6
• Python 3.1
• Lua 5.1.4
• Script.NET
• IL Emit
• Irony + DLR (dummy)
• DLR AST with objects
• DLR AST with doubles

Before boring you with all the different implementation details, here is the fancy Code Generation Performance comparison. This article became very long; I just wanted to post a few sentences about the DLR and this is what happened. The following graphs were created with Excel with some help from my good old friend ViperDK. The second graph (the first one was already shown above) shows just the execution time of the while-loop code block in each of the used language or technique.


And this graph also shows the other times like compile & build time where it makes sense or script or DLR start-up times to see how much this could affect the overall time from coding to seeing the result on the screen. This is important for my approach to my own language since I want a extremely low time for the coding to see result loop. Start-up time can be ignored in those cases, but it is still important since it is still kinda slow with the DLR (but will hopefully get better in the future).



Here are the values used to generate those graphs. Tested on my almost 3 years old 2.4 Ghz 6600 CPU; this is probably way faster on my i7@3.6 Ghz at work:

Language or Technique	Code lines	Compile&Build	Script Startup	Parsing&AST	Execution
C# 3.0	10	~500ms	-	-	~37ms
Native C++	8	1500ms	-	-	~31ms
Assembler	15+	~1100ms	-	-	~15ms
IronPython 2.6	6	-	~300ms DLR	160ms	~9000ms (~1250ms for files)
ToyScript	8	-	~430ms DLR	103ms	~8400ms
Python 2.6	6	-	~50ms	~100ms	~3500ms
Python 3.1	6	-	~50ms	~100ms	~4500ms
Lua 5.1.4	7	-	~25ms	~150ms	~1400ms
Script.NET	8	-	~100ms	-	~39752ms
IL Emit	30+	-	~100ms	~1ms	~16ms
Irony + DLR (dummy)	30 + 6	-	265ms Irony	87ms	~138ms (dummy)
DLR AST with objects	68+	-	433ms DLR	4ms	~459ms
DLR AST with doubles	60+	-	434ms DLR	5ms	~98ms

Implementing and testing the following code was not too bad for me since I was learning how to generate AST statements in the DLR, generating System.Reflection.Emit IL code, doing stuff with the ToyScript DLR sample, etc. anyway. I was of course interested in how to get the best performance for my code generation, but it is useful to understand why each approach is slower or faster. Sorry if the comparison presents your project or technology you might use in a bad light, but for example Script.NET is slow in comparison (this is explained below in more detail), but still fast enough for probably 99% of its uses. It still took much longer than I expected, writing this article did take some time too, but most of the time was spend writing and testing code in each language and technique and then gathering the execution time results. But performance evaluation is always fun :) Again: This is not a representative performance comparison of the languages and techniques involved, it only applies to my very short and stupid test code and should only give you a general idea.

C# 3.0

First of all the very simple implementation of the while loop:

public static double DoLoop(int loops)
{
    double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0;
    }
    return counter;
} // DoLoop(loops)

And now some code to measure the execution time and to print out the result to see if the loop was executed as many times as we expected:

    long startTime = WindowsHelper.GetPerformanceCounter();
    
    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    // Note: We did printing in other tests too (does not matter anyway)!
    Console.WriteLine("DoLoop result=" + result);
    
    long endTime = WindowsHelper.GetPerformanceCounter();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + WindowsHelper.ConvertToTime(endTime - startTime));

Note that the WindowsHelper class is just providing a little bit more accurate information than just using a StopWatch (which is used in most other samples here), which is fine too. With System.Diagnostics.StopWatch this code looks like the following (actually results in the same results on Windows 7 and Vista):

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    //Note: We did printing in other tests too (does not matter much anyway)!
    Console.WriteLine("DoLoop result=" + result);

    timer.Stop();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + timer.ElapsedMilliseconds + "ms");

The following output is generated:

DoLoop result=10000000
Executing the while loop 10000000 times took: 37ms

Native C++

Here is the whole program used for testing. This has nothing to do with the DLR or .NET anymore, I just wanted to know how my little while loop would perform in native C++ and Assembler. Sometimes when using for loops the C++ compiler will optimize the code away and just replace it with something static that results in the same value as if the for loop would have been executed. Please also note that it did not make any difference to use a 64bit platform and doubles or just 32bit floats, but since most script languages and especially .NET compile to x64 too, we should not use 64bit numbers when compiling a 32bit application here in C++. Switching to x64 is also not too hard, you just need the correct 64bit libraries and set the linker target platform to X64.

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    // C++ version
    int loops = 10000000;
    float counter = 0.0f;
    //64bit code: double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0f;
    }
    
    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    // Wait for user input (not required for Debug.StartWithoutDebugging)
    //char input[128];
    //scanf(input);

    return 0;
}

And after compiling (usually a lot faster than 1s when just incrementally compiling) and running this code, it generates this output:

counter=10000000.000000
This took 31ms

The result is not very accurate, I either get 31ms or 47ms, but not values in between, the accuracy of GetTickCount pretty much sucks and I should use similar code as the GetPerformanceCounter() stuff above from C#, but I do not really care about more accurate values here, C++ is more than fast enough and we do not need to discuss its runtime performance!

Assembler

Based on the C++ solution I was able to write some Assembler instructions. It has been a very long time since I used assembler code at all, probably in 2002 when optimizing some path finding algorithms for ArenaWars, which were called from C++ code, which was called from C# code. Before that in the C++ days I used it now and then, but never for more than some performance critical inner loops. Please also note that sometimes you make non-optimal assembler code (as I probably did in this example) and just having your C++ compiler generate assembler instructions can be better, so always profile your code and only optimize to Assembler if you have no other option left and if you are too lazy to rethink the problem :D

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();
    
    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
startLoop:
        // counter += 1.0f
        fld counter
        fadd counterInc
        fstp counter

        // loops--
        dec loops

        // loops > 0? 
        cmp loops, 0
        // Then goto startLoop (else we are done)
        jg startLoop
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

I don't have much to say about this ugly assembler code, as you can see it is a lot more lines of code and this was the first idea not even using registers as you should in assembler. After running this code and getting 31-47ms too (like the C++ version), I added some minor optimization by using registers more explicitly and this is the resulting code, it ran about 2-3 faster than the above code:

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
        // Copy loop counter to eax register (32bit)
        mov eax, loops
        // Load both float values
        fld counterInc
        fld counter
        
startLoop:
        // counter += 1.0f
        fadd st(0), st(1)
        //st(0) has still the counter value! no need to store: fst st(0)

        // loops--
        dec eax

        // loops > 0? 
        cmp eax, 0
        // Then goto startLoop (else we are done)
        jg startLoop

        // We are done, copy counter value back!
        fstp counter
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

This program does produce the following output:

counter=10000000.000000
This took 15ms

This is pretty good and I did not want to optimize it further, especially not when the timer is so inaccurate. Running the code 10 times resulted in 140ms. Again, fast enough, no reason to argue the superior runtime performance. But this comes at a high price since the code is in no way easy to understand, its hard to write and read.

IronPython 2.6

Speaking of code, I would say the implementation in Python is the shortest and very beautiful since it is so easy to read and write:

loops = 10000000
counter = 0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

I just used my ScriptManager from my .NET libraries, which can handle IronPython (and other DLR languages) as well as Lua and wrote this short unit test to see how IronPython performs:

    public void TestPythonDoLoop()
    {
        // Create script manager without any output redirection or special directories
        ScriptManager manager = new ScriptManager(null, null);
        
        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Simply execute line by line as we would in a console window
        manager.ExecuteCode("loops = 10000000", ScriptType.Python);
        manager.ExecuteCode("counter = 0", ScriptType.Python);
        manager.ExecuteCode(@"while loops > 0:
loops = loops - 1
counter = counter + 1", ScriptType.Python);
        manager.ExecuteCode("print counter", ScriptType.Python);

        timer.Stop();
        Console.WriteLine(
            "Executing the while loop 10000000 times in python took: " +
            timer.ElapsedMilliseconds + "ms");
    } // TestPythonDoLoop()

And this results in the following output:

Executing the while loop 10000000 times in python took: 9151ms

Update 2009-04-21: As Simon Davy correctly states in the comments of this post, ipy.exe (IronPython) is actually a lot faster when executing a file with the code above (e.g. use import time). This results in an execution time of just 1250ms, so a lot faster! I have tested it too with ipy, but I entered the code line by line (same way as I called my ScriptManager code in the unit test above) and it still takes 9 seconds. But Simon is right, once I execute everything at once (with ipy test.py) it only takes 1.25s (can also be done with my ScriptManager, but I have not thought of that). So it is probably just an issue with REPL (entering statements one by one .. less optimizations that way). While this certainly does make IronPython a much faster dynamic language than all the other ones, it still is slower than the compiled languages or the DLR code at the end of this article.

Thanks for noticing!This is certainly not good. The same thing happend after I tried using the IronPython 2.6 console or IronPython 2.0, it always took about 9 seconds to complete the big while loop on my PC. I wanted to figure out why this is so slow compared to C#, C++ or Assembler runtime, but then again we have to remember that Python is a dynamic language. The most important reason for the long runtime here is probably just all the boxing and unboxing of values, if you can say that for dynamic languages. What I mean is figuring out which type we got, and then performing the add/sub operation, which can be slow if you do it 10 million times. Please note again that this is probably not affecting any real Python program and performance for more complex programs will be way better. More on the type-performance and this issue in general at the end of this article!

ToyScript

To learn more about the DLR I played around with the DLR ToyScript sample quite a bit the last days. Since a lot of its code is based on the same ideas as IronPython and IronRuby, you can't expect much different performance results. This is the ToyScript code for the while loop:

loops = 10000000
counter = 0
while (loops > 0)
{
    loops = loops - 1
    counter = counter + 1
}
print counter

Similar to the ScriptManager above I wrote some helper classes for the ToyScript language allowing me just to execute code, viewing the AST (abstract syntax tree) and write a lot of unit tests. The following code is such a unit test for the ToyScript class to check the performance for these 10 million while loop iterations:

    public void TestBigLoop()
    {
        const int NumberOfLoops = 10000000;

        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Initialize the language first (takes its own time, around 400ms)
        ToyLanguage language = new ToyLanguage();
        // Execute something else (takes maybe 100ms)
        Assert.Equal(3.0, language.Execute("print 1+2"));

        timer.Stop();
        Console.WriteLine("Startup time: " + timer.ElapsedMilliseconds + "ms");
        timer.Reset();
        timer.Start();

        Assert.Equal((double)NumberOfLoops,
            language.Execute(
@"loops = " + NumberOfLoops + @"
counter = 0
while (loops > 0)
{
loops = loops - 1
counter = counter + 1
}
print counter
"));

        timer.Stop();
        // Show total execution time
        Console.WriteLine("Executing the while loop " + NumberOfLoops +
            " times took: " + timer.ElapsedMilliseconds+"ms");
    } // TestBigLoop()

This code runs for about the same time as IronPython, maybe a little faster since ToyScript has a lot less features than IronPython, but in general it uses the same approach. This is the output from the unit test:

Startup time: 650ms
Executing the while loop 10000000 times took: 8914ms

Investigating the Lambda Expression AST for the code block shows that quite a lot of statements were created in order to make this ToyScript code run on the DLR:

 // Expression: Expression`1
 
 .lambda (<toyblock>$2 Microsoft.Func`1[System.Object] #1)
 
 // LambdaExpression: <toyblock>$2(1)
 .lambda System.Object <toyblock>$2 ()(
 ) {
   .label 0x03d11d12:
   .comma  {
     .block  {
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)10000000)
       ,
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)0)
       ,
       .loop break:.label 0x03fe2d21  {
         .block  {
           .if (
             .site (Boolean) CallSiteBinder(ConvertTo to System.Boolean) (
               .extension CodeContextExpression ( )
               ,
               .site (Object) CallSiteBinder(DoOperation GreaterThan) (
                 .extension CodeContextExpression ( )
                 ,
                 .extension GlobalVariableExpression ( )
                 ,
                 0
               )
             )
           ) {
             /*empty*/
           } .else {
             .block  {
               /*empty*/,
               .break .label 0x03fe2d21
             }
           }
           ,
           .block  {
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             .site (Object) CallSiteBinder(DoOperation Subtract) (
               .extension CodeContextExpression ( )
               ,
               .extension GlobalVariableExpression ( )
               ,
               1
             ))
             ,
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             'ToyHelpers.Add'(
               (Object).extension GlobalVariableExpression ( )
               ,
               (Object)1
             ))
             ,
             /*empty*/
           },
           /*empty*/,
           /*empty*/
         }
       },
       'ToyHelpers.Print'(
         .extension CodeContextExpression ( )
         ,
         .extension GlobalVariableExpression ( )
       ),
       /*empty*/
     },
     .null
   }
 }

The IL code was even more confusing so I only tried to figure out the simple "print 1+2" expression IL code that can be produced by calling ScriptCode.SaveToAssembly(..), more DLR IL code analysis will happen at the end of this article:

  .method public specialname static object <toyblock>$1$1(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope $scope,
 class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext $language) cil managed
  {
    .maxstack 3
    .locals init (
      [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context)
    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
    L_0007: stloc.0 
    L_0008: ldloc.0 
    L_0009: ldc.r8 1
    L_0012: box float64
    L_0017: ldc.r8 2
    L_0020: box float64
    L_0025: call object ToyScript.ToyHelpers::Add(object, object)
    L_002a: call void ToyScript.ToyHelpers::Print(class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext, object)
    L_002f: ldnull 
    L_0030: ret 
  }

There is a lot of extra stuff in there, but you can clearly see the 1+2 in L_0009 to L_00020 with all its boxing fun. Then those two are added with help of a method Add that is defined in the ToyHelpers class (unboxing the doubles again, adding and boxing them to an object again, which is returned). Then the result is passed along to the ToyHelpers.Print method, which just calls Console.WriteLine in my case (usually it is used by the ConsoleHost class for the normal REPL console interaction).

I am not a DLR expert and there is probably a lot of other things going on, but to me all this boxing, unboxing and extra code just to add some numbers is clearly too much for the big loop. At the end of this article I will try to write some DLR code that does less boxing/unboxing.

Python 2.6

After testing the DLR it was time to check out the native Python implementations to see if this is releated to the dynamic nature of dynamic languages like Python or Lua. Well it obviously is related, but more specifically I wanted to see if certain implementaions are maybe more clever or optimized.

The Python code is the same as above for IronPython:

loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

Executing this takes about 3.5 seconds, we could cheat a little and use the following code to execute this loop a little faster:

counter = 0.0
for loops in range(0, 10000000):
    counter = counter + 1.0
print counter

This runs about 2 times faster (less than 2 seconds) because Python is clever and optimizes for loops. The same actually applies to IronPython, a for loop is much faster here too (5.5s instead of 9.1s). But this defeats the whole point of this exercise, I wanted to know what is slow about the DLR and implementations like IronPython or ToyScript and why. It is nice that Python executes for loops quicker, but the whole counter = counter + 1 thing is still very slow compared to C#, C++, etc.

Python 3.1

Executing Python 3.1 is pretty much the same as Python 2.6. It is slighly slower, but that can also be because I use a alpha build (3.1 alpha 2 from 2009-04-04). Again, executing the for-loop version with range was twice as fast (2.1s instead of 4.3s). In general is is quite known that Python is not one of the fastest languages, there are many dynamic languages that can be faster (for example Ruby or Lua), but there are also much slower ones like PHP. If you are more interested about comparing IronPython with Python read the performance comparisons at the IronPython project at codeplex: http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Performance

Lua 5.1.4

The lua code is pretty similar to Python, but it executes much faster:

loops = 10000000
counter = 0.0
while (loops > 0) do
    loops = loops-1
    counter = counter+1
end
print(counter)

This executes in less than 1.5 seconds and returns the expected 10000000 (not 10000000.0 as Python however since Lua treats everything as a 64bit double anyway). The for-loop trick works here too and reduces the time in half (0.7s instead of 1.4s):

counter = 0.0
for i=1,10000000 do
    counter = counter+1.0
end
print(counter)

The reason why Lua is much faster than all the other dynamic languages I have tested is probably because Lua has much simpler data types and is optimized a lot for math, for loops and stuff like that. I use Lua more than Python these days and the performance is quite good I have to say (using it for AI, some physics and some game logic). As you can see here in many benchmarks Lua is several times faster than Python (but Python is still a lot more powerful with the huge amount of libraries available for it).

Script.NET

Script.NET is a great project that uses the Irony compiler toolkit to parse and evaluate simple C# code. It does not generate any IL or immediate code, it just evaluates the AST directly. This is a very nice approach and easy to understand because all the code is C#, but there is a slight problem with it: This approach is it VERY slow. It was optimized many times already and runs much faster than a year ago, but I would not recommend it for anything other than some scripting in C# (guess what, that's the whole name of the project). If you do not need high performance Script.NET is just fine for executing a couple lines of code, even for games as long as you do not do any physics or rendering code with it ^^

loops = 10000000;
counter = 0;
while (loops > 0)
{
    loops = loops-1;
    counter = counter+1;
}
Console.WriteLine(counter);

The output appears after almost 40 seconds and displays correctly:

10000000

At least I now know why generating some IL code is very important, with or without the DLR. Next I will investigate IL code generation with Reflection.Emit, but I can still remember the pain from the last time I tried to do something with it.

IL Emit

So let's get right to it, the following code will generate the required IL for us:

    // Create a dynamic method called DoLoop where we put all our IL code
    DynamicMethod DoLoop = new DynamicMethod("DoLoop",
        typeof(double), null, typeof(string).Module);

    // Start a timer once again to measure creation time and execution time!
    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Get an ILGenerator and emit a body for our dynamic method
    ILGenerator il = DoLoop.GetILGenerator();
    
    // Create counter (double)
    LocalBuilder counter = il.DeclareLocal(typeof(double));
    il.Emit(OpCodes.Ldc_R8, 0.0);
    il.Emit(OpCodes.Stloc, counter);
    // Assign loops variable to 10000000 (10 million) for the loop!
    LocalBuilder loops = il.DeclareLocal(typeof(int));
    il.Emit(OpCodes.Ldc_I4, 10000000);
    il.Emit(OpCodes.Stloc, loops);
    
    // Start loop label
    Label startLabel = il.DefineLabel();
    il.MarkLabel(startLabel);
    
    // Do this 10000000 times:    
    // Increase counter by 1.0
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ldc_R8, 1.0);
    il.Emit(OpCodes.Add);
    il.Emit(OpCodes.Stloc, counter);

    // Decrease loops by 1
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 1);
    il.Emit(OpCodes.Sub);
    il.Emit(OpCodes.Stloc, loops);

    // Check if we still have to loop (as long as loops > 0)
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 0);
    il.Emit(OpCodes.Bgt, startLabel);

    // End loop and return counter!
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ret);

    // Create a delegate that represents the dynamic method. This
    // action completes the method, and any further attempts to
    // change the method will cause an exception.
    DoLoopDelegate loop = (DoLoopDelegate)
        DoLoop.CreateDelegate(typeof(DoLoopDelegate));

    timer.Stop();
    Console.WriteLine("DoLoop IL creation time: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();

This creates the dynamic DoLoop method, which we can call with loop() from here on, which will also return the result (counter) once we call it. This all takes almost no time at all:

DoLoop IL creation time: 0ms

Next, we need to invoke this dynamic method, which is rather easy:

    timer.Start();
    Console.WriteLine("Invoking DoLoop() returned: " + loop());
    timer.Stop();
    Console.WriteLine("DoLoop IL execution time: " + timer.ElapsedMilliseconds + "ms");

And this only takes about 16ms and also returns the correct value:

Invoking DoLoop() returned: 10000000
DoLoop IL execution time: 16ms

So IL code is rather fast (faster than everything else here except the fine-tuned assembler, which is similar to this), but don't ask in how many problems I ran into just writing these few lines of code. Let's just say a lot of times I used wrong opcodes, did not know how to implement stuff and finally when stuff compiled I got syntax errors when executing, not really telling me what I did wrong, so I had to comment out code until it worked again and then slowly added code until I could figure out what had to be changed. Let's just say not a pleasant thing to write a compiler this way.

Irony + DLR (dummy)

As mentioned before Script.NET actually uses the Irony compiler toolkit for parsing and building the AST (abstract syntax tree). Irony itself is rather fast (can parse 15000 lines of code or more per second), only the execution part of Script.NET wasn't so good because no code is generated, everything is evaluated on the fly.

While I have started using Irony just a week ago after playing around with ANTLR first (see earlier blog post here), but I still like it very much and I hope Roman Ivantsov (the guy behind Irony) will soon release a long promised update with some cool new features I can sink my teeth into. As you will see in my sample below I have added some own functionality to Irony already to make it easier to test code and I will probably extend it even more when I write more of my language grammar.

If you don't know how Irony works and you are interested in building compiler grammars directly in C# you should check out the project and the Lang.NET talks Roman did this and last year about it, plus the great Irony article on CodeProject of course. Basically you define some non-terminals (variables, numbers) and terminals (logical blocks of your code) in the constructor of a Irony.Grammar derived class and then define some rules for it (in BNF = Backus-Naur Form). Then you set the root node and you are ready to parse some code. Please note that this grammar is very simple and will only handle this sample code and very similar code blocks. I'm still just playing around with grammars, there is no concrete implementation of my language yet, just a lot of ideas .. Writing some grammars is a good training, I have written one in ANTLR and 4 other more complex ones with Irony yet.

    public TestDoLoopGrammar()
    {
        // 1. Terminals
        var variable = new IdentifierTerminal("variable");
        var number = new NumberLiteral("number");

        // 2. Non-terminals
        var program = new NonTerminal("program");
        //var commandList = new NonTerminal("commandList");
        var command = new NonTerminal("command");
        var assignmentOperator = new NonTerminal("assignmentOperator");
        var assignment = new NonTerminal("assignment");
        var whileLoop = new NonTerminal("whileLoop");
        var compareOperator = new NonTerminal("compareOperator");
        var condition = new NonTerminal("condition");
        var binaryOperator = new NonTerminal("binaryOperator");
        var operation = new NonTerminal("operation");

        // 3. BNF rules
        // Please note that this program grammar is rather limited and not really
        // useful, it is only used for testing and learning Irony ..
        // We can have many commands in case we want to go crazy
        program.Rule = MakePlusRule(program, null, command);
        assignmentOperator.Rule = Symbol("=");
        assignment.Rule = variable + assignmentOperator + number;
        whileLoop.Rule = Symbol("while") + condition + ":" +
            program + "end";
        // We only allow some simple compare conditions
        compareOperator.Rule = Symbol("<") | "==" | ">";
        condition.Rule = variable + compareOperator + number;
        // We can either use + or -
        binaryOperator.Rule = Symbol("+") | "-";
        operation.Rule =
            variable + assignmentOperator + variable + binaryOperator + number;
        // Commands can be assignments, while loops or operations
        command.Rule = assignment | whileLoop | operation;

        // 4. Operators precedence and punctuation
        RegisterPunctuation(":", "end");

        // 5. Global settings
        this.CaseSensitive = false;
        this.ThrowGrammarExceptionsOnError = true;

        // 6. Set grammar root
        this.Root = program;
    } // TestDoLoopGrammar()

Note that ThrowGrammarExceptionsOnError is not a property of Irony, I added it to make sure all Grammar errors immediatly cause an exception to be thrown, which makes it much easier for unit tests and figuring out what when wrong. Not really sure why Irony does not do that, I can understand not throwing exceptions for syntax errors when parsing some source code (because that might be slower than just reporting errors in a list form and more helpful because its easier to continue after the error). But grammar errors mean usually that you cannot use the grammar for anything anyway. Some things also just produce a warning, but I throw Exceptions then too because I don't want hidden fixes of my grammar, which would make it hard to fix real problems later.

Anyway, with the TestDoLoopGrammar class we can now test and parse some code. Irony usually expects you to use its Grammar Explorer program to do this job, which is a pretty nice application, but it is not unit test and we all know I'm all about unit testing (don't we?). I still use the Irony Grammar Explorer from time to time and I recommend it very much if you want to test out grammars and visualize the AST a little better than just throwing some text into a console window. The following simple unit test will parse our while-loop program source code and produce a nice AST for us, which it kindly prints out:

    public void GenerateDoLoopAST()
    {
        AstNode program = IronyHelpers.ParseWithErrorHandling(
            new TestDoLoopGrammar(),
@"loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
end");

        IronyHelpers.PrintAst(program);
    }

This will quickly print out the AST in the following way, which might seem a little complex and verbose at first, but you have to remember that we do not have to generate code for every single node. Often several nodes together will just be one single instruction. Other nodes like program are just used to hold code blocks together and allow several child nodes.

 program
   assignment
     loops [variable]
     assignmentOperator
       = [Symbol]
     10000000 [number]
   assignment
     counter [variable]
     assignmentOperator
       = [Symbol]
     0 [number]
   whileLoop
     while [variable]
     condition
       loops [variable]
       compareOperator
         > [Symbol]
       0 [number]
     program
       operation
         loops [variable]
         assignmentOperator
           = [Symbol]
         loops [variable]
         binaryOperator
           - [Symbol]
         1 [number]
       operation
         counter [variable]
         assignmentOperator
           = [Symbol]
         counter [variable]
         binaryOperator
           + [Symbol]
         1 [number]
     end [variable]

Now this tree has to be translated to some code. We could use Reflection.Emit for this, but we just saw how much work it was for this very simple sample. Thats the reason I'm using the DLR instead, which also gives my language some cool dynamic capabilities for free (plus the other benefits like interacting with other DLR languages). But since the DLR is not a piece of cake and I'm not done exploring it yet (just started last two week with it, gimme more time), I will not present a full project with this language here yet (totally out of the scope of this article). I used a modified version of ToyScript and some other sample projects (following next) to generate some DLR code here, but it is still hacked up very much. I will probably write another long article when I'm done with that and have some cool results to show.

DLR AST with objects

Okay, so let's dig deeper into the DLR. While I got some parts of the above Irony AST working in the DLR with help of the ToyScript sample, the whole DLR framework, the ToyScript sample and especially IronPython is still very hard to understand and its also kinda annoying that recent code drops do not work at all with ToyScript. They compile, but running ToyScript will just produce funny errors like "node not reducible", which is not very helpful if you just want to learn about the DLR. A lot of stuff in the DLR still constantly changes (just see the huge amount of code drops at dlr.codeplex.com). I totally like that the DLR team releases all this wonderful code and does it so often, but without any working documentation (almost everything you find on the internet is based on DLR versions from a year ago, when stuff worked in a totally different way) and not even the included samples working all the time is kinda painful. So what does a programmer do in such a case? Yes, he writes tests and figures it out on his own. If there were not this many classes that all do almost nothing in functionality (what the hell, in my projects each class has many hundred lines of code, but in the DLR it seems like 50% of all classes are just 10-30 lines long) it would a much easier job. Debugging is also much harder because stack-traces become 2 miles long after just calling one of two actual functions, that do not just call other functions. Okay, enough ranting, let's just hope these things get better in the final 1.0 release since Microsoft is not allowing any contributions from the community to the DLR and I certainly do not want to write my own. It is pretty amazing what the DLR can do and how well it performs, it just has to become easier to use. Someone had to say it :)

So the bare minimum you need for the DLR to compile some AST for you is a LanguageContext and a DefaultBinder, define one class of each and derive it accordingly. Then define a constructor for your language context, set the binder and while you are at it add the default System (mscorlib) assembly to the script domain manager. Note this code uses DLR change set 21259 (from Apr 5 2009), but I could also use the more recent 22459 version (from today), but I reverted to make the ToyScript sample work again.

    public YourLanguageContext(ScriptDomainManager manager,
        IDictionary<string, object> options)
        : base(manager)
    {
        Binder = new YourBinder(manager);
        manager.LoadAssembly(typeof(string).Assembly);
    } // YourLanguageContext(manager, options)

Next you need to provide an implementation of ParseSourceCode to make the compiler happy and to actually allow your program to parse some source code in your language:

    protected override ScriptCode CompileSourceCode(SourceUnit sourceUnit,
        CompilerOptions options, ErrorSink errorSink)
    {
        // Create lambda expression
        LambdaBuilder program = AstUtils.Lambda(typeof(object), "YourLanguage");
        program.Body = Expression.Block(GenerateStatements(program, sourceUnit.GetCode()));
        LambdaExpression ast = program.MakeLambda();
    
        Expression<DlrMainCallTarget> globalAst =
            new GlobalLookupRewriter().RewriteLambda(ast);
    
        return new LegacyScriptCode(globalAst, sourceUnit);
} // ParseSourceCode(context)

The interesting thing here is how we build an DLR AST from statements (Expression.Block) with help of the LambdaBuilder. The statements do not exist yet, that's what the GenerateStatements method will be used for. We pass the source code to this method and let its do its magic (this is a reduced version, the actual unit test uses a more complex version with some extra code paths and statements to make sure we can initialize the DLR first and to generate code for the next technique too):

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(object), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Convert(
                    Expression.Constant(0.0), typeof(object))
            )
        );
        Expression loops = program.Variable(typeof(object), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Convert(
                    Expression.Constant((int)10000000), typeof(object))
            )
        );
        
        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    Expression.Convert(loops, typeof(int)),
                    Expression.Constant(0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Convert(
                            Expression.Add(
                                Expression.Convert(counter, typeof(double)),
                                Expression.Constant(1.0)
                            ), typeof(object))),
                    Expression.Assign(loops,
                        Expression.Convert(
                            Expression.Subtract(
                                Expression.Convert(loops, typeof(int)),
                                AstUtils.Constant(1)
                            ), typeof(object)))
                ),
                AstUtils.Empty()
            )
        );
        
        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(object) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

This is obviously not dynamic code, it just adds a bunch of expression statements to a list and returns it. Please also note that I tried to use object for counter and loops to demonstrate this in a similar way ToyScript or IronPython will generate code (thats how I figured out some of these expressions). But since .NET or the DLR does not know how to add two objects for example and does not even allow to assign a number to an object, we have to do a lot of casting, which is in essense just boxing and unboxing.

This code is then executed by a unit test, which does basically create a ScriptRuntime for us, makes sure it uses this language implementation and then calls engine.CreateScriptSourceFromString with the code to be executed. Please note that GenerateStatements completely ignores the source code, it will always generate the same statements right now.

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Create runtime
    ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
    Type provider = typeof(YourNamespace.YourLanguageContext);
    setup.LanguageSetups.Add(
        new LanguageSetup(provider.AssemblyQualifiedName, provider.Name));
    ScriptRuntime runtime = new ScriptRuntime(setup);

    // Load Engine
    ScriptEngine engine =
        runtime.GetEngineByTypeName(provider.AssemblyQualifiedName);
    timer.Stop();
    Console.WriteLine("Create DLR runtime: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();
    timer.Start();

    // Execute command
    ScriptSource code = engine.CreateScriptSourceFromString("Objects");
    code.Execute();

    timer.Stop();
    Console.WriteLine("Execute code: " + timer.ElapsedMilliseconds + "ms");

This test will spend about 430ms in the DLR runtime creation (which is kinda slow and will hopefully be faster in the future) and that is the reason it is checked separately. Then it will spend a few ms for the statement creation (4ms as mentioned in the table above) and finally it will execute the code in about 459ms, which is a lot faster than the other DLR languages. But then again, we cheated quite a bit by providing the exact statements we need. The only thing we can optimize is to get rid of all those boxing and unboxing actions. While it is nice to be under half a second for these 10 million while loop iterations with one add to counter and one subtract from loops statement each, it is still a far cry from the performance we have seen in the compiled languages. Let's check out the IL code generated from the DLR AST to see whats going on here:

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] object obj2,
    [2] object obj3)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: box float64
  L_0016: stloc.1 
  L_0017: ldc.i4 0x989680
  L_001c: box int32
  L_0021: stloc.2 
  L_0022: ldloc.2 
  L_0023: unbox.any int32
  L_0028: ldc.i4.0 
  L_0029: cgt 
  L_002b: brfalse L_0035
  L_0030: br L_003a
  L_0035: br L_0063
  L_003a: ldloc.1 
  L_003b: unbox.any float64
  L_0040: ldc.r8 1
  L_0049: add 
  L_004a: box float64
  L_004f: stloc.1 
  L_0050: ldloc.2 
  L_0051: unbox.any int32
  L_0056: ldc.i4.1 
  L_0057: sub 
  L_0058: box int32
  L_005d: stloc.2 
  L_005e: br L_0022
  L_0063: ldloc.1 
  L_0064: call void [mscorlib]System.Console::WriteLine(object)
  L_0069: ldnull 
  L_006a: ret 
}

It might be a little hard to figure out which line does what, but once you found the constants like 0x989680 (which is just 10000000) and the add and sub operations, you can get an idea how the statements were translated. The code between L_0035 and L_0063 is the inner loop and the performance critical code. There are 2 unboxing and 2 boxing operations in that code and probably some other unnecessary assignments.

DLR AST with doubles

While playing around with the code even more and getting to a version like the one that will be shown in this section, I tried to understand why the DLR did certain things in the way that it does. A lot of dynamic behavior is only possible when you use very generic types, but I still think for my language implementation it should be possible to use a much more strict rule set and only allow dynamic code where necessary or wanted, not in thinks like loops or math, where we deal with numbers anyway. Lua is a great example of that, numbers are just numbers. They are always doubles, so there is no need to box or unbox them all the time, which is probably one of the reasons it can handle math better than other dynamic languages.

I also found this useful explanation of what IL the DLR emits for dynamic code, it could explain why stuff is slower than IL code, but still a lot faster than evaluating everything ever time. The DLR does a great job of caching and optimizing dynamic code, but as you can see the IL generated is quite long and verbose too, which is ok for dynamic code I guess. I'm just interested in ways to optimize this further to at least provide proof that the DLR is in fact just as good as a compiler for more static languages.

After the last section it is quite obvious what we have to do do generate statements that can be executed quicker: Get rid of all that boxing and unboxing, use specific data types and make sure all the assignments use the correct format too (else convert constants or other variables). This should not be too hard to do in an actual language once we figure out that we are only dealing with numbers (similar to Lua we could only allow one kind of number, like doubles). This is the updated code for GenerateStatements:

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(double), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Constant(0.0)
            )
        );
        Expression loops = program.Variable(typeof(double), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Constant(10000000.0)
            )
        );

        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    loops,
                    Expression.Constant(0.0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Add(counter,
                            Expression.Constant(1.0))),
                    Expression.Assign(loops,
                        Expression.Subtract(loops,
                            AstUtils.Constant(1.0)))
                ),
                AstUtils.Empty()
            )
        );

        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(double) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

BTW: To analyze the IL code I use the ScriptCode.SaveToAssembly method and use Reflector to show the IL code. As you can see the code has become shorter and we can't see any signs of boxing and unboxing anymore. The code is not even optimal because we use doubles for everything (remember that the C#, C++ and Assembler samples all used a simple integer for the while loop), but as the execution time with less than 100ms shows us, the performance is quite good with this approach.

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] float64 num,
    [2] float64 num2)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: stloc.1 
  L_0012: ldc.r8 10000000
  L_001b: stloc.2 
  L_001c: ldloc.2 
  L_001d: ldc.r8 0
  L_0026: cgt 
  L_0028: brfalse L_0032
  L_002d: br L_0037
  L_0032: br L_0054
  L_0037: ldloc.1 
  L_0038: ldc.r8 1
  L_0041: add 
  L_0042: stloc.1 
  L_0043: ldloc.2 
  L_0044: ldc.r8 1
  L_004d: sub 
  L_004e: stloc.2 
  L_004f: br L_001c
  L_0054: ldloc.1 
  L_0055: call void [mscorlib]System.Console::WriteLine(float64)
  L_005a: ldnull 
  L_005b: ret 
}

This is a more complex version of the above unit test from last section, the 10000000 is printed by the DoLoop DLR code:

runtime creation: 336ms
get engine: 16ms
create script: 4ms
Init code and first script startup time: 232ms
10000000
execute again (DoLoop code): 98ms
and shutdown: 0ms

Okay, I would say this blog post is long enough now and I wrote all day on it and the code samples presented here. Could even be the longest blog post on my blog (html is 150kb). Who knows .. it is probably a good idea to go to bed now.

Good fight, good night, till next time :)

Intel just released a Drive Firmware Update Tool for its Intel X18-M and X25-M Solid State Drives (SSDs). You can read more about it here (info) and here (pdf) or just download the firmware update.

I have been using my Intel SSD for less than a month now and I did not really notice any major slowdowns, but thanks to this firmware I'm confident the drive will stay fast and very usable for our big projects. At home I'm pretty happy with my old crappy OCZ Core SSD (crappy because the J-Micron controller just sucks and freezes the whole PC after 50+ open files) because Windows 7 handles it quite nicely and my projects are usually small enough. The only annoyance is with ASP.NET web projects because the IIS is quite slow and because of other issues like the slow HttpWebRequest.

Maybe in 2 Months when the new i7 950 CPUs (replacement of i7 940) come out I will think about some new PC stuff for my almost 3 year old Dual-Core PC at home, but for now I'm pretty happy. The only real improvement besides the CPU I could make is probably more ram (which is pretty cheap anyway these days) and a much faster SSD like the Intel X25-M, which I use at work, but thats a little expensive .. maybe Intel will think about reducing the prices of their SSDs after many competitors (especially OCZ) are catching up and getting better and better.

I was going nuts yesterday night after I wrote a couple of unit tests that involved the C# HttpWebRequest class. For some strange reason it always takes 20-30 seconds to create a HttpWebRequest with an url and calling the GetResponse method on Windows 7. I was thinking maybe the server I'm contacting is kinda slow, but it was really fast in the browser. So maybe my SSD drive is messed up? Ok, tried on a normal HDD, same thing. Okay, then maybe TestDriven.net is doing something strange, lets write a simple console application. Wtf, still the same issue! Then I tried it on a different Windows 7 PC, same issue.

I gave up (was very late anyway) and went to work to test it again in the morning on my work PC (also Windows 7). Still the same freaking issue, it takes forever to get anything connected with HttpWebRequest and HttpWebResponse. Getting the actual data is like 30 times faster (and should not matter anyway), still slower than I remembered, but the issue was just in the following two lines (first line took 2-5s, second line 20-25s):

    HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
    HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();

Maybe this isn't the right comic, but its always good to make fun of not-perfectly-working operating systems:




Back to the Problem: I rewrote my console test application again (from last night) and compiled it on my work PC:

using System;
using System.Net;

namespace TestHttpRequest
{
    class Program
    {
        static void Main(string[] args)
        {
            //WTF is going on here? Takes 30s on Windows 7, <100ms on Windows Vista
            string url = "http://www.google.com";

            Console.WriteLine("Connecting: " + url);
            HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
            HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();

            Console.WriteLine("Connected ..");
            if (httpResponse.StatusCode == HttpStatusCode.OK)
            {
                // Dummy
                Console.WriteLine("Getting data ..");
                httpResponse.GetResponseStream();
            } // if
            else
                Console.WriteLine("Failed to get any data ..");

            Console.WriteLine("Done ..");
        }
    }
}

Guess what, still the same issue, takes around 30 seconds to do this very simple job in Windows 7. Could not really figure out the real cause of this (I'm not really that interested, I just wanted to write a couple of unit tests to make some things easier), but it seems like the Conhost.exe file (Console Window Host) of Windows 7 is doing a lot while this is going on (I even have different Windows 7 versions installed, it is the same issue on all of them). Could not see any networking going on (maybe 0.01 kb every 5 seconds) either. If I completely disable networking the GetResponse immediately crashes (it does not take 2-3s for the WebRequest.Create to do its work, and the exception occurs instantly), but that is not really helpful because I wanted to test the response ..

Instead of guessing what this could be about I just wanted to make sure this is a problem with my Windows 7, so I gave my test console application to my colleague that is running Windows Vista and the thing ran in about 20ms, there is nothing wrong with the code, it runs perfectly fine on a non-Windows 7 machine!

So I tried the same stuff in different ways, like using the WebClient class, same result, still 30 seconds for the request.

            // Same stuff with webclient
            WebClient webClient = new WebClient();
            Console.WriteLine("Downloading: " + url);
            byte[] data = webClient.DownloadData(url);
            Console.WriteLine("Done ..");
            

Next I implemented the same stuff with sockets (more specifically TcpClient, which is based on Sockets, but a little easier to use), which is kinda ugly and more work, but at least I finally got the response time I was looking for (<100ms). Microsoft should maybe look into this Windows 7 issue .. for my unit tests I can now use my own HttpWebResponse class and all problems are gone :)

            // Another try, this time with TcpClient ..
            TcpClient client = new TcpClient("www.google.com", 80);

            // Get the stream from the tcp client (allows us to read and write)
            NetworkStream stream = client.GetStream();

            // Write the http request ourselfs (Note: Needs 2 empty lines at the end)
            byte[] requestBytes = Encoding.ASCII.GetBytes(
@"GET / HTTP/1.1
Host: www.google.com
Connection: Close

");
            // Send this ound
            stream.Write(requestBytes, 0, requestBytes.Length);
            stream.Flush();

            // We can't use DataAvailable here because we don't know when the data
            // will arrive. So just call ReadByte until it returns -1 and then
            // we know we are done (Note: This will block our application,
            // if you really care about performance write it asyncronly).
            Console.WriteLine("Getting data ..");
            string result = "";
            int aByte = 0;
            do
            {
                aByte = stream.ReadByte();
                if (aByte >= 0)
                    result += (char)aByte;
            } while (aByte >= 0);
            Console.WriteLine("Result=" + result);
            client.Close();

Pretty good blog post from Jeff Atwood at CodingHorror.com:
http://www.codinghorror.com/blog/archives/001248.html

Lots of interesting comments too and a nice comic:

In accordance to the very good blog with many useful links for IronPython at ironpython-urls.blogspot.com I will also try to post some useful links about my recent DLR (Dynamic Language Runtime for .NET) research.
I have been working a bit on the DLR before, mostly together with Silverlight, which was cool, but Silverlight was way too hard to work with and I still think it is not distributed enough. On that note if you are interested in Silverlight, check out the . And even before that quite a bit with the Visual Studio SDK, early IronPython versions and other language implementations in .NET and even with native c code (but usually I just modified existing samples). I have also modified Lua for my own needs recently and made it run on the Xbox 360 and PS3 and modified some behaviour for my projects, since we use it for our upcoming game and I use it quite a lot together with IronPython on the tools I write at work.`

But my overall goal remains: Writing my own language, preferbly running on .NET and the DLR seems to be a good fit. Today after we finished the latest demo of our game for the GDC next week, I had finally some time and motivation to get this thing started again :) I probably should do some more XNA projects since I'm a DirectX/XNA MVP and I have still something I work on from time to time, but it is still a long way to go until this project is completely done.

Okay, to get things started again I searched for some new DLR information and samples on the net and here are some helpful links:

The Tools:

• DLR - Dynamic Language Runtime, which runs on top of the CLR in .NET (even in Silverlight)
  
  
• ANTLR v3 - The best tool ever for constructing your own interpreter/compiler/parser, etc.
  And these are the most useful links to get started (and other links that interested me):
  
  • Five minute introduction to ANTLR 3
    
    
  • ANTLR 3 - FAQ - Getting Started
    
    
  • Test-Driven Development with ANTLR
    
    
  • Quick Starter on Parser Grammars - No Past Experience Required
    
    
  • How do I get case insensitivity?
    
    

Learning the basics and also some good blog posts:

• Create your own language on top of the DLR - Detailed article about creating a language with the DLR
  
  
• DLR - Build Your Own Language (without tears). Kinda fast paced article that skips a lot of steps, but at least it got me motivated.
  Part 2, Part 3, Extra Tips
  
  
• CLR Inside Out - IronPython and the Dynamic Language Runtime - Good article about the DLR, IronPython and DLR ASTs
  
  
• Generate AST for the DLR (Code Project). Nicely written, easy to understand and gives a good overview of writing your own custom AST for the DLR. The sample itself is kinda useless because it always generates the same AST, but you will at least understand how code should be parsed and passed to the DLR as an AST. In this tutorial there is no parser or lexer, the generated AST is just static code, but at least the source code is easy to understand :) The article will also explain all the basics you need to understand ASTs and how the DLR uses them.
  
  
• Building a DLR Language - Answering Some Questions - Also a very helpful blog. Goes into detail about whats going on under the hoods, investigates with Reflector. Be sure to also read the other posts of Martin Maly.
  
  
• How to write a simple DLR host in C# using Hosting API
  
  
• Building a DLR Language - ToyScript (Martin Maly's blog)
  
  
• Jim Hugunin's Thinking Dynamic blog - The One True Object articles are interesting, also links to a lot of good DLR talks
  
  
• DevHawk - Blog by Harry Pierson, one of the main guys behind IronPython
  
  
• Martin Maly - Building a DLR Language (more general, see above for more comments)
  
  
• Sapphire Steel blog: Antlr 3 and the Dynamic Language Runtime
  
  
• DLR Hosting and related stuff...
  
  
• Knowing .NET blog about IronPython and .NET
  
  
• The .Net DLR makes building a new language implementation ridiculously easy
  
  
• Building On The DLR - by Tomas Restrepo - An approach to create a simple language with the DLR
  
  
• I still have a problem with the DLR - by mithrandir (Incanus On Rails) - A critice on how the DLR beta was written. I think most of the issues have been solved or not as bad as portraied there.
  
  
• Darrell Hawley's blog, mostly about IronPython integration. Has some cool "Note to self" posts on there.
  
  
• Another critice about the DLR, this time about the fact that it is not really open source since no contributions are accepted. Little bit too harsch.
  
  
• The Weekly Source Code 11 - LOLCode DLR Edition (from 2007) - by Scott Hanselman's ComputerZen.com - fun read.
  
  
• Another post like this from a year ago with helpful DLR links: Adventures in Compilers - Building on the DLR
  
  
• Embedding the Dynamic Language Runtime - IronPython & .NET Applications - A nice tutorial by Michael J. Foord (at voidspace.org.uk)
  
  

Compiler Books:

• Basics of Compiler Design by Torben Ægidius Mogensen - Free eBook - Easy and enjoyable read about a hard topic!
  
  
• Compiler Construction by Niklaus Wirth - Also a free eBook (about Oberon)
  
  
• Compiler Design and Implementation - Free eBook about some guy writing his own compiler - step by step
  
  
• Compilers: Principles, Techniques, and Tools (2nd Edition) - A heavy 1000+ pages and kinda expensive book about a heavy topic, was written in 1986, but is still very useful and recommended
  
  
• The Definitive ANTLR Reference: Building Domain-Specific Languages by Terence Parr - This is the book you should read when using ANTLR seriously. Very nice book; I plan to read it when I use ANTLR more.
  
  
• Lots more Free Compiler Design / Construction Ebooks
  
  
• And even more Free Compiler Books (newer)
  
  

Other useful DLR links:

• Homepage of IronPython - The main usecase for DLR - quite a lot of releases, performance comparisons and a small, but active community
  
  
• DLR Pad, a very simple app that lets you write and test code with a simple XAML WPF app using IronPython and IronRuby
  
  
• IronPython and the Dynamic Language Runtime
  
  
• Make Your Application Extendable Using the DLR - simple example on how to use IronRuby or IronPython in a little app.
  
  
• Boo also uses ANTLR and is a very interesting project overall (mixture of C# and Python). I used it a bit 2 years ago, but I stopped because of the lack of a decent IDE (yes, I know about SharpDevelop, but that is not decent enough for me).
  
  
• IronPython (and DLR) nightly builds. In case you want to be on the bleeding edge.
  
  
• Silverlight Dynamic Languages SDK
  
  

Okay, back to improving my lexer and parser. I think I got the research aspect covered for now.

I have been working quite a bit in Python and Lua last year and I also tried to use those languages in our current game project at exDream (will be announced shortly, so I'm finally allowed to talk about it ^^). While both Python and Lua are written in C (there are obviously other ports like the great IronPython project for .NET, but I'm not going to talk about that in this post), it was not easy at all to make Python run on the Xbox 360 or PS3. I was able in early testing to get some parts running on the Xbox 360, but too much was missed and commented out and the PS3 port looked like way too much work. So I decided to just use Lua, which is probably a better choice for our game anyway since I only use it to do some AI and physics code with it along with some settings. I'm not going to benefit from the Python libraries either way, they are not needed for our game and they won't run on those consoles without serious effort anyway.

Ok, so I was writing some Lua code and used mostly LuaInterface to make the Lua code run from .NET and of course to work with all my .NET code and even XNA. The second step was to make this run in our new game engine, which is written in native code (C++) since the whole engine is based on the Unreal 3 Engine. For that I used the normal Lua 5.1.4 release, which compiles just fine and works great. Since Lua is written completely in ANSI C I was able to quickly test some code on the Xbox 360 and PS3 too. This is important because our game will be a multi-platform title (PC, Xbox 360 and PS3), which is thanks to the Unreal 3 Engine a much simpler task than writing our own engine for all those platforms (it has obviously it own disadvantages, not only do we have to use native code, but we also have to work with a lot of code in the engine that we don't really need for our game).

So far so good. While it sounds like a simple task to make Lua run on all platforms, there is a lot of pain involved to make it run perfectly and behave the same way on all platforms. Since the Unreal 3 Engine is written in C++ and uses its own scripting Language UnrealScript, it does not help out a lot when you want to integrate your own code or libraries in there, so this was half of the work (just see my previous post about compile times and you see why this is a much lengthier process than writing some .NET code). Secondly there are many special rules on both the Xbox 360 platform and the PlayStation 3, some ANSI C methods are just missing or just don't do anything because of the platform design. For example on the Xbox 360 you can't use relative paths, you always have to specify a full path when opening a file, which is something that the Lua engine does not do out of the box. After figuring out where to store the Lua scripts on the consoles and how to load them, everything else just falls into place and works finally as expected.

To make things simpler for other people trying to run Lua code on the Xbox 360 or PS3 I wrote the following little solution:

• LuaXbox360PS3PC.zip (241.92 KB)
  

As you can see on the right the solution consists of 4 projects:

• LuaSupport, which is based on the current Lua 5.1.4 release from 22 Aug 2008 with my own additions to support the XBOX and PS3 platforms (uses XBOX and PS3 defines).
  
  
• TestLuaPC is a simple test project for the PC to make sure everything still runs as expected on the PC. Please note that on the PC Lua assumes that all files can be found in the current directory. I just change to the Lua script directory and then execute the Lua files there normally.
  
  
• TestLuaPS3 will probably only compile if you have the PS3 SDK installed, just unload this project if you don't have it installed yet. Also please test other SDK samples first to make sure everything is correctly setup (it uses a lot of extra include paths). In case you are not able to open the files, check for yourself if the /app_home/Lua/ directory contains the required files after building the project (use the ProDG Target Manager for that). Debugging can also be complicated, I guess it does not work on Windows 7 at all, but I was able to debug and run the app on a normal Windows Vista PC. In any case you can also just run the .elf file from the ProDG Target Manager and then see the output in the console.
  
  
• TestLuaXbox360 is the project to test Lua on the Xbox 360. You will also need the XDK (Xbox Development Kit SDK) installed and have a Xbox 360 DevKit in your network for testing. A TestKit also works, but you won't be able to debug, you can just run the app and check the console output yourself.
  
  

Some of the Lua functionality is not available on the Xbox 360 or PS3 like lua_popen or lua_pclose (among others), and there is also some code that will not do anything like os_execute, os_getenv and some other os methods because they won't make sense on a console anyway or could not be implemented. I did not use these methods anyway, but if you need those and want to write your own code for them, go ahead.

Lets focus on the 2 most important code changes. First of all the Lua default directories had to be adjusted in luaconf.h:83 to support Xbox 360, PS3, PC and Linux platforms:

#if defined(XBOX)
/* On the Xbox we just use the D:\ drive for everything, all lua files should be
   located there */
#define LUA_LDIR    "D:\\Lua\\"
#define LUA_CDIR    "D:\\"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(PS3)
/* On the PS3 we just use the /app_home/Lua/ sub directory for everything,
   all Lua files should be located there */
#define LUA_LDIR    "/app_home/Lua/"
#define LUA_CDIR    "/app_home/"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(_WIN32)
/*
** In Windows, any exclamation mark ('!') in the path is replaced by the
** path of the directory of the executable file of the current process.
** Note: We use the ..\Lua\ directory by default, but that is also the current
** directory, we change it in AIEngineInterface before running scripts!
*/
#define LUA_LDIR    "!\\..\\Lua\\"
#define LUA_CDIR    "!\\"
#define LUA_PATH_DEFAULT  \
        ".\\?.lua;"  LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua;" \
        LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#else
/* Default linux/mac directories */
#define LUA_ROOT    "/usr/local/"
#define LUA_LDIR    LUA_ROOT "share/lua/5.1/"
#define LUA_CDIR    LUA_ROOT "lib/lua/5.1/"
#define LUA_PATH_DEFAULT  \
        "./?.lua;"  LUA_LDIR"?.lua;"  LUA_LDIR"?/init.lua;" \
                    LUA_CDIR"?.lua;"  LUA_CDIR"?/init.lua"
#define LUA_CPATH_DEFAULT \
    "./?.so;"  LUA_CDIR"?.so;" LUA_CDIR"loadall.so"
#endif

And the most important code change is located in luaL_loadfile method, which is called by the lua_dofile macro. Please note that I did not change the code formating and tried to leave as much code as is. I have my own coding conventions, but it does not make sense to apply them when only changing <10% of the code of an existing library.

LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  char fullFilename[128];
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  lf.extraline = 0;
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    // Note: Always open as the requested file, Lua should not care about
    // our crazy directory remapping.
    lua_pushfstring(L, "@%s", filename);

#if (PS3)
    // On the PS3, check always the /app_home/Lua/ directory!
    // This works both for dev-testing via remote HDD and on the game disc.
    // These work probably too just for developing, but app_home is fine!
    // /host_root/Lua/ or /dev_bdvd/PS3_GAME/USRDIR/Lua/
    CheckForValidFilenameWithPath(fullFilename, filename, "/app_home/Lua/");
#elif XBOX
    // For the Xbox we have to make sure always to load files from D:\ because
    // fopen ALWAYS expects a full paths, there are no current directories on the
    // Xbox 360 and therefore no relative paths! Check always "D:\Lua\<file>"
    CheckForValidFilenameWithPath(fullFilename, filename, "D:\\Lua\\");
#else
    // On the PC we just use the default search logic (see luaconf.h) and we
    // don't care about directories since we will already be in the correct one!
    // In earlier versions we had a lot of extra checks here.
    strcpy(fullFilename, filename);
#endif

    // Rest of this code is untouched, we just use fullFilename now!
    lf.f = fopen(fullFilename, "r");
    if (lf.f == NULL)
      return errfile(L, "open", fnameindex);
  }
  c = getc(lf.f);
  if (c == '#') {  /* Unix exec. file? */
    lf.extraline = 1;
    while ((c = getc(lf.f)) != EOF && c != '\n') ;  /* skip first line */
    if (c == '\n') c = getc(lf.f);
  }
  if (c == LUA_SIGNATURE[0] && fullFilename) {  /* binary file? */
    lf.f = freopen(fullFilename, "rb", lf.f);  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, "reopen", fnameindex);
    /* skip eventual `#!...' */
   while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;
    lf.extraline = 0;
  }
  ungetc(c, lf.f);
  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (filename) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, "read", fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}

The code in luaL_loadfile will call the new method CheckForValidFilenameWithPath, which helps us to build a full path filename for the Xbox 360 and PS3 platforms. It will just add D:\Lua\ or /app_home/Lua/ in front of the filename if it does not exist yet. Here we can see another pain point of C code, it takes so freaking long to do very simple tasks (in .NET this would be a 2-liner).

static void CheckForValidFilenameWithPath(char* fullFilename,
  const char* filename, const char* platformPath)
{
  int i = 0;
  int pathLen = (int)strlen(platformPath);
  // Already have a filename with the correct path?
  if ((int)strlen(filename) > pathLen)
    for (i = 0; i < pathLen; i++)
      if (filename[i] != platformPath[i])
        break;

  // Found path?
  if (i == pathLen)
  {
    // Then just use the existing file, happens when loading libraries from Lua
    strcpy(fullFilename, filename);
  }
  else
  {
    // Copy path
    for (i = 0; i < pathLen; i++)
      fullFilename[i] = platformPath[i];
    // Add the relative filename
    for (i = 0; i < (int)strlen(filename); i++)
      fullFilename[i+pathLen] = filename[i];
    // And finally close the string
    fullFilename[(int)strlen(filename)+pathLen] = 0;
  } // else
} // CheckForValidFilenameWithPath(fullFilename, filename, platformPath)

And that are the most important code changes to make everything run on the Xbox 360, PS3 and PC. Here is the main method for all the TestLua projects (always pretty much the same, only the Xbox 360 uses a slightly different way to print to the console):

int _tmain(int argc, _TCHAR* argv[])
{
    printf("Initialize Lua\n");

    Lua = lua_open();
    luaL_openlibs(Lua);
    lua_register(Lua, "print", lua_print);

    // Simple print test from Lua:
    luaL_dostring(Lua, "print(\"Hi from Lua\")");

    // Now load a Lua file, which also loads another Lua file (LuaUnit.lua)
    if (luaL_dofile(Lua, "SomeClass.lua") != 0)
    {
        printf("Error: Unable to execute Lua file %s: %s\n", "SomeClass.lua",
            lua_tostring(Lua, 1));
    } // if (luaL_dofile)

    // Stop the Lua engine and cleanup
    printf("Closing Lua\n");
    lua_close(Lua);
    Lua = NULL;

    // Keep command window for debug mode (ctrl+f5 does not have this issue)
    char input[100];
    gets_s(input);

    return 0;
} // _tmain

Summary:

• Using Lua on other platforms is easier than Python or other dynamic languages I have checked because Lua is written completely in ANSI
• Making code run on multiple platforms is still hard in C++ because you not only end up with ugly #define code, but you also have to test a lot
• Building native applications is freaking time-intensive, it took me almost a full day to write this app and test it on every platform from serval PCs. In .NET it would take me less than an hour, but then again there is no .NET on the PS3 and our current game engine is using native code anyway.
• On the Xbox 360 you always have to use full paths like D:\Lua\SomeClass.lua, relative paths won't work. The SDK says you should use Game:\, but since our game engine uses D:\ too, I left it that way in this sample too.
• The PS3 is even more complicated with lots more possible mounted paths, but it seems /app_home did always work in both my samples and our game, so I'm sticking with it :)

If you are interested in Lua from .NET, check out the LuaInterface project. Someone also started NUA (Lua on the DLR) 2 years ago, but it seems this project (and many other attempts) have died a long time ago. There was also a port by the Lua guys (especially the great Fabio Mascarenhas, who has also started the LuaInterface project and is still involved) called LuaCLR, but it is still in alpha stage and pretty useless, LuaInterface on the other hand is a fully working library! The difference here is that LuaCLR emits CLR code like any other .NET language, while LuaInterface still relys on the Lua runtime. LuaInterface uses only 2 very simple C++ files, the rest is in C#, but because of that you currently cannot run it with XNA on the Xbox 360. IronPython, while fully written in C#, does not work on the Xbox 360 either because reflection is not supported by the Xbox 360 .NET framework.

And finally here is a link to Lua running on the iPhone: Lua v5.1.3 for iPhone/iPod, kinda cool, but since we can run C# code on the iPhone with help of the great Unity3D engine, I'm happy with that at the moment. The main problem for other languages on the iPhone is the missing libraries anyway (can't use Cocoa or any own UI) ..

Just in case you have not read about it yet: John Carmack has ported the good old Wolfenstein 3D game from 1992 to the iPhone as well.

Last week I finally bought a Intel's X25-M 80GB SSD and I have been using it for about a week now at work. A few weeks back I also posted my impressions about the cheap OCZ 64 GB SSD I used before and how bad it worked out with Visual Studio compiling for me (also some helpful links in there if you want to optimize your SSD).

My first impressions of the Intel X25-M 80GB are as follows:

• Initially very fast read, easily archived over 210 MB/s read performance with HDTune and other benchmarks.
• Blocks above 4kb were faster, smaller blocks were slightly slower. 4096 bytes also seems to be a good NFTS block size for them.
• Writing on it is slower with 70-80 MB/s, which is less than the OCZ SSD I had before in benchmarks. But real world usage tells another story as I was usually limited by the network, reading data from other disks and other factors. Other than the OCZ SSD the Intel one stayed very fast even with many files and programs open and doing crazy compiling in the background.
• Using Windows Vista is fine, but I remembered how much better Windows 7 was for SSD when I was using it a month ago. You can tweak Vista quite a bit with the helpful tips from the OCZ Foraum, but it is not the same thing as having the operating system handle all the dirty stuff for you in a much better way.
• So I'm back at Windows 7 again. Yes some apps and games do not work and I even have problems now doing some work related stuff (building content) for which I have to use another PC or use my secondary disk and boot Vista for that. But most tools I use everyday at work are just fine in Windows 7. Boot time is reduced, memory usage is way better, SSDs work great and the overall desktop experience is just so much better (I love the jiggle windows trick to show the desktop).
• Compile times for our game have gone way down (more on that in a second).
• When I'm going home and do some work there I clearly notice the difference. But even if I just open up FireFox or some other complex program, it seems like it takes forever on a normal hard disk. I was pretty happy with my system at home before, but I have not upgraded it for 2 years. It has never really bothered me before since I can still play all the games in Full-HD and everything is quick in Windows. I will probably wait until some cool new DirectX 11 cards will come out and of course some cool 8-core CPUs for some serious multicore action.
• I can sense a slight slowdown to last week, probably due the fact that I use the SSD each day heavily and it is pretty full. Hopefully it will not get worse over time, but even then I could quickly just copy everything over to my secondary disk, reformat and copy everything back.
• Do not believe most SSD reviews, it is really stupid that in the last 6 months almost no SSD review checked the usability of those things, even Linus Torvalds agrees with that (he is a Intel SSD user too ^^).
  
• Also check out this very good new article on AnandTech: The SSD Anthology: Understanding SSDs

I could probably go on and on about this and there are certainly advantages (yeah, reading is fast) and disadvantages (wtf, this is costly, and you will run out of disk space sooner than you can say "copy this over there"). Lets take a look at some random numbers of compile-time from our current game project (which I'm still not allowed to talk about, so just pretend it is a huge C++ project with many many files). Please note that all those times are not in any way accurate, both because I just checked a few times and estimated others and it is also very different from PC to PC anyway plus there are many other factors like having intermediate files already present or not, etc. I'm also not a hardware tester like Tom's Hardware Guide, go there if you want more and real benchmarks, this is just what a noticed in the last weeks. Keep in mind that I could only overclock my current Intel i7 setup a few days ago since the default CPU cooler by Intel for i7 is really crap, loud and can't cool very good. So I bought a Zalman 9900 instead and it allowed me to go to 3.75 Ghz without any problems (up from 2.66 Ghz default).

Older PCs we have at work are obviously slower anyway, but the point here is that the SSD gives an additional bonus. I also only included the PCs I have worked on, some interns have slower PCs, but they usually do never have to do a full recompile anyway. Keep in mind that I do not compare .NET compile times here (they are all below 2 seconds on all those systems even in the larger .NET projects I have). This is all unmanaged C++ code, but since we also do a lot of our code in scripts we do not have to compile C++ most of the time - and even when - we usually only change certain parts of the code and then a incremental build is used.



PS: I was watching the current South Park S13E02 Episode while writing this and I think it was pretty crappy (superheroes, no real story). Last weeks episode with Kenny's girlfriend was way better and a really good season starter.

One of my interns, Stefan, just send me this, this is hilarious:


The Samsung 256GB SSD MLC still seems to be expensive (and is not even available in online shops here in Germany), costs probably around 800 euro anyway. It does not seem to use the crappy JMicron controller, but I could not find any good benchmarks for it.

Today I bought a Intel X25-M 80G SSD today, which I will use instead of my current one from OCZ, which pretty much sucks for Visual Studio compiling as reported before.

After not sleeping too much since my long day yesterday I headed to the CeBIT 2009 here in Hannover. I was not expecting to see much new innovation, but since I have covered the CeBIT 2008 (My day at the CeBIT 2008), CeBIT 2007 (Impressions from the CeBIT 2007) and CeBIT 2006 (CeBIT 2006 pictures) on this blog (check those blog posts for many more pictures), I did not want to leave out CeBIT 2009 :)


Some flags. The first few halls were pretty boring, many small hardware vendors from Asia, a lot of equipment like keyboards, mouses, memory readers, etc. The only interesting thing was many book reading devices like the Amazon Kindle, but much cheaper and IMO easier to use without all that stupid DRM and restrictions. But buying eBooks at Amazon for the Kindle is still way too expensive, gotta check back in a few years ..


Intel had a whole hall to themselves (hall 22), which they called "Intel Extreme Masters". What a stupid name. This was more or less the e-Sport hall and it was pretty interesting, but way packed with people.


I still do not understand why so many people think it is a good idea to show people playing WoW competitively. It is just boring! Other than that a lot of WarCraft 3 and Counter-Strike could be seen. A friend of mine also told me that StarCraft 2 might be somewhere here to play, but I could not find anything about that.


The big ESL stage. Some guy is playing WarCraft 3 I guess.


The e-Sport player area. I heard that Intel sponsored a price pool of about $ 750k, nice nice.


The next hall was also very full. I was annoyed a bit that everyone walked so slowly.


NVidia and many other graphic hardware vendors were advertising the new GeForce 285, 295 and the cheaper 250 and 240 variants. Still just DirectX 10, where are the DirectX 11 cards so I can try out some crazy tessellation? :)


Some cool cooling equipment to keep everything cooled down. That reminds me that I still have to buy a better cooler for my new i7 at work, the default cooler from Intel just sucks. It is very loud and does not cool very well!


Some guys playing WarCraft 3 and a lot of people watching. I prefer watching VODs on YouTube ^^


Power consumption of servers is going down as well as for desktop PCs and laptops. Not sure why everyone is so excited about it, I never cared much about power consumption because power is cheap anyways. But it is still nice since less power consumption means less heat and more overclocking potential.


Extreme Performance DDR3 Ram from Corsair, costs more than your whole PC ^^ While the numbers seem to be quite high, most benchmarks and reviews tell us over and over again that even the most expensive DDR3 ram modules are not much faster than everything else, which is sad because I think ram should be much faster by now.


At Saturday a lot of shows and free stuff thrown into the crowd prevented me from walking any faster. My brother warned me and told me the first day of the CeBIT is always the best!


A transparent screen. Looked pretty cool and was way big. I guess great use might be advertisement in the future on normal glass surfaces (trains, buildings, etc.).


There were also very cool water screens on the CeBIT:


Some impressions from outside the halls.


After a little snack I headed back into some of the less interesting halls for me.


But after seeing this (are we at a fashion fair?) and getting pretty tired I decided to call it a day and leave the CeBIT.


See you next year on the CeBIT ^^ BTW: Thanks to Microsoft for the CeBIT tickets :)


Wtf? I'm not entering that train!

While it was fun staying in Seattle for a week and meeting all the MVPs and Microsoft people again, I was happy to finally return back home. I missed my computer :D


I took some more random pics of the streets of Seattle.


It took almost 30 minutes until the bus to the airport came, but I still had 3 hours left, more than enough time. I also met up with a German Windows Mobile MVP, which was pretty much pissed about Windows Mobile 6.5, which did not has any new useful features and just can't compete with the iPhone.


As usual, for me it is always fun to take pictures of people taking pictures.


Heading to the Seattle airport.


Arriving at the check-in gates. Still 3 more hours to kill.


Oh no, self-service check-in computers. This will end horribly as usual! This crappy thing did not detect mine or that other MVPs guys passport and due to all the back and forth it takes like 3 times longer just to get the boarding pass. I also only did get the boarding pass for Seattle to Amsterdam, more on that problem later.


I think I saw this crazy thing before last year at the Seattle airport ..


Waiting in line for the security check. Always fun to have some Asian guys in front of you that did not understand the security officer very well and made everything wrong they could do (not taking off their shoes, not keeping their passports in their hands, not removing the laptops from their bags, etc.). But I have to say getting the boarding pass and going to security is taking less time every year. Some guy had a T-Shirt explaining Geeks to us, which is helpful to know ^^


This thing is strange, buy a DVD for $ 5.99 and you can only watch it for a day because the DVD has a protective film on it and once to remove it for watching the DVD will be unusable after a day or two.


After walking around in some shops, but not finding anything useful (no good books or other stuff I wanted), we could finally board the ship, erm, plane. As you can see it was raining outside, same weather conditions as in Amsterdam and Hannover.


Seattle is getting smaller and smaller ..


For most of the flight we could not see much, it was very cloudy in the beginning, then we flew through the night from Thursday to Friday and in the darkness you cannot see anything either (not much going on around the North pol area anyway).


After about 10 hours it got bright again. I could not sleep on the plane as usual. Now it is Friday and we are flying over the UK right now.


And about an hour later we finally landed in Amsterdam, where it was raining too. The airport is really huge, I'm always amazed when landing there.


A last look at the plane that got us here so far.


Now I'm quite confused. I do not have a ticket to Hannover yet and I thought I have to get my luggage and then check-in again for my boarding pass to Hannover. This happened to me few years ago, but I had a lot of time between the flights, not just a little over an hour.


So I headed over to the Luggage Drop off just to find out that all the luggage is transfered to the next flight anyway. I took a closer look at the first boarding pass and on the back there was a little sticker telling me just that. Warg, let's get back to the gates.


But security would not let me through the other way. They said I had to go through Customs, then to the Arrival area, up to the Departure area and then check-in there again, go through security and so on. OK, I did that but in order to get through security you need a boarding pass, which I obviously did not have. So I headed to the self-service check-in area to one of those stupid computers. I tried to get my boarding pass and the computer freaked out. My first attempt was aborted and then I tried it again just to get a random error and the computer telling me I had to go to the custom check-in area. Some guy came and told me to try it once again, same thing happened. Then I had to wait in line for about 25 minutes to finally get my boarding pass from the custom check-in area. But the boarding time to that plane to Hannover was just 5 minutes away and I still had to get to security and then head to that gate. I rushed over there with some girls in front of me at the security gates, which similar to the Asian guys in Seattle did everything wrong they could (not taking off their jewelry, not unpacking their bags, etc.). But I arrived just in time at my gate and got on the bus to the plane. Pooh, this is just too much stress after being awake for 18 hours ..


After about just 1 hour we arrived in Hannover with this little plane. The stewardess was a little bit annoying not letting me hear some music from my mp3 player, because that could crash the plane. Yeah, sure .. just leave me freaking alone ..


The Hannover airport is not very big, but a lot of people probably arrived here this week because of the CeBIT, which I will attend to tomorrow as well. My brother already told me about some cool things from there.


I probably have photographed this model of the Hannover airport before, but I was too tired to remember. Looks cool though.


And this was my train (S-Bahn) getting me home. I could stay away by hearing very loud music, which probably annoyed some people in the train, but I did not care at that point :)


I took a little nap and headed to the office in the evening just to test out if the US-Version of Halo Wars works with my 3 year old Xbox 360. And yes, it does. Great! Good thing most games to not use that stupid region-code anymore these days.
Halo Wars itself is nicely produced, the cut-scenes and videos are top-notch. It is probably also fun to play for Halo fans and RTS newbies. But this game is in no way comparable to any PC RTS game. It is more like playing shooter or simple role playing game from above. The controls are pretty much the same as in Lord of the Rings and C&C Red Alert 2 on the Xbox 360, which is pretty much crap and highly simplified. The game is also way too easy, I just have played the first 3-4 missions, but you spend 90% of your time to press LB (selects all units you have) and then X to attack or move. The rest of the time you go back to your base with the horrible move control and build the very simplified base and some units, which is even annoying because you can only build when you have enough money, you cannot queue anything up for later. Last but not least the graphics are not very good, after watching those very high quality cut-scenes you will be thrown back to reality with the poorly lit scenes in-game and the horrible shadow-artifacts and very low polygon models (which does not make sense since the Xbox 360 is usually fill-rate limited, not so much vertex-limited). Also I would like to mention that I really hate the fact that every single Xbox 360 game says 1080p on the box and none of them support that mode, they all just render to 720p and scale everything up, which just looks horrible! Especially if you are used to Full-HD and much higher resolutions on the PC as I am every day.
It will probably sell well anyway as all Halo games have. But do not expect this game to be released on the PC ever. It would just not work and probably require much different controls and be way too boring and easy for PC gamers. Also Essemble Studios is no more and Microsoft will probably not want to port it to PC anyway.
But it is very cool that US Xbox 360 games work just fine in Europe. NTSC/Pal issues are non-existant anyway because for HD TVs these formats are not used (and would be supported anyway).


This trip was a lot of fun and I hope you enjoyed my pictures and comments. My webserver crashed several times in the last days and was not able to handle all requests (its running on a 1mbit dsl line, which pretty much sucks). I switched to another 1mbit dsl line, which seems to be more stable (but is still slow). I will have to switch to a real server next week, this is just too much for my little server at the office. Luckily I have access to some other servers. I just was too lazy in the past to setup my blog there (and clean it up a little), but now it makes sense since I'm blogging again, yay!

The last day of the MVP Summit (Wednesday) was actually quite short, just a couple of keynotes. But since the last few days have been so packed and exciting, it was nice to have some time to finally relax. I also needed that time to finally finish blogging these posts since I lagged several days behind.

Walking over to the Convention Center was the usual 1 minute walk. Initially I was a little confused because there was no brakefast at the hotel, but then I remembered today it will be at the Convention Center.


My breakfast for today. I was kinda late because I slept relatively long, but then again we stayed at the EMP Party to the very end last evening.


Tony Richards got everything started this day too.


The theme of this MVP Summit seemed to be Windows 7 and this stupid "I'm a PC" slogan. We even got some stickers for that. Nick had one of those stickers on his Mac, but that seemed strange too.


As you can see the crowd was very excited about something. Who can guess what is happening next?


Yes, you are correct, Steve Ballmer is coming up and bringing a lot of excitement with him, as usual. Even though Steve just reiterated whats going on at Microsoft, it was fun to watch him present and talk.


The Q&A had to be messed up with stupid questions and giving gifts to Ballmer as usual. But it got even worse in the next talk, which was supposed to be about Visual Studio 2010, but I did not even see a single screenshot of it. Probably have to check out the VS2010 PDC sessions sometime soon again (when the next beta/ctp of VS2010 comes out, hopefully that one is not on a virtual PC, which was just way too slow for me to do anything).


There was a short break between the first two keynotes and the last two keynotes.


Some guy had a T-Shirt explaining Geeks to us, which is helpful to know ^^


In the next keynote we saw a video about a little girl telling us how great VS is. Seems like that Video has been on YouTube for about a year and it does not seem to have been watched much.

After some other "This is coming up" slides Somasegar introduced us to the technical panel, which even included Anders Hejlsberg. But the Q&A just messed everything up again, most people were just asking why VB6 is not supported better or why Fox Pro is not supported. Same procedure as every year. The last talk about Windows Server 2008 RC2 and Hyper-V and other cool virtualization technologies was interesting again.


And thats it, the MVP Summit 2009 is over. In usual tradition all the XNA/DirectX MVPs got together at the last lunch and we even put together 2 tables again, same as last year.
Richard Thomson (on the right) has been a DirectX MVP for a long time and usually has a lot of stories to tell.


Nick Gravelyn and Vicente Cartas Espinel (Spanish XNA MVP) talking.


Even Andy is listening in, which is a very rare sight (him not talking that is).


Catalin Zima and Andy Dunn. After about 2 hours of talking, everyone split up and left the Conference Center. Some of us also went to some side sessions.


I took a couple more pictures in the streets.


And I went to the movies once again after blogging a bit. This time I saw Taken, which is a very action packed movie, that might not much sense in regard of one guy killing 20 people per minute, but the overall story was good and the pace was also quick so it never got any boring.


And finally some pictures of the Seattle Skyscrapers.


Tomorrow morning I will head to the airport and begin my long travel back to Germany. Not really excited about getting back, but at home it is always most comfortable. At Saturday I will also attend the CeBIT 2009 in Hannover, but I will probably make a short post of tomorrow as well. I hope you enjoyed my pictures and reporting so far.

Time to get up for Day 5 (Tuesday), the second day on the Microsoft Campus with deep technical sessions.


Looking at Mt. Helen from the bridge between Seattle and Redmond.


And finally getting over to the same buildings as yesterday after arriving at the Microsoft Convention Center. I will also come back here later this day for some extra sessions.


Back at the XNA Building I decided to take a look at some cool DirectX 11 sessions after the first XNA session, which was all about XNA Community games, which is still not available in Germany. So this talk not too exiciting for me yet, but stay tuned, hopefully later this year XNA Community Games will be introduced to Germany as well.


Since all the talks (especially everything about DirectX 11) was heavily under NDA, I can't talk about any of it, neither could I take any pictures. All I can say DirectX 11 is pretty amazing and I will definately check it out once hardware and Windows 7 is available (hopefully also later this year). So to keep the pictures going I took some boring photos like this from the ceiling of the XNA/DirectX sessions building.


Later a couple of guys helped out Kathleen getting some beers up to the last session and Q&A, which was a nice oppertunity to see under a Microsoft building. Might not be terribly exciting, but at least something to remember this day by.


After the last XNA session and some beer, some guys headed over to the company store. The only thing I wanted to buy here in the US is the Intel X25-M SATA SSD hard drive, which I could not found anywhere. Buying it online is also impossible since noone is willing to ship it to europe. So I have to buy it in Germany at 135% of the price here.
At around 5 pm we went back to the Convention Center and most guys went back to their hotels, but I stayed to check out a Paralell Programming session, which was quite interesting at first. The presenter was not very excited. But it was nice to see some cool tools of Visual Studio 2010, especially for debugging multithreaded applications. The stuff he was talking about involved a raytracing application with some spheres, which took even multithreaded 9 seconds for a 200x200 pixel image, so not very convincing to me (this kind of problems are easy to split up into multiple threads) and the other topics he scratched on did not sound very exciting either and since I hoped to see more VS2010 tomorrow at the keynote, I also headed back to the hotel at around 6 pm.


It was raining quite a lot and the EMP party (here are some videos from last year) has already started, but I headed out to finally buy some new batteries for my camera.


Since it was not allowed to take photos of the Science Museum at the EMP and I just wanted to drink and talk I did not take the newly operational camera with me. Instead you can see a boring picture of my hotel room until more pictures will be done tomorrow.

Day 4 (Monday) was actually just the second day of the MVP Summit and the first day all MVPs went to the Microsoft Campus. I was looking forward to see all the XNA MVPs from last years again.

The bus was packed as usual and I thought I was prepared very well, which turned out to be a wrong assumtion in a few minutes. Sorry, my main camera batteries are still dead, so not many pictures today, but at least I try to mention the people on the pictures and link to them.


As you can see the weather was not very good in the morning. Typical Seattle weather, but it got much better later this day.


At the MSCC (Microsoft Convention Center) at the Campus I pulled out my schedule for the day and tried to find out where do I have to go next. But there were no freaking building or room numbers on the schedule. Apparently if you have printed it out earlier than 3 days before the summit, there was no building/room information. Wtf? I tried to go online with the iPod Touch and it took me 5 minutes just to type in the stupid password and wait for timeouts. Then I tried to find out the schedule, but no, you have to logon to the MVP Summit site first, which again required a password I had not with me. I gave up and asked someone and was told Building 85 is probably the right one for me. I took the Tour 5 bus to get over there (the Microsoft Campus is quite huge).


At the building no one seemed to know what I was talking about when I asked where the XNA/DirectX session is. So I just pulled out my laptop and had to login again to the MVP Summit site just to find out the freaking room number. At least it just took a minute to get to that information from my laptop. I just should not use a freaking mobile device to find that kind of hidden information in the future. The meeting had already started, but I did not miss much because it was still a general talk about XNA and game development in general by the XNA game platform general manager Boyd Multerer. BTW on the right in the white T-Shirt you can see Catalin Zima, which I met last year for the first time. He seems to be on the XNA Community Forums quite a lot with the other crazy Forum MVPs like Andy "ZMan" Dunn and Nick "MrTwitter" Gravelyn.


The next talk was about upcoming XNA features and you can see all of that on the screen in the middle. You probably only get that joke as an MVP with an NDA that has to keep quiet about upcoming XNA stuff ^^


After some more talks and lunch we went to some test laps to test some games and play around (probably can't tell you more about this). On the right we see Nick smiling and probably twittering as usual. Apparently he was a little annoyed about some pictures I took of him last years, where he was in the middle of telling something ^^


At the Xbox 360 Building there have a lot of cool stuff staying around and many posters on the walls. Some guy had even some cool Left4Dead posters on his door, which I now want too since I'm a huge Left4Dead fan (on the PC however). No idea where to get such things. That Halo guy is not bad either.


Later that day we had some time to kill and went to the mall and checked out this book store for some XNA books, where some MVP was involved. Andy is grinning into the camera as usual and is continuing to tell us that we should not write books ^^


We also found a board and card game shop, where people could actually and play and try out games, cool stuff. But I have not played anything like that since I'm out of school. BTW on the right you can see Vicente Cartas Espinel, which is now my most favorite XNA MVP since he asked me to give him an autograph for one of his friends ^^


And yes, there was even a StarCraft board game. Also a lot of World of Warcraft, Star Wars, Settlers, etc. But since I do not know any people that would play such things with me, it is not really interesting. Probably fun stuff.


And finally we went to the XNA Party at some stylish bar, which included billard, rock band, pizzas, beer and cocktails. Very fun evening and it was very nice to learn more about the new MVPs and speak to them.


We also got some cool extra stuff from the Xbox/XNA team including the new Halo Wars Xbox 360 game, that will come out tomorrow. I'm a huge RTS fan, but I'm still not sure this will be as great as everyone says because both C&C and Lord of the Rings pretty much sucked on the Xbox 360 for me (I think it is even worse than controlling 3d shooters, which is pretty crappy too). Anyway, probably still a very good game and I'm very excited to try it out once I'm back in Germany. Hopefully it will even run on my German Xbox because it says NTSC on the box (we got Pal in Europe), but as far as I heard most games do not check for the region code anymore and PAL/NTSC does not matter anymore with HDTVs.

On day 3 it was time to move from my first hotel to the second one, which is payed by Microsoft ^^
The streets are empty as usual and the weather was not very good in the morning (raining).


The walk from the first hotel (8ths Ave In) to the next (Sheraton) was about 1km and not much of an issue.


After dropping off my luggage I headed over to the Convention Center next door and the raining had stopped.


At around 12 am I registered for the MVP Summit and got my badge and swag. Since the keynote does not start till 4 pm, I planed to go watch another movie. This time in the Regal 16 like last year right in front of the Convention Center.


Since I still had some time until the first movies start at around 1 pm, I watched some StarCraft VODs via the Wifi on my iPod. I have to say it is nice that Apple added YouTube support to the iPhone/iPod, but the interface is so incredibly bad, I much rather search for stuff myself via Google or the YouTube website.


Okay, time to see another movie. It was raining again. I saw Gran Torino with Clint Eastwood and this was most definitively the funniest movie Clint Eastwood has ever made.


After that I walked a little through the city and got something to eat.


And this is how police cars look in Seattle.


Ohh, its time to get back to the Convention Center, the keynote will start soon. This is the area where each region will later meet with all the other MVPs from each region.


Getting ready for the keynote.


Some details about the agenda. The next 2 days will be packed with deep sessions. Most keynotes will actually be on the last day, which will contain probably the most interesting announcements about Windows 7 and Visual Studio 2010. Also notice that there is no more Salmon this years, since this seemed to be the number one complain from last yea