Buyaa, I'm Microsoft MVP once again in the XNA/DirectX category for one more year (2009/2010). I have been an XNA MVP since 2006 and I'm still very proud of it :)

I was pretty busy this year with our current game project "Fireburst" at exDream (a racing game for Xbox 360, PS3 and PC using the Unreal3 engine, more about it soon) and theirfore I did not do many other things (except writing some iPhone games, starting to develop my own dynamic language and some tools). But once that project is done in 1-2 months, I will do lots more XNA fun stuff and hopefully XNA Community Games (now called Indie Games on the Xbox 360, I really hate that term) will be available in Germany so I can finally review and submit some games myself. I'm still pretty fit in XNA 3.1 and DirectX 11 (played around with it a lot in March), but for XNA I'm still waiting for availablilty in Germany and for DirectX 11 I'm really waiting for some cool hardware at the end of this year.

At exDream we recently also had some interesting discussions about using .NET for PS3 (recently possible thanks to Novell), Xbox 360 (hello XNA) and PC or even more dynamic script languages for upcoming projects. Maybe also including some other promising platforms such as iPhone (also .NET able thanks to Unity), Android, PSP Go, WII, whatever, but that all should depend more on the game and if we are able to manage so many platforms. Working with XNA was almost no extra work to make a game run on the Xbox 360 plus the PC, the game just has to fit and you obviously should allow control with the Xbox 360 controller. Our multiplatform game Fireburst is also not that bad to develop for because most issues are handled by the great Unreal3 engine (except the fact of course that it is all ugly C++ and UnrealScript code, which just looks like C++ anyway and even has to be compiled), but it is still quite a lot more work than just doing a PC only game, especially because of optimizations and testing required for all platforms.

Enough rambling, I'm going to celebrate this day by installing Windows 2008 server (omg) because our pre-release version we had on there just ran out (warg) .. stupid thing has to be completely reinstalled, no upgrade option ..

I want to quote from this great article I just read: Singletasking: The Next Trend in Web Working?

"Singletasking is just what it sounds like: approaching and tackling one task at a time, sequentially, instead of trying to do a whole bunch of things at once, as has become de rigeur in our modern multitasking age. If you’re like me, the thought is probably at least a little refreshing, and maybe more than a little appealing right off the bat.

The principle is sound. Take on one task at a time, and don’t begin another until the one you’ve already started is complete. It sounds simple, but you know as well as I do that actually implementing that kind of thing in real life will take a lot more effort than you might first think. For one, it means ignoring any urge to procrastinate, and making sure that you prioritize very carefully in advance, lest you realize too late that what you thought was most urgent actually could’ve taken a back seat to something else."

The article goes on with tools used to track work tasks and even ideas on only use one Tab in Firefox and only using one monitor. I can hardly agree with that, I currently have 10 tabs open (after reading all emails and stuff on the web) because I don't have time right now to finish reading those sites or because some things are still pending (waiting for a response and keeping the tab open as a reminder). I also currently look at 3 monitors. Well not really, I only write this on my right monitor, but I have my main task (some optimization work) on my main screen in the center in Visual Studio. I really hate putting anything above Visual Studio because then I won't focus on writing code, but do other stuff instead. I only do it if I have to write some document, have to do some web research or reading some documentation.

While I totally agree with doing (and finishing) one task at a time (that's actually how my TODO list works that I use in every single project for the last 6+ years), I just can't agree with just having one monitor (or just using one at a time like the writer of the article proposes). For example when opening an editor like for our Unreal3 game, you absolutely need 2+ monitors because else you just have 5 overlapping windows and spend half of your time moving them around trying to see whats behind them.

In Visual Studio I only have the Solution Explorer sometimes and some Find Results or Console Windows on my secondary screen, but all the screens are filled with useful stuff anyway. I also have basically 2 screens at once at my primary screen anyway because I always look at 2 concurrent tabs in Visual Studio (really can't live without it). That does not mean that I really multitask. I ALWAYS only work on one tab, but I can quickly look over, see all the variables and methods that I need and I can code much faster that way.

If you want to have some distractions when working I suggest only checking them out after completing a task. I usually do it as a reward kinda thing, like playing a game I wanted to test out only after I finished my current task. And then continue with the next task.

As Jamie Cansdale posted in my comments ^^:

There's now a version of TestDriven.Net what works with VS 2010:
http://weblogs.asp.net/nunitaddin/archive/2009/06/03/testdriven-net-2-22-support-for-visual-studio-2010-beta-1.aspx

Finally time to use VS2010 a little bit more, without TestDriven.Net it was just unusable before.

I worked all weekend on AI and game logic scripts in Lua and UnrealScript .. would like to talk about it, but since our game project is still not announced yet, I'm not allowed to .. grr ..

Also reading (or hearing) right now: http://www.brainrules.net/
And working through the great new release of the DLR and the new example script language Sympl: http://www.codeplex.com/dlr

Download link for CR_RandomBackgroundChanger now at CodePlex.

I used several tools over the years to change my background image automatically every few minutes. I even collected over 50k desktop images over the years ^^ While this is great and always fun to watch at changing backgrounds every few minutes, most of the time I do not even notice because all my screens are completely covered by windows.

As a programmer I have Visual Studio open most of the time, especially at work, where distractions like changing background images would have the biggest impact. Luckily Visual Studio has a boring white (or black for me) background and will not distract us that much. It would be terrible if Visual Studio would have changing background images that even make it harder to look at code and could be distracting having a background image changing every 5 minutes. Let's explore that!

First of all: How to even change or set a background image directly in the Visual Studio code window?
There is a great collection of gadgets from the SlickEdit guys, called Free SlickEdit Gadgets, which are completely free to use and include a feature to set a background image in the code editor. Searching for this on a search engine like Google isn't that easy, it is easier to find if you search for "Visual Studio Dancing Banana" :)

After you have installed the SlickEdit Gadgets (which currently only work on VS2005/VS2008, not on VS2010 yet), you should see a SlickEdit menu item. From there you can select Editor Gadgets, which just opens the Visual Studio Options dialog. There you can go to the Editor graphic tab and select a background image to be displayed in the editor. I use the following settings:

• Display an image: Checked
• Image textbox: The selected image
• Tile Radiobutton checked (use bottom left if you just want a small transparent image there, IMO not distracting enough!)
• Lock the image location checked
• Transparency checked and set to 75% (which means 25% visibility, 75% of your background color remains).

Now your code window might look like the following:


While this is kinda fun, it is not distracting enough yet. We need changing background images and preferably a way to quickly change the editor background image with a click at anytime in case it doesn't fit. First of all we need to find out where SlickEdit saves this background image. After searching for some SlickEdit setting file and not able to find any, I checked the registry and found this key: HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\SlickEdit\EditorGadgets\ImageFilename This is where the image name is saved. Just for fun I changed it to "C:\Users\Public\Pictures\Sample Pictures\Penguins.jpg", but obviously nothing happend because there was no reason for Visual Studio to reload and apply any new settings from the Options dialog. So how to we force that to happen?

Well, after trying to find some solution for that for a while, I gave up. There is no event I can trigger to update this and there is simply no functionality in the SlickEdit Gadgets to get this update working.

Since I would need to write something to set a new background image every few minutes anyway and this won't be easy or even impossible with my little Wallpaper Changer app I decided to write a VS addin myself. I started a new CodeRush plugin (which also runs on the free DXCore framework) and called it CR_RandomBackgroundChanger (a little bit too long for CodePlex, so there it is just called VSBackgroundChanger). Next I used the EditorPaint event and wrote the following code to display an image in the code editor:

 private void BackgroundChangerPlugIn_EditorPaint(EditorPaintEventArgs ea)
 {
     //Log.Write("BackgroundChangerPlugIn_EditorPaint");
     //tst: ea.DrawLine(1, 1, 50, Color.Red);
     if (currentBackgroundImage == null)
         currentBackgroundImage = Bitmap.FromFile(
             @"C:\Users\Public\Pictures\Sample Pictures\Desert.jpg");
     ea.Graphics.DrawImage(backgroundImage, 0, 0);
 } // BackgroundChangerPlugIn_EditorPaint(ea)
 

This produced terrible results. The image got displayed for a fraction of a second and then was replaced by the default background color. Using other events from the StandardPlugIn class in the CodeRush framework like EditorPaintBackground, EditorPaintForeground, EditorPaintLanguageElement, EditorvalidateClipRegion, etc. also did not help much, the background image was usually not displayed and even if it was flickering terribly and disappearing all the time.

So CodeRush was not very helpful in this particular instance, but no reason to give up. Actually I gave up that day after trying out so many things for many frustrating hours. The next day I tried something different: Use the NativeWindow class to intercept any event that is send to the Visual Studio code editor window (called View in CodeRush btw). Basically you derive from NativeWindow and then call the AssignHandle method to set the window handle (I just do it in the constructor). From then on you will get all WndProc events and you can decide whether to pass it on to the original WndProc or do something yourself.

public class ActiveViewBackgroundHelper : NativeWindow
{
    #region Constructor
    /// <summary>
    /// Create active view background helper class, each document view
    /// will get an instance of this class to handle the background drawing!
    /// </summary>
    public ActiveViewBackgroundHelper(IntPtr windowHandle)
    {
        base.AssignHandle(windowHandle);
    } // ActiveViewBackgroundHelper(windowHandle)
    #endregion

    #region WndProc
    /// <summary>
    /// WndProc, we are only interested at the WM_PAINT event!
    /// If a paint happens and we got a valid  we will call DrawBackgroundImage
    /// </summary>
    /// <param name="m"></param>
    protected override void WndProc(ref Message m)
    {
        // Your own code goes here
        base.WndProc(ref m);
    } // WndProc()
    #endregion
} // class ActiveViewBackgroundHelper

Now I was able to intercept all WM_ERASEBKGND and WM_PAINT messages. WM_ERASEBKGND turned out to be non-relevant because all the rendering happens in WM_PAINT. Otherr events like WM_NCPAINT and all the code rush events happened between a single WM_PAINT event anyway. So I had to dig deeper into WM_PAINT. I could not render a background image at the very start of WM_PAINT and if I aborted rendered right there, it would be displayed in the editor. But there was no text anymore making VS just useless. I had to figure out how to render text on top of my image by copying all the VS editor content to a bitmap and then later drawing it on top of my background image. To make all the flickering go away by drawing into a targetBitmap and then displaying that at once to the window (everything else flickers). There is also a lot of other optimizations and tricks in there. We will get back to the code in a little bit.

After working on this for the better part of the last weekend, I finally got it working and put some extra features in this CR_RandomBackgroundChanger addin, which you can see in this nice option screen for it. Note that I have a black background in Visual Studio, if you have a white one the preview image would be white with black text and it would have 25% of the background image mixed in.



You can download the addin and the source code at http://vsbackgroundchanger.codeplex.com/

I'm trying out CodePlex for this addin. I will probably improve it a little in the future and upgrade it to VS2010. I will also put up other projects on CodePlex like CR_Commenter and maybe some of my XNA games. I also plan to work a little bit more on my language and then put it on CodePlex too when I got a alpha version working with all basic features.

In the option screen of CR_RandomBackgroundChanger you can specify 5 settings:

• Enabled: Whether to use this addin at all. This allows you to quickly disable the addin and keeping all your previous settings once you need more distractions again.
• Directory: All images in this directory and all images in all the sub directories will be used for displaying random background images in your Visual Studio code editor. Please note that searching for 100 000+ images can take a few seconds (I tested it with that many), but the subsequent VS startups and every background image change will be very fast because of all the caching involved. The disadvantage of all the caching is that if you add new images or even delete the whole images folder, you need to update this directory to get all those changes reflected by the addin by either deleting the cached ImageFilenames.txt in your %TEMP% directory or by just selecting the same directory again.
• Transparency (default: 25% opacity): This is the most important setting because having colorful background images at 100% opacity can often make the already colorful editor text unreadable. If you know that your background images have mostly the same color as your code background you can use higher settings (for example I used some black only images with some stuff on the right in the beginning, while this was cool at first, it got boring after a while because I'm used to changing background images every few minutes). If you have photos or other images that do not really fit to having text on top of them use even lower opacity settings (e.g. 10%) to make Visual Studio text easier to read. You can mostly still see the background image if your code window is big enough and you have some empty lines in there :)
• Change image every x seconds: Use this setting to allow the addin to change the background images every 5 minutes (or use whatever number of minutes you like). Note that using 0 minutes will result in constantly changing backgrounds while you are typing and scrolling, not really sure if this is useful, but was good for testing ^^
• And finally the last settings is to use different background images for different document windows. If you have 20 documents opened in Visual Studio, every one of them will get a different background image (which can then change every 5 minutes too). This even makes it easier to figure out where you are, I probably need to improve the Tab-Rendering too to make it more useful like the ColorfulTabs addin for Firefox. And this is of course great fun, usually I get annoyed by having too many documents open so I often close them all after I get 50, but now I really like having many different background images open.

The option screen also features the Preview box, which lets you see your selected background images and tweak the transparency setting until it looks right for you. Once you close the Options screen all settings will be applied to your VS code editor. Please note that this addin only works whenever a WM_PAINT event happens. This is usually only in the active document you are working on. Most of the time the background image will stay even if you change to another tab on the other side (if you use 2 tabs at once, else you will never notice this anyway). Sometimes you see some black or white background lines drawn on top of your background image because just some text was updated, but this happens very rarely. I hope you enjoy the addin, I will probably use it for a long time and I will port it to VS2010 once CodeRush runs there or I'm using VS2010 (still waiting for a working TestDriven.net version). Hopefully implementing this addin with the MEF (Managed Extensibility Framework) is not hard.

Let's go back to the code. Last time I stopped talking about the WndProc method, which is most important for catching and handling the WM_PAINT event. From here we can call our DrawBackgroundImage method, which does all the magic. Please note that the source code for the addin is a lot more complex than the code presented here. This is mostly because of heavy optimizations and a lot of caching to make the code run as fast as possible (one of the most important things about this addin, it would be unusable if it makes VS slow).

#region WndProc
/// <summary>
/// WndProc, we are only interested at the WM_PAINT event!
/// If a paint happens and we got a valid  we will call DrawBackgroundImage
/// </summary>
/// <param name="m"></param>
protected override void WndProc(ref Message m)
{
    // Just make sure we mark the erase background message as handled.
    // Will not do anything anyway (all painting done in WM_PAINT).
    if (m.Msg == (int)WindowsMessages.WM_ERASEBKGND)
    {
        // Mark event is already handled
        m.Result = (IntPtr)1;
    } // if

    // Handle the paint event
    if (m.Msg == (int)WindowsMessages.WM_PAINT &&
        // Are we currently drawing our own background? Then make sure we do
        // not handle this and use the default message handling instead!
        drawingBackground == false &&
        // Make sure we got a view, else we don't have a windows handle!
        CodeRush.TextViews.Active != null &&
        // Check for invalid hwnd, then we can't paint backgroundImage!
        (int)CodeRush.TextViews.Active.Handle != 0)
    {
        TextView view = CodeRush.TextViews.Active;
        IntPtr activeWindowHandle = view.Handle;
        Rectangle rect = new Rectangle();
        Win32.GetUpdateRect(activeWindowHandle, ref rect, false);
        // Only proceed if we have a valid rect
        if (Win32.IsRectEmpty(ref rect) == false)
        {
            // Make sure we mark this flag so subsequent calls to WM_PAINT
            // will actually just paint the normal stuff, not just our
            // background rendering!
            drawingBackground = true;
            DrawBackgroundImage(activeWindowHandle, rect);
            drawingBackground = false;

            // Mark event is already handled
            m.Result = (IntPtr)1;

            // Do not call base.WndProc, we don't want to process it here!
            // Instead we invalidate inside DrawBackgroundImage and force
            // a new WM_PAINT event inside there, which will be executed
            // because drawingBackground is still true while in there.
            return;
        } // if (rect)
    } // if (WM_PAINT)

    base.WndProc(ref m);
} // WndProc()
#endregion
    

And finally the DrawBackgroundImage method, which copies the current editor text as an image, applies transparency to it. Then we draw our background image and the now transparent editor text on top into a helper targetBitmap, which is finally displayed on the screen at once (else we would get flickering issues). Please note that in the source code you can download there is a some testing code, e.g. for drawing bitmaps with transparency using ImageAttributes and SetColorMatrix, which works great, but is just too slow. Our approach is to pre-calculate the opacity in the backgroundImage and use that over and over again (3-4 times faster). There is still a performance penalty for all this painting, but any future optimizations will be hard. I recommend using 2 tabs, rendering is twice as fast (because only one side is updated when you type or scroll) and you should have a fast PC :)

#region DrawBackgroundImage
/// <summary>
/// This method draws the background and is called from WndProc whenever
/// it intercepts a WM_PAINT message. Again, some caching and confusing
/// optimized code is in here too, again for getting good performance.
/// </summary>
private void DrawBackgroundImage(IntPtr activeWindowHandle,
    Rectangle rect)
{
    // Create an image for storing the orginal editor screen.
    Bitmap sourceImage = new Bitmap(rect.Width, rect.Height);
    // Always create new graphics object, else we won't have current data
    Graphics sourceImageGraphics = Graphics.FromImage(sourceImage);

    // And grab current editor window content and copy it to it!
    IntPtr hdc = sourceImageGraphics.GetHdc();
    Win32.PrintWindow(activeWindowHandle, hdc, 1);
    sourceImageGraphics.ReleaseHdc(hdc);

    // Next find the color on the bottom right and use it as the
    // transparent color! Note: Antialasing will cause artifacts, make sure
    // the backgroundImage fits to the background color, e.g. by using a
    // lot of alpha transparency (<25% visibility)!
    backgroundColorPixel =
        sourceImage.GetPixel(rect.Width - 1, rect.Height - 1);
    sourceImage.MakeTransparent(backgroundColorPixel);

    // Create target image where we wanna paint to, this is important
    // because drawing directly to the VS window will produce flickering!
    Bitmap targetBitmap = new Bitmap(rect.Width, rect.Height);
    Graphics targetBitmapGraphics = Graphics.FromImage(targetBitmap);

    // Clear background with pixel color
    //not required, we draw solid image now (use this for transparent drawing):
    //targetBitmapGraphics.Clear(backgroundColorPixel);

    // Draw background image (tiled and transparent if specified)
    float transparency = Options.transparency / 100.0f;
    Image backgroundImage = GetBackgroundImage((int)activeWindowHandle,
        backgroundColorPixel, transparency);
    
    // Do the drawing as many times as we need to tile to fill everything!
    for (int y = 0; y < rect.Height; y+=backgroundImage.Height)
        for (int x = 0; x < rect.Width; x += backgroundImage.Width)
        {
            targetBitmapGraphics.DrawImage(backgroundImage, x, y);
        } // for for

    // Now draw source image on top (text and foreground stuff), else we
    // would only see our distracting background image and while that is
    // fun, we sometimes still need to be productive and see the editor
    // text ^^
    targetBitmapGraphics.DrawImage(sourceImage, 0, 0);

    // Finally draw on the VS window, but only do one single draw here
    // to make sure we do not have any flickering!
    Graphics graphics = Graphics.FromHwnd(activeWindowHandle);
    graphics.DrawImage(targetBitmap, 0, 0);

    // This is important: Validate the rect so all this can now be
    // displayed! All WM_PAINT calls during this method will
    // be handled normally (without our background painting), which is
    // used for normal updates because we mess everything up ^^
    Win32.ValidateRect(activeWindowHandle, ref rect);

    // Dispose everything we do not need anymore
    // Note: We always have to create new graphics and dispose them here,
    // else updating sourceImage, targetBitmap, etc. does not work!
    if (graphics != null)
        graphics.Dispose();
    if (sourceImageGraphics != null)
        sourceImageGraphics.Dispose();
    if (targetBitmapGraphics != null)
        targetBitmapGraphics.Dispose();
} // DrawBackgroundImage(activeWindowHandle, rect)
#endregion

For more information check out the source code, the important code for the addin is in ActiveViewBackgroundHelper.cs and the Options and PlugIn classes. The rest of the classes are just helpers for logging, string operations, random methods and Win32 helpers for some calls.

Download this addin at http://vsbackgroundchanger.codeplex.com/

I hope you will enjoy this addin. Here are 2 more screens from using this addin with different background colors and images:

White background theme (25% opacity):


Black background theme (25% opacity):

Last week when I installed VS2010 I played around with it for a few minutes, but I've been very busy at work because our game is at its final stage and we got a lot of heat going on (this is not a hint, no no, no hint from me, I'm not allowed to talk about it. Damn it, this could be a hint).

Currently the most interesting new feature besides the cool VS2010 IDE is the Parallel Extensions for me. Sadly the IDE still unusable for real work IMO because all the addins just don't work, there isn't even a fix for TestDriven.net yet, Jamie Cansdale is probably busy too ^^

VS2010 support for parallel programming goes beyond just adding a few extra classes in .NET 4.0: You also got a great IDE implementation which lots of useful features and new tool debugging and profiling windows for checking out parallel tasks, threads and the scheduling. Next there are native C++ libraries that work with Parallel Extensions too (using lambda functions) and work good together with STL. You can check out all the new features at the official VS2010 page!

Let's take a quick look on how to use these Parallel Extensions. I wrote this in a few minutes last week, I was just too lazy (I mean busy of course) to post it yet, it is obviously very simple stuff, but was still useful to test out some of the new parallel IDE features and check out some new .NET 4.0 classes. First of all let's do a boring foreach loop, which displays numbers from 0 to 9, which get added to expectedTotalNum. This should obviously be 45 because sum(0..9)=45. Later we will do some parallel foreach adding and check if we got the same result. Since it does not matter in which order we add these numbers, it is also a good test to just start a bunch of parallel tasks and let them do their work. Obviously you would never write parallel code just to add some numbers, but this should illustrate the point and it does not hurt your performance much anyway (as long as you have a few lines of code executing that take more than a few instructions).

 // Initialize a list for some parallel testing :)
 List<int> someInts = new List<int>();
 for (int num = 0; num < 10; num++)
   someInts.Add(num);

 // Print out numbers sequentially
 int expectedTotalNum = 0;
 foreach (int num in someInts)
 {
   Console.WriteLine("sequential num=" + num);
   expectedTotalNum += num;
 }

 Console.WriteLine("expectedTotalNum=" + expectedTotalNum);

This outputs the obvious sequential adding of 0 to 9 to expectedTotalNum, which is 45 at the end of the loop:

sequential num=0
sequential num=1
sequential num=2
sequential num=3
sequential num=4
sequential num=5
sequential num=6
sequential num=7
sequential num=8
sequential num=9
expectedTotalNum=45

Now let's do the same in parallel, just by replacing foreach with Parallel.ForEach:

 // And do it with the new Parallel Programming classes in .NET 4
 int totalNum = 0;
 System.Threading.Parallel.ForEach(someInts, num =>
   {
     Console.WriteLine("parallel num=" + num);
     totalNum += num;
   });
 Console.WriteLine("totalNum=" + totalNum);

While the result is still the same because totalNum is 45 at the end of this loop, we get there in a different way. As you can see this is a little bit confusing at first because the adding does not happen sequentially anymore, but in parallel instead. Also note that this output can change when you execute it again and can even be much different on different platforms with different number of CPUs:

parallel num=0
parallel num=2
parallel num=5
parallel num=7
parallel num=8
parallel num=9
parallel num=6
parallel num=3
parallel num=4
parallel num=1
totalNum=45

Okay, good stuff so far, but you might not always have a foreach loop and you might not want to wait for it to complete anyway. Maybe you just have some work tasks that need to be executed, no matter in what order and you might not even care if they complete their work right away or in a little bit. A method could just add some work that has to be done and then return while the work tasks are executed in the background. This is when you would have used the Thread or ThreadPool classes in the past, which are great on their own, but you always have some setup code and it was hard to test and you could not use them all over the place because setting up threads is a costly operation and if your work is just few lines of code, it was always a better idea to execute it right away. But fear no more, now you can just create new tasks with the new Task class in the System.Threading namespace. This is a much easier job and has many advantages because .NET 4.0 handles all the thread creation for you, will reuse threads once they are completed with their task and all this works in a very performant way. Setting up tasks is a little bit more work than just executing Parallel.ForEach, but it allows much greater flexibility and you can add more tasks from whereever you are. Tasks can even have children and you can have a lot of complex code using all these tasks. Testing multi-threaded code is obviously harder than just writing sequential code, but with all the great additions to the IDE, it is now easier than ever with the new Parallel Tasks window and by checking out the Parallel Stacks, which shows you all the running threads and were all the task code is.

The following code will create 10 tasks and do the same thing as above, but with several additions. To be able to wait for all the tasks to finish we will add all 10 tasks to the allTasks list. We also have to make sure that our local foreach variable does not change while we are executing tasks because the foreach loop will quickly create all tasks, but might not execute them right away theirfore using num can cause problems. Instead we just create a local copy of num and use that instead, which can't change because we never increase it like we do with num. Finally we add some boring thread sleeping to make it easier to debug this code and check out whats going on by adding breakpoints. Without the sleep the code still works, but checking out the Parallel Tasks and Parallel Stacks windows will most likely give us no results or only the last few tasks that are still being executed at the end of the foreach loop because the tasks are so simple and executed very quickly. We even wait a little after the foreach to make sure all the tasks have been added and are being executed right now or are scheduled (waiting for execution).

 // And finally some tasks, yeah!
 totalNum = 0;
 var allTasks = new List<Task>();
 foreach (int num in someInts)
 {
   // We need a local variable for num because num itself can change
   // at the end of this loop before the task might even be executed!
   int numToBeAdded = num;
   allTasks.Add(Task.Factory.StartNew(delegate
   {
     totalNum += numToBeAdded;
     Console.WriteLine("Adding " + numToBeAdded + " in task with id=" + Task.Current.Id);
     // Wait a little for checking out the tasks in the new Tasks window in VS2010!
     Thread.Sleep(1000);
   }));
 } // foreach
 Console.WriteLine("Done with foreach loop. Tasks might still be pending");

 // Wait a little for all tasks to start
 Thread.Sleep(100);
 Task.WaitAll(allTasks.ToArray());

 // And finally return the result (45 again if everything worked)
 Console.WriteLine("totalNum=" + totalNum);

First of all the results again, which is 45 again. This only work with numToBeAdded, if you use num instead it will sometimes give you different results because at the time you execute a task num already might have changed, especially if you have more tasks than CPUs used for execution and time consuming code like Console.Writeline or even a Thread.Sleep is in there!

Adding 1 in task with id=1
Adding 7 in task with id=6
Adding 8 in task with id=7
Adding 6 in task with id=5
Adding 4 in task with id=3
Adding 3 in task with id=4
Done with foreach loop. Tasks might still be pending
Adding 9 in task with id=8
Adding 0 in task with id=10
Adding 5 in task with id=2
Adding 2 in task with id=9
totalNum=45

As you can see this even looks more confusing than the way Parallel.ForEach added those numbers because while we might create those tasks sequentially (number 1-10) it does not mean there are also executed in exactly that way. To make it easier to check these things out there is the Parallel Tasks window, which shows the following after starting all 10 tasks by adding a breakpoint after Thread.Sleep(100) just before we wait for all tasks to complete:



Not only do we see all over our 10 tasks here, we also see right away that 8 of them are currently being executed (because I have 8 CPUs with my hyper-threaded i7) and all of them are waiting because of our stupid Thread.Sleep(1000) we added for each of those tasks. A second later those tasks are done and the last 2 are also executed, most likely with 2 thread ids already created earlier. You can click on each task and see where it is currently executing in the source code and you can also check out all the thread information in the normal Threads window. But even more useful than the Parallel Tasks window is the Parallel Stacks, which shows how all this code is related, which task or thread created which new task and so on:



All good stuff. While I already have some ideas how to use this on some of my current projects, I have not ported anything to .NET 4.0 / VS2010 yet because of the many issues I have with the IDE (no addins, color theme not really working, I always have to reset it when starting VS2010, also I don't like that the project and solution formats have changed so I cannot easily switch back to VS2008, etc.). But hopefully more and more addins will work for VS2010 and some of the issues are fixed, then it will be great to use all this new .NET 4.0 stuff (dynamics, parallel extensions, MEF, etc.).

VS2010 beta came out a few hours ago, installing it took way too long (over an hour, VS2008 takes less than 10 min for me) and it seems some addins don't install or just don't work, but other than that everything else seems to work great:

• TestDriven.NET provides an VS2010 option and it will appear in Addins, but the Menu Items will not appear anywhere. Just use keyboard shortcuts and everything still works :) .. at least it worked for a short while (see below). Hopefully Jamie Cansdale fixes this soon because without TestDriven.NET I can't really use VS2010 right now. MS provides its own unit testing framework, which is implemented into VS2010 and even available in the Pro edition now (was only in more advanced versions before), but I still don't like it. It does not output anything to the console except exceptions; it is not possible to just start some Ad-Hoc unit test and for all my functional tests it is also useless. For just "normal" unit test projects VS test integration is quite ok, but I would suggest trying out NUnit or xUnit together with TestDriven.NET, which is just more flexible, easier to use and improves productivity IMO.
  
• Completely unable to install any CodeRush version (neither 2, 3, 9.1, Xpress), so I'm also unable to use my CR_Commenter :( hopefully DevExpress provides a version soon, they always were very quick with early VS2005 and VS2008 support!
• Most other addins I tried also did not install or just do not appear anywhere (only in VS2008)
• One of my own addins I wrote 2 years ago for VS2008 also does not appear, it probably needs to be configured to appear in the VisualStudio/10 reg key!

Most annoying:

• Even though Rico Mariani (the .NET performance god) is helping out the VS team to give VS2010 better performance, it currently lags all over the place. I had 10 second delays in Options, Add References and Renaming dialogs already. Seems to be always an issue with opening up stuff for the first time. After a while everything seems to be fast, but I would not say this is faster than VS2008 yet.
• Only tried this at my home PC till now, but everytime I close VS2010 I cannot open it up again until I restart (or at least log off/log on) again, which is REALLY annoying. I will just keep VS2010 open now all the time and hope it never crashes ^^ The start splash-screen is also messed up and completely black. Maybe this is only on my home PC because I tried to install so many addins, will try this tomorrow at work too.
  Update 2009-05-18: I just found out what was causing this. Some addins tried to load, but failed for some reasons and then the addin loading code locks up or whatever and VS never finishes starting up (the process devenv.exe is still there, but you just don't see any IDE window, they are never created). It seems to be related to the TestDriven.NET-2.21.2448 version, I also tried TestDriven.NET-2.20.2438 - same problem :( (Jamie Cansdale, the creator of TestDriven.NET was also recently blogging about this). Then I used , which also already has VS2010 support and that seems to work fine (no menus, but keyboard shortcuts work fine). I posted more details on Jamie's blog!
  

Every project I have opened, had to be converted to the new VS 10 format, which is kinda stupid because it seems nothing has changed. All the code works just fine and no changes were required. I will start using some .NET 4.0 features and test out stuff in the next days, especially using Parallel Programming and MEF (Managed Extensibility Framework). Probably will play around with the C# 4.0 dynamic keyword a bit also, but can't think of anything yet where I would really need it. Maybe to talk easier to IronPython or my own DLR language ..

Cool features:

• Start page is pretty good, easily customizable, but I will probably never see it as I setup my VS to always load the last solution (like my Firefox ^^)
• It is easy to drag tabs around and move them to different screens. Finally I can use some multi-monitoring.
• Old exported VS2008 settings also work just fine in VS2010, even coloring :) For some strange reason the background color is always white when I start VS2010 for the first time, then after going to options and closing it, it will be the correct color (black for me).
• Overall the IDE is very nice and fits perfectly into my color theme. Its also a lot easier to see whats going on, which tab is selected and positioning tool windows all over the place is better than ever before.

Go check out VS2010 yourself, available on MSDN right now and on MS site on Wednesday for everyone else.

While I'm waiting for VS2010 beta today (refreshing blogs and MSDN every few minutes :D), I just found out that you can have your Visual Studio Tabs be automatically ordered by adding the following registry key in HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0:

• UseMRUDocOrdering: 1 (DWORD)

Now when you select any tab in Visual Studio it will automatically move to the very front thus keeping your recent documents at the start. It is kinda weird at first, but it seems to be a very useful feature. I like it already :)



In case you don't want to create the registry key yourself and want me to help you out modifying your registry (always a safe idea), just execute this file to do the job:

• VS9UseMRUDocOrdering.reg (.26 KB)
• Update: Does not seem to work yet in VS2010, maybe use /10.0_Config key instead?: VS10UseMRUDocOrdering.reg (.26 KB)

PS: Since my brother is running a TOR exit node server Google does not like when I do searches anymore because it thinks I search too much stuff now and might be infected with viruses and trojans (yeah, sure ..). I could use Google Custom Search sites like Blackle.com, which return the exact same results as Google, but for now I'm trying out Yahoo once again (was using it a lot almost 15 years ago, when there was no Google). Sometimes Yahoo is stupid and does not give good answers, e.g. searching for "year yahoo was launched" gives you random answers on Yahoo, but google returns the Yahoo Questions link with the correct answer as the first result, wtf?

However, most of the time you do not get stupid SEO optimized search results and totally messed up product searches you get when using Google or Live Search. As an Microsoft MVP I also get asked a lot why I would not use Live Search and Microsoft is constantly trying to get more people (and MVPs) using Live Search, but the results are very close to Google most of the time anyway and the interface is just strange if you are used to Google for 10 years (the English Live Search is actually nice with a new image every day, but the German version is totally messed up, just white boxes??). Yahoo at least gives different results, which are sometimes better, sometimes worse .. Maybe we all should switch search engines from time to time, its really stupid Google has 95% market share here in Germany almost and it returns so crappy results so often (finding drivers has become 10 times as hard in the last years, searching files or torrents is impossible, because you only get SEO sites, no real results, product searches are totally messed up, Google groups is not used much anymore, etc.). Okay, enough ranting about search engines .. time to refresh MSDN again.

Today I had a discussion with a few other programmers (Hi ViperDK and Timo ^^) about foreach and for loop performance. Everyone had his own ideas and experiences, but we talked mostly about XNA on the Xbox 360 (using the compact framework, that does not like foreach, just use for all the time in tight render loops for your XNA games, foreach will create too much IEnumerator garbage that will not be collected) or other special cases (huge arrays or lists).

Anyway, just guessing around is obviously not as good as actually writing a performance test application and see whats going on (see end of this article for download link). Again, we need a disclaimer: You will probably never write loops with 1 billion iterations, these numbers are similar for less iterations, but you should always try to write clean code and ONLY optimize when it is required after finding performance issues! Lets go directly to the results for 1 billion for or foreach iterations in release mode (will loop through a 10k array or list for 100k times, see below for code):

As you can see there are some major differences between using foreach with an array or using a generic list (boxing/unboxing is not the issue here, you would not be able to archive this performance with an ArrayList). Another thing you can notice right away is that using simple arrays is always faster than using dynamic lists, even converting them from a list to an array via .ToArray() and then executing the foreach is much faster.

It also makes a HUGE difference executing this in debug mode (I execute my code almost 99% in debug mode), everything is 4 times slower for this special performance test. Except for Foreach+List, which is in comparison not that bad in debug mode. What is really bad is For+List, which almost takes 16 seconds to execute 1 billion times (again using my Core 2 Duo 6600 CPU with 4 GB Ram).



Thanks to some helper methods the code for this performance test is pretty short and easy (check end of this article for the source code download link):

 Stopwatch timer = new Stopwatch();
 // Run through all tests
 for (int test = 0; test < 8; test++)
 {
     // Find out which test we wanna make
     bool rememberLength = (test / 4) % 2 == 1;
     bool useForeach = (test / 2) % 2 == 1;
     bool useList = test % 2 == 1;

     // Start timer for test 
     timer.Start();

     // Execute
     int result = RunForOrForeachLoops(rememberLength, useForeach, useList);

     // Stop timer
     timer.Stop();

     // Skip this test if we got no result!
     if (result == 0)
         continue;

     // Report results!
     resultText = ...
     Console.WriteLine(resultText + ": " + timer.ElapsedMilliseconds + "ms");

     // Reset timer for next run
     timer.Reset();
 } // for

The interesting code obviously happens in RunForOrForeachLoops. It basically executes the 1 billion iterations in several different ways (8 ways, as you can guess by the for main test for loop). All of those produce the same result, half of the tests use int[] intArray, the other half uses List<int> intList. And again half of the tests uses just for loops, the other half uses foreach for the iterations. Lets examine the useForeach=true and rememberLength=false cases (btw: Iterations is 100000 and the intList and intArrays have a size of 10000 elements):

if (useForeach)
{
    if (useList)
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intList)
                result += val;
        } // for
    } // if
    else
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intArray)
                result += val;
        } // for
    } // else
} // if
else // useForeach == false
...

To figure out the differences between those few lines of code in RunForOrForeachLoops, that almost look the same, we have go to IL code once again. First lets check out what the fastest code block Foreach+Array is being compiled to in IL:

result = 0;
foreach (int val in intArray)
    result += val;

This becomes in IL:

 L_0007: ldsfld int32[] TestSimpleForeachApp.Program::intArray
 L_000c: stloc.2 
 L_000d: ldc.i4.0 
 L_000e: stloc.3 
 L_000f: br.s L_001d
 L_0011: ldloc.2 
 L_0012: ldloc.3 
 L_0013: ldelem.i4 
 L_0014: stloc.1 
 L_0015: ldloc.0 
 L_0016: ldloc.1 
 L_0017: add 
 L_0018: stloc.0 
 L_0019: ldloc.3 
 L_001a: ldc.i4.1 
 L_001b: add 
 L_001c: stloc.3 
 L_001d: ldloc.3 
 L_001e: ldloc.2 
 L_001f: ldlen 
 L_0020: conv.i4 
 L_0021: blt.s L_0011

In L_0021 we jump up to L_0011 as long as we are not done with the loop yet. Inside all the result += val happens, but we do not see any slow code, there are no calls to any methods, there is no unboxing and this all can probably be executed quickly after being complied to assembler by the JIT (just-in-time) compiler.

On the other hand, if we look at pretty much the same code, just with intList instead of intArray, the whole story changes:

result = 0;
foreach (int val in intList)
    result += val;

becomes much more complex IL code with lots of function calls, the creation of an Enumerator and there is even a dispose at the end (hello GC):

  L_0007: ldsfld class [mscorlib]System.Collections.Generic.List`1<int32> TestSimpleForeachApp.Program::intList
  L_000c: callvirt instance valuetype [mscorlib]System.Collections.Generic.List`1/Enumerator<!0>
[mscorlib]System.Collections.Generic.List`1<int32>::GetEnumerator()
  L_0011: stloc.2 
  L_0012: br.s L_0020
  L_0014: ldloca.s CS$5$0000
  L_0016: call instance !0 [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::get_Current()
  L_001b: stloc.1 
  L_001c: ldloc.0 
  L_001d: ldloc.1 
  L_001e: add 
  L_001f: stloc.0 
  L_0020: ldloca.s CS$5$0000
  L_0022: call instance bool [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::MoveNext()
  L_0027: brtrue.s L_0014
  L_0029: leave.s L_0039
  L_002b: ldloca.s CS$5$0000
  L_002d: constrained [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>
  L_0033: callvirt instance void [mscorlib]System.IDisposable::Dispose()
  L_0038: endfinally 

What even looks worst too me is the code that is executed in MoveNext (get_Current just returns this.current, no extra instructions here). Since the IL code is very long and confusing (31 IL instructions), here is the C# code of MoveNext:

public bool MoveNext()
{
  List<T> list = this.list;
  if ((this.version == list._version) && (this.index < list._size))
  {
    this.current = list._items[this.index];
    this.index++;
    return true;
  }
  return this.MoveNextRare();
}

And MoveNextRare could also be called at the end of MoveNext if something changes for the list (not the case in our example). But we have to keep in mind that this is just IL code, not actual code that will be executed on the CPU. The JIT compiler can still optimize things out, which we won't see here. But something must be going on since Foreach+List is definiately slower than using all the other approaches to get through all the values in intList. I recommend Foreach+Array converted from List because the IL for Foreach+Array is really fast and does not use any extra methods or Enumerators, especially if you know that the array won't change much and if you do not have to call .ToArray every time when you do a foreach.

And here is the sample project for all this performance testing fun:

• TestForEachPerformance.zip (4.56 KB) project (C#)
  

The last few days I was working a little bit with the DLR and got to a point today where I was wondering about the performance of a very simple while loop in the DLR, specifically in the ToyScript DLR sample project and of course IronPython. I am totally aware that the following comparison DOES NOT really provide accurate information about the actual performance of the languages and techniques used. Please keep this in mind when reading this article, it only covers a very certain aspect of all used languages. More specially how they perform when adding numbers in a while loop a lot of times! Some of it reflects that compiled languages are faster than dynamic languages, but you should not compare the absolute values, Python or Lua is not really 100 times slower, in many cases you can reach almost similar performance as long as you implement or call performance-critical code in a clever way (e.g. calling C++ code, using frameworks, etc.).

I was only interested about very ruff comparisons to figure out which ways are fast and which languages or techniques are kinda slow (click image on the right for a quick overview). Some code written in Assembler or direct IL code execution was really fast (duh, no wonder), but all execution times of C++, C#, Assembler or IL are way faster than all other dynamic languages or DLR approaches. The important thing for me was to figure out why dynamic languages like IronPython or ToyScript on top of the DLR were like 100 times slower for this specific code. At the end of writing this article I was able to provide pretty good performance with the DLR, which is just 2-3 times slower than the fastest execution time. That's pretty good and could probably be even improved more.

Basically I used the following code sample, which adds 1.0 to counter 10 million times while decreasing the loops variable until it reaches 0. The following code block is not valid syntax in any of the presented languages, but it is very similar to Python. In languages that use compile time types, counter would be a declared as a double (64bit floating point number), most dynamic languages use double as the run-time type for floating point numbers also.

loops = 10000000
counter = 0.0
while loops > 0
  loops = loops - 1
  counter = counter + 1.0

print counter

Languages and techniques used and tested (all with detailed code examples and explanations):

• C# 3.0
• Native C++
• Assembler
• IronPython 2.6
• ToyScript
• Python 2.6
• Python 3.1
• Lua 5.1.4
• Script.NET
• IL Emit
• Irony + DLR (dummy)
• DLR AST with objects
• DLR AST with doubles

Before boring you with all the different implementation details, here is the fancy Code Generation Performance comparison. This article became very long; I just wanted to post a few sentences about the DLR and this is what happened. The following graphs were created with Excel with some help from my good old friend ViperDK. The second graph (the first one was already shown above) shows just the execution time of the while-loop code block in each of the used language or technique.


And this graph also shows the other times like compile & build time where it makes sense or script or DLR start-up times to see how much this could affect the overall time from coding to seeing the result on the screen. This is important for my approach to my own language since I want a extremely low time for the coding to see result loop. Start-up time can be ignored in those cases, but it is still important since it is still kinda slow with the DLR (but will hopefully get better in the future).



Here are the values used to generate those graphs. Tested on my almost 3 years old 2.4 Ghz 6600 CPU; this is probably way faster on my i7@3.6 Ghz at work:

Language or Technique	Code lines	Compile&Build	Script Startup	Parsing&AST	Execution
C# 3.0	10	~500ms	-	-	~37ms
Native C++	8	1500ms	-	-	~31ms
Assembler	15+	~1100ms	-	-	~15ms
IronPython 2.6	6	-	~300ms DLR	160ms	~9000ms (~1250ms for files)
ToyScript	8	-	~430ms DLR	103ms	~8400ms
Python 2.6	6	-	~50ms	~100ms	~3500ms
Python 3.1	6	-	~50ms	~100ms	~4500ms
Lua 5.1.4	7	-	~25ms	~150ms	~1400ms
Script.NET	8	-	~100ms	-	~39752ms
IL Emit	30+	-	~100ms	~1ms	~16ms
Irony + DLR (dummy)	30 + 6	-	265ms Irony	87ms	~138ms (dummy)
DLR AST with objects	68+	-	433ms DLR	4ms	~459ms
DLR AST with doubles	60+	-	434ms DLR	5ms	~98ms

Implementing and testing the following code was not too bad for me since I was learning how to generate AST statements in the DLR, generating System.Reflection.Emit IL code, doing stuff with the ToyScript DLR sample, etc. anyway. I was of course interested in how to get the best performance for my code generation, but it is useful to understand why each approach is slower or faster. Sorry if the comparison presents your project or technology you might use in a bad light, but for example Script.NET is slow in comparison (this is explained below in more detail), but still fast enough for probably 99% of its uses. It still took much longer than I expected, writing this article did take some time too, but most of the time was spend writing and testing code in each language and technique and then gathering the execution time results. But performance evaluation is always fun :) Again: This is not a representative performance comparison of the languages and techniques involved, it only applies to my very short and stupid test code and should only give you a general idea.

C# 3.0

First of all the very simple implementation of the while loop:

public static double DoLoop(int loops)
{
    double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0;
    }
    return counter;
} // DoLoop(loops)

And now some code to measure the execution time and to print out the result to see if the loop was executed as many times as we expected:

    long startTime = WindowsHelper.GetPerformanceCounter();
    
    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    // Note: We did printing in other tests too (does not matter anyway)!
    Console.WriteLine("DoLoop result=" + result);
    
    long endTime = WindowsHelper.GetPerformanceCounter();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + WindowsHelper.ConvertToTime(endTime - startTime));

Note that the WindowsHelper class is just providing a little bit more accurate information than just using a StopWatch (which is used in most other samples here), which is fine too. With System.Diagnostics.StopWatch this code looks like the following (actually results in the same results on Windows 7 and Vista):

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    //Note: We did printing in other tests too (does not matter much anyway)!
    Console.WriteLine("DoLoop result=" + result);

    timer.Stop();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + timer.ElapsedMilliseconds + "ms");

The following output is generated:

DoLoop result=10000000
Executing the while loop 10000000 times took: 37ms

Native C++

Here is the whole program used for testing. This has nothing to do with the DLR or .NET anymore, I just wanted to know how my little while loop would perform in native C++ and Assembler. Sometimes when using for loops the C++ compiler will optimize the code away and just replace it with something static that results in the same value as if the for loop would have been executed. Please also note that it did not make any difference to use a 64bit platform and doubles or just 32bit floats, but since most script languages and especially .NET compile to x64 too, we should not use 64bit numbers when compiling a 32bit application here in C++. Switching to x64 is also not too hard, you just need the correct 64bit libraries and set the linker target platform to X64.

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    // C++ version
    int loops = 10000000;
    float counter = 0.0f;
    //64bit code: double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0f;
    }
    
    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    // Wait for user input (not required for Debug.StartWithoutDebugging)
    //char input[128];
    //scanf(input);

    return 0;
}

And after compiling (usually a lot faster than 1s when just incrementally compiling) and running this code, it generates this output:

counter=10000000.000000
This took 31ms

The result is not very accurate, I either get 31ms or 47ms, but not values in between, the accuracy of GetTickCount pretty much sucks and I should use similar code as the GetPerformanceCounter() stuff above from C#, but I do not really care about more accurate values here, C++ is more than fast enough and we do not need to discuss its runtime performance!

Assembler

Based on the C++ solution I was able to write some Assembler instructions. It has been a very long time since I used assembler code at all, probably in 2002 when optimizing some path finding algorithms for ArenaWars, which were called from C++ code, which was called from C# code. Before that in the C++ days I used it now and then, but never for more than some performance critical inner loops. Please also note that sometimes you make non-optimal assembler code (as I probably did in this example) and just having your C++ compiler generate assembler instructions can be better, so always profile your code and only optimize to Assembler if you have no other option left and if you are too lazy to rethink the problem :D

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();
    
    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
startLoop:
        // counter += 1.0f
        fld counter
        fadd counterInc
        fstp counter

        // loops--
        dec loops

        // loops > 0? 
        cmp loops, 0
        // Then goto startLoop (else we are done)
        jg startLoop
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

I don't have much to say about this ugly assembler code, as you can see it is a lot more lines of code and this was the first idea not even using registers as you should in assembler. After running this code and getting 31-47ms too (like the C++ version), I added some minor optimization by using registers more explicitly and this is the resulting code, it ran about 2-3 faster than the above code:

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
        // Copy loop counter to eax register (32bit)
        mov eax, loops
        // Load both float values
        fld counterInc
        fld counter
        
startLoop:
        // counter += 1.0f
        fadd st(0), st(1)
        //st(0) has still the counter value! no need to store: fst st(0)

        // loops--
        dec eax

        // loops > 0? 
        cmp eax, 0
        // Then goto startLoop (else we are done)
        jg startLoop

        // We are done, copy counter value back!
        fstp counter
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

This program does produce the following output:

counter=10000000.000000
This took 15ms

This is pretty good and I did not want to optimize it further, especially not when the timer is so inaccurate. Running the code 10 times resulted in 140ms. Again, fast enough, no reason to argue the superior runtime performance. But this comes at a high price since the code is in no way easy to understand, its hard to write and read.

IronPython 2.6

Speaking of code, I would say the implementation in Python is the shortest and very beautiful since it is so easy to read and write:

loops = 10000000
counter = 0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

I just used my ScriptManager from my .NET libraries, which can handle IronPython (and other DLR languages) as well as Lua and wrote this short unit test to see how IronPython performs:

    public void TestPythonDoLoop()
    {
        // Create script manager without any output redirection or special directories
        ScriptManager manager = new ScriptManager(null, null);
        
        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Simply execute line by line as we would in a console window
        manager.ExecuteCode("loops = 10000000", ScriptType.Python);
        manager.ExecuteCode("counter = 0", ScriptType.Python);
        manager.ExecuteCode(@"while loops > 0:
loops = loops - 1
counter = counter + 1", ScriptType.Python);
        manager.ExecuteCode("print counter", ScriptType.Python);

        timer.Stop();
        Console.WriteLine(
            "Executing the while loop 10000000 times in python took: " +
            timer.ElapsedMilliseconds + "ms");
    } // TestPythonDoLoop()

And this results in the following output:

Executing the while loop 10000000 times in python took: 9151ms

Update 2009-04-21: As Simon Davy correctly states in the comments of this post, ipy.exe (IronPython) is actually a lot faster when executing a file with the code above (e.g. use import time). This results in an execution time of just 1250ms, so a lot faster! I have tested it too with ipy, but I entered the code line by line (same way as I called my ScriptManager code in the unit test above) and it still takes 9 seconds. But Simon is right, once I execute everything at once (with ipy test.py) it only takes 1.25s (can also be done with my ScriptManager, but I have not thought of that). So it is probably just an issue with REPL (entering statements one by one .. less optimizations that way). While this certainly does make IronPython a much faster dynamic language than all the other ones, it still is slower than the compiled languages or the DLR code at the end of this article.

Thanks for noticing!This is certainly not good. The same thing happend after I tried using the IronPython 2.6 console or IronPython 2.0, it always took about 9 seconds to complete the big while loop on my PC. I wanted to figure out why this is so slow compared to C#, C++ or Assembler runtime, but then again we have to remember that Python is a dynamic language. The most important reason for the long runtime here is probably just all the boxing and unboxing of values, if you can say that for dynamic languages. What I mean is figuring out which type we got, and then performing the add/sub operation, which can be slow if you do it 10 million times. Please note again that this is probably not affecting any real Python program and performance for more complex programs will be way better. More on the type-performance and this issue in general at the end of this article!

ToyScript

To learn more about the DLR I played around with the DLR ToyScript sample quite a bit the last days. Since a lot of its code is based on the same ideas as IronPython and IronRuby, you can't expect much different performance results. This is the ToyScript code for the while loop:

loops = 10000000
counter = 0
while (loops > 0)
{
    loops = loops - 1
    counter = counter + 1
}
print counter

Similar to the ScriptManager above I wrote some helper classes for the ToyScript language allowing me just to execute code, viewing the AST (abstract syntax tree) and write a lot of unit tests. The following code is such a unit test for the ToyScript class to check the performance for these 10 million while loop iterations:

    public void TestBigLoop()
    {
        const int NumberOfLoops = 10000000;

        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Initialize the language first (takes its own time, around 400ms)
        ToyLanguage language = new ToyLanguage();
        // Execute something else (takes maybe 100ms)
        Assert.Equal(3.0, language.Execute("print 1+2"));

        timer.Stop();
        Console.WriteLine("Startup time: " + timer.ElapsedMilliseconds + "ms");
        timer.Reset();
        timer.Start();

        Assert.Equal((double)NumberOfLoops,
            language.Execute(
@"loops = " + NumberOfLoops + @"
counter = 0
while (loops > 0)
{
loops = loops - 1
counter = counter + 1
}
print counter
"));

        timer.Stop();
        // Show total execution time
        Console.WriteLine("Executing the while loop " + NumberOfLoops +
            " times took: " + timer.ElapsedMilliseconds+"ms");
    } // TestBigLoop()

This code runs for about the same time as IronPython, maybe a little faster since ToyScript has a lot less features than IronPython, but in general it uses the same approach. This is the output from the unit test:

Startup time: 650ms
Executing the while loop 10000000 times took: 8914ms

Investigating the Lambda Expression AST for the code block shows that quite a lot of statements were created in order to make this ToyScript code run on the DLR:

 // Expression: Expression`1
 
 .lambda (<toyblock>$2 Microsoft.Func`1[System.Object] #1)
 
 // LambdaExpression: <toyblock>$2(1)
 .lambda System.Object <toyblock>$2 ()(
 ) {
   .label 0x03d11d12:
   .comma  {
     .block  {
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)10000000)
       ,
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)0)
       ,
       .loop break:.label 0x03fe2d21  {
         .block  {
           .if (
             .site (Boolean) CallSiteBinder(ConvertTo to System.Boolean) (
               .extension CodeContextExpression ( )
               ,
               .site (Object) CallSiteBinder(DoOperation GreaterThan) (
                 .extension CodeContextExpression ( )
                 ,
                 .extension GlobalVariableExpression ( )
                 ,
                 0
               )
             )
           ) {
             /*empty*/
           } .else {
             .block  {
               /*empty*/,
               .break .label 0x03fe2d21
             }
           }
           ,
           .block  {
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             .site (Object) CallSiteBinder(DoOperation Subtract) (
               .extension CodeContextExpression ( )
               ,
               .extension GlobalVariableExpression ( )
               ,
               1
             ))
             ,
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             'ToyHelpers.Add'(
               (Object).extension GlobalVariableExpression ( )
               ,
               (Object)1
             ))
             ,
             /*empty*/
           },
           /*empty*/,
           /*empty*/
         }
       },
       'ToyHelpers.Print'(
         .extension CodeContextExpression ( )
         ,
         .extension GlobalVariableExpression ( )
       ),
       /*empty*/
     },
     .null
   }
 }

The IL code was even more confusing so I only tried to figure out the simple "print 1+2" expression IL code that can be produced by calling ScriptCode.SaveToAssembly(..), more DLR IL code analysis will happen at the end of this article:

  .method public specialname static object <toyblock>$1$1(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope $scope,
 class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext $language) cil managed
  {
    .maxstack 3
    .locals init (
      [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context)
    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
    L_0007: stloc.0 
    L_0008: ldloc.0 
    L_0009: ldc.r8 1
    L_0012: box float64
    L_0017: ldc.r8 2
    L_0020: box float64
    L_0025: call object ToyScript.ToyHelpers::Add(object, object)
    L_002a: call void ToyScript.ToyHelpers::Print(class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext, object)
    L_002f: ldnull 
    L_0030: ret 
  }

There is a lot of extra stuff in there, but you can clearly see the 1+2 in L_0009 to L_00020 with all its boxing fun. Then those two are added with help of a method Add that is defined in the ToyHelpers class (unboxing the doubles again, adding and boxing them to an object again, which is returned). Then the result is passed along to the ToyHelpers.Print method, which just calls Console.WriteLine in my case (usually it is used by the ConsoleHost class for the normal REPL console interaction).

I am not a DLR expert and there is probably a lot of other things going on, but to me all this boxing, unboxing and extra code just to add some numbers is clearly too much for the big loop. At the end of this article I will try to write some DLR code that does less boxing/unboxing.

Python 2.6

After testing the DLR it was time to check out the native Python implementations to see if this is releated to the dynamic nature of dynamic languages like Python or Lua. Well it obviously is related, but more specifically I wanted to see if certain implementaions are maybe more clever or optimized.

The Python code is the same as above for IronPython:

loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

Executing this takes about 3.5 seconds, we could cheat a little and use the following code to execute this loop a little faster:

counter = 0.0
for loops in range(0, 10000000):
    counter = counter + 1.0
print counter

This runs about 2 times faster (less than 2 seconds) because Python is clever and optimizes for loops. The same actually applies to IronPython, a for loop is much faster here too (5.5s instead of 9.1s). But this defeats the whole point of this exercise, I wanted to know what is slow about the DLR and implementations like IronPython or ToyScript and why. It is nice that Python executes for loops quicker, but the whole counter = counter + 1 thing is still very slow compared to C#, C++, etc.

Python 3.1

Executing Python 3.1 is pretty much the same as Python 2.6. It is slighly slower, but that can also be because I use a alpha build (3.1 alpha 2 from 2009-04-04). Again, executing the for-loop version with range was twice as fast (2.1s instead of 4.3s). In general is is quite known that Python is not one of the fastest languages, there are many dynamic languages that can be faster (for example Ruby or Lua), but there are also much slower ones like PHP. If you are more interested about comparing IronPython with Python read the performance comparisons at the IronPython project at codeplex: http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Performance

Lua 5.1.4

The lua code is pretty similar to Python, but it executes much faster:

loops = 10000000
counter = 0.0
while (loops > 0) do
    loops = loops-1
    counter = counter+1
end
print(counter)

This executes in less than 1.5 seconds and returns the expected 10000000 (not 10000000.0 as Python however since Lua treats everything as a 64bit double anyway). The for-loop trick works here too and reduces the time in half (0.7s instead of 1.4s):

counter = 0.0
for i=1,10000000 do
    counter = counter+1.0
end
print(counter)

The reason why Lua is much faster than all the other dynamic languages I have tested is probably because Lua has much simpler data types and is optimized a lot for math, for loops and stuff like that. I use Lua more than Python these days and the performance is quite good I have to say (using it for AI, some physics and some game logic). As you can see here in many benchmarks Lua is several times faster than Python (but Python is still a lot more powerful with the huge amount of libraries available for it).

Script.NET

Script.NET is a great project that uses the Irony compiler toolkit to parse and evaluate simple C# code. It does not generate any IL or immediate code, it just evaluates the AST directly. This is a very nice approach and easy to understand because all the code is C#, but there is a slight problem with it: This approach is it VERY slow. It was optimized many times already and runs much faster than a year ago, but I would not recommend it for anything other than some scripting in C# (guess what, that's the whole name of the project). If you do not need high performance Script.NET is just fine for executing a couple lines of code, even for games as long as you do not do any physics or rendering code with it ^^

loops = 10000000;
counter = 0;
while (loops > 0)
{
    loops = loops-1;
    counter = counter+1;
}
Console.WriteLine(counter);

The output appears after almost 40 seconds and displays correctly:

10000000

At least I now know why generating some IL code is very important, with or without the DLR. Next I will investigate IL code generation with Reflection.Emit, but I can still remember the pain from the last time I tried to do something with it.

IL Emit

So let's get right to it, the following code will generate the required IL for us:

    // Create a dynamic method called DoLoop where we put all our IL code
    DynamicMethod DoLoop = new DynamicMethod("DoLoop",
        typeof(double), null, typeof(string).Module);

    // Start a timer once again to measure creation time and execution time!
    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Get an ILGenerator and emit a body for our dynamic method
    ILGenerator il = DoLoop.GetILGenerator();
    
    // Create counter (double)
    LocalBuilder counter = il.DeclareLocal(typeof(double));
    il.Emit(OpCodes.Ldc_R8, 0.0);
    il.Emit(OpCodes.Stloc, counter);
    // Assign loops variable to 10000000 (10 million) for the loop!
    LocalBuilder loops = il.DeclareLocal(typeof(int));
    il.Emit(OpCodes.Ldc_I4, 10000000);
    il.Emit(OpCodes.Stloc, loops);
    
    // Start loop label
    Label startLabel = il.DefineLabel();
    il.MarkLabel(startLabel);
    
    // Do this 10000000 times:    
    // Increase counter by 1.0
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ldc_R8, 1.0);
    il.Emit(OpCodes.Add);
    il.Emit(OpCodes.Stloc, counter);

    // Decrease loops by 1
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 1);
    il.Emit(OpCodes.Sub);
    il.Emit(OpCodes.Stloc, loops);

    // Check if we still have to loop (as long as loops > 0)
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 0);
    il.Emit(OpCodes.Bgt, startLabel);

    // End loop and return counter!
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ret);

    // Create a delegate that represents the dynamic method. This
    // action completes the method, and any further attempts to
    // change the method will cause an exception.
    DoLoopDelegate loop = (DoLoopDelegate)
        DoLoop.CreateDelegate(typeof(DoLoopDelegate));

    timer.Stop();
    Console.WriteLine("DoLoop IL creation time: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();

This creates the dynamic DoLoop method, which we can call with loop() from here on, which will also return the result (counter) once we call it. This all takes almost no time at all:

DoLoop IL creation time: 0ms

Next, we need to invoke this dynamic method, which is rather easy:

    timer.Start();
    Console.WriteLine("Invoking DoLoop() returned: " + loop());
    timer.Stop();
    Console.WriteLine("DoLoop IL execution time: " + timer.ElapsedMilliseconds + "ms");

And this only takes about 16ms and also returns the correct value:

Invoking DoLoop() returned: 10000000
DoLoop IL execution time: 16ms

So IL code is rather fast (faster than everything else here except the fine-tuned assembler, which is similar to this), but don't ask in how many problems I ran into just writing these few lines of code. Let's just say a lot of times I used wrong opcodes, did not know how to implement stuff and finally when stuff compiled I got syntax errors when executing, not really telling me what I did wrong, so I had to comment out code until it worked again and then slowly added code until I could figure out what had to be changed. Let's just say not a pleasant thing to write a compiler this way.

Irony + DLR (dummy)

As mentioned before Script.NET actually uses the Irony compiler toolkit for parsing and building the AST (abstract syntax tree). Irony itself is rather fast (can parse 15000 lines of code or more per second), only the execution part of Script.NET wasn't so good because no code is generated, everything is evaluated on the fly.

While I have started using Irony just a week ago after playing around with ANTLR first (see earlier blog post here), but I still like it very much and I hope Roman Ivantsov (the guy behind Irony) will soon release a long promised update with some cool new features I can sink my teeth into. As you will see in my sample below I have added some own functionality to Irony already to make it easier to test code and I will probably extend it even more when I write more of my language grammar.

If you don't know how Irony works and you are interested in building compiler grammars directly in C# you should check out the project and the Lang.NET talks Roman did this and last year about it, plus the great Irony article on CodeProject of course. Basically you define some non-terminals (variables, numbers) and terminals (logical blocks of your code) in the constructor of a Irony.Grammar derived class and then define some rules for it (in BNF = Backus-Naur Form). Then you set the root node and you are ready to parse some code. Please note that this grammar is very simple and will only handle this sample code and very similar code blocks. I'm still just playing around with grammars, there is no concrete implementation of my language yet, just a lot of ideas .. Writing some grammars is a good training, I have written one in ANTLR and 4 other more complex ones with Irony yet.

    public TestDoLoopGrammar()
    {
        // 1. Terminals
        var variable = new IdentifierTerminal("variable");
        var number = new NumberLiteral("number");

        // 2. Non-terminals
        var program = new NonTerminal("program");
        //var commandList = new NonTerminal("commandList");
        var command = new NonTerminal("command");
        var assignmentOperator = new NonTerminal("assignmentOperator");
        var assignment = new NonTerminal("assignment");
        var whileLoop = new NonTerminal("whileLoop");
        var compareOperator = new NonTerminal("compareOperator");
        var condition = new NonTerminal("condition");
        var binaryOperator = new NonTerminal("binaryOperator");
        var operation = new NonTerminal("operation");

        // 3. BNF rules
        // Please note that this program grammar is rather limited and not really
        // useful, it is only used for testing and learning Irony ..
        // We can have many commands in case we want to go crazy
        program.Rule = MakePlusRule(program, null, command);
        assignmentOperator.Rule = Symbol("=");
        assignment.Rule = variable + assignmentOperator + number;
        whileLoop.Rule = Symbol("while") + condition + ":" +
            program + "end";
        // We only allow some simple compare conditions
        compareOperator.Rule = Symbol("<") | "==" | ">";
        condition.Rule = variable + compareOperator + number;
        // We can either use + or -
        binaryOperator.Rule = Symbol("+") | "-";
        operation.Rule =
            variable + assignmentOperator + variable + binaryOperator + number;
        // Commands can be assignments, while loops or operations
        command.Rule = assignment | whileLoop | operation;

        // 4. Operators precedence and punctuation
        RegisterPunctuation(":", "end");

        // 5. Global settings
        this.CaseSensitive = false;
        this.ThrowGrammarExceptionsOnError = true;

        // 6. Set grammar root
        this.Root = program;
    } // TestDoLoopGrammar()

Note that ThrowGrammarExceptionsOnError is not a property of Irony, I added it to make sure all Grammar errors immediatly cause an exception to be thrown, which makes it much easier for unit tests and figuring out what when wrong. Not really sure why Irony does not do that, I can understand not throwing exceptions for syntax errors when parsing some source code (because that might be slower than just reporting errors in a list form and more helpful because its easier to continue after the error). But grammar errors mean usually that you cannot use the grammar for anything anyway. Some things also just produce a warning, but I throw Exceptions then too because I don't want hidden fixes of my grammar, which would make it hard to fix real problems later.

Anyway, with the TestDoLoopGrammar class we can now test and parse some code. Irony usually expects you to use its Grammar Explorer program to do this job, which is a pretty nice application, but it is not unit test and we all know I'm all about unit testing (don't we?). I still use the Irony Grammar Explorer from time to time and I recommend it very much if you want to test out grammars and visualize the AST a little better than just throwing some text into a console window. The following simple unit test will parse our while-loop program source code and produce a nice AST for us, which it kindly prints out:

    public void GenerateDoLoopAST()
    {
        AstNode program = IronyHelpers.ParseWithErrorHandling(
            new TestDoLoopGrammar(),
@"loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
end");

        IronyHelpers.PrintAst(program);
    }

This will quickly print out the AST in the following way, which might seem a little complex and verbose at first, but you have to remember that we do not have to generate code for every single node. Often several nodes together will just be one single instruction. Other nodes like program are just used to hold code blocks together and allow several child nodes.

 program
   assignment
     loops [variable]
     assignmentOperator
       = [Symbol]
     10000000 [number]
   assignment
     counter [variable]
     assignmentOperator
       = [Symbol]
     0 [number]
   whileLoop
     while [variable]
     condition
       loops [variable]
       compareOperator
         > [Symbol]
       0 [number]
     program
       operation
         loops [variable]
         assignmentOperator
           = [Symbol]
         loops [variable]
         binaryOperator
           - [Symbol]
         1 [number]
       operation
         counter [variable]
         assignmentOperator
           = [Symbol]
         counter [variable]
         binaryOperator
           + [Symbol]
         1 [number]
     end [variable]

Now this tree has to be translated to some code. We could use Reflection.Emit for this, but we just saw how much work it was for this very simple sample. Thats the reason I'm using the DLR instead, which also gives my language some cool dynamic capabilities for free (plus the other benefits like interacting with other DLR languages). But since the DLR is not a piece of cake and I'm not done exploring it yet (just started last two week with it, gimme more time), I will not present a full project with this language here yet (totally out of the scope of this article). I used a modified version of ToyScript and some other sample projects (following next) to generate some DLR code here, but it is still hacked up very much. I will probably write another long article when I'm done with that and have some cool results to show.

DLR AST with objects

Okay, so let's dig deeper into the DLR. While I got some parts of the above Irony AST working in the DLR with help of the ToyScript sample, the whole DLR framework, the ToyScript sample and especially IronPython is still very hard to understand and its also kinda annoying that recent code drops do not work at all with ToyScript. They compile, but running ToyScript will just produce funny errors like "node not reducible", which is not very helpful if you just want to learn about the DLR. A lot of stuff in the DLR still constantly changes (just see the huge amount of code drops at dlr.codeplex.com). I totally like that the DLR team releases all this wonderful code and does it so often, but without any working documentation (almost everything you find on the internet is based on DLR versions from a year ago, when stuff worked in a totally different way) and not even the included samples working all the time is kinda painful. So what does a programmer do in such a case? Yes, he writes tests and figures it out on his own. If there were not this many classes that all do almost nothing in functionality (what the hell, in my projects each class has many hundred lines of code, but in the DLR it seems like 50% of all classes are just 10-30 lines long) it would a much easier job. Debugging is also much harder because stack-traces become 2 miles long after just calling one of two actual functions, that do not just call other functions. Okay, enough ranting, let's just hope these things get better in the final 1.0 release since Microsoft is not allowing any contributions from the community to the DLR and I certainly do not want to write my own. It is pretty amazing what the DLR can do and how well it performs, it just has to become easier to use. Someone had to say it :)

So the bare minimum you need for the DLR to compile some AST for you is a LanguageContext and a DefaultBinder, define one class of each and derive it accordingly. Then define a constructor for your language context, set the binder and while you are at it add the default System (mscorlib) assembly to the script domain manager. Note this code uses DLR change set 21259 (from Apr 5 2009), but I could also use the more recent 22459 version (from today), but I reverted to make the ToyScript sample work again.

    public YourLanguageContext(ScriptDomainManager manager,
        IDictionary<string, object> options)
        : base(manager)
    {
        Binder = new YourBinder(manager);
        manager.LoadAssembly(typeof(string).Assembly);
    } // YourLanguageContext(manager, options)

Next you need to provide an implementation of ParseSourceCode to make the compiler happy and to actually allow your program to parse some source code in your language:

    protected override ScriptCode CompileSourceCode(SourceUnit sourceUnit,
        CompilerOptions options, ErrorSink errorSink)
    {
        // Create lambda expression
        LambdaBuilder program = AstUtils.Lambda(typeof(object), "YourLanguage");
        program.Body = Expression.Block(GenerateStatements(program, sourceUnit.GetCode()));
        LambdaExpression ast = program.MakeLambda();
    
        Expression<DlrMainCallTarget> globalAst =
            new GlobalLookupRewriter().RewriteLambda(ast);
    
        return new LegacyScriptCode(globalAst, sourceUnit);
} // ParseSourceCode(context)

The interesting thing here is how we build an DLR AST from statements (Expression.Block) with help of the LambdaBuilder. The statements do not exist yet, that's what the GenerateStatements method will be used for. We pass the source code to this method and let its do its magic (this is a reduced version, the actual unit test uses a more complex version with some extra code paths and statements to make sure we can initialize the DLR first and to generate code for the next technique too):

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(object), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Convert(
                    Expression.Constant(0.0), typeof(object))
            )
        );
        Expression loops = program.Variable(typeof(object), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Convert(
                    Expression.Constant((int)10000000), typeof(object))
            )
        );
        
        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    Expression.Convert(loops, typeof(int)),
                    Expression.Constant(0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Convert(
                            Expression.Add(
                                Expression.Convert(counter, typeof(double)),
                                Expression.Constant(1.0)
                            ), typeof(object))),
                    Expression.Assign(loops,
                        Expression.Convert(
                            Expression.Subtract(
                                Expression.Convert(loops, typeof(int)),
                                AstUtils.Constant(1)
                            ), typeof(object)))
                ),
                AstUtils.Empty()
            )
        );
        
        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(object) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

This is obviously not dynamic code, it just adds a bunch of expression statements to a list and returns it. Please also note that I tried to use object for counter and loops to demonstrate this in a similar way ToyScript or IronPython will generate code (thats how I figured out some of these expressions). But since .NET or the DLR does not know how to add two objects for example and does not even allow to assign a number to an object, we have to do a lot of casting, which is in essense just boxing and unboxing.

This code is then executed by a unit test, which does basically create a ScriptRuntime for us, makes sure it uses this language implementation and then calls engine.CreateScriptSourceFromString with the code to be executed. Please note that GenerateStatements completely ignores the source code, it will always generate the same statements right now.

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Create runtime
    ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
    Type provider = typeof(YourNamespace.YourLanguageContext);
    setup.LanguageSetups.Add(
        new LanguageSetup(provider.AssemblyQualifiedName, provider.Name));
    ScriptRuntime runtime = new ScriptRuntime(setup);

    // Load Engine
    ScriptEngine engine =
        runtime.GetEngineByTypeName(provider.AssemblyQualifiedName);
    timer.Stop();
    Console.WriteLine("Create DLR runtime: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();
    timer.Start();

    // Execute command
    ScriptSource code = engine.CreateScriptSourceFromString("Objects");
    code.Execute();

    timer.Stop();
    Console.WriteLine("Execute code: " + timer.ElapsedMilliseconds + "ms");

This test will spend about 430ms in the DLR runtime creation (which is kinda slow and will hopefully be faster in the future) and that is the reason it is checked separately. Then it will spend a few ms for the statement creation (4ms as mentioned in the table above) and finally it will execute the code in about 459ms, which is a lot faster than the other DLR languages. But then again, we cheated quite a bit by providing the exact statements we need. The only thing we can optimize is to get rid of all those boxing and unboxing actions. While it is nice to be under half a second for these 10 million while loop iterations with one add to counter and one subtract from loops statement each, it is still a far cry from the performance we have seen in the compiled languages. Let's check out the IL code generated from the DLR AST to see whats going on here:

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] object obj2,
    [2] object obj3)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: box float64
  L_0016: stloc.1 
  L_0017: ldc.i4 0x989680
  L_001c: box int32
  L_0021: stloc.2 
  L_0022: ldloc.2 
  L_0023: unbox.any int32
  L_0028: ldc.i4.0 
  L_0029: cgt 
  L_002b: brfalse L_0035
  L_0030: br L_003a
  L_0035: br L_0063
  L_003a: ldloc.1 
  L_003b: unbox.any float64
  L_0040: ldc.r8 1
  L_0049: add 
  L_004a: box float64
  L_004f: stloc.1 
  L_0050: ldloc.2 
  L_0051: unbox.any int32
  L_0056: ldc.i4.1 
  L_0057: sub 
  L_0058: box int32
  L_005d: stloc.2 
  L_005e: br L_0022
  L_0063: ldloc.1 
  L_0064: call void [mscorlib]System.Console::WriteLine(object)
  L_0069: ldnull 
  L_006a: ret 
}

It might be a little hard to figure out which line does what, but once you found the constants like 0x989680 (which is just 10000000) and the add and sub operations, you can get an idea how the statements were translated. The code between L_0035 and L_0063 is the inner loop and the performance critical code. There are 2 unboxing and 2 boxing operations in that code and probably some other unnecessary assignments.

DLR AST with doubles

While playing around with the code even more and getting to a version like the one that will be shown in this section, I tried to understand why the DLR did certain things in the way that it does. A lot of dynamic behavior is only possible when you use very generic types, but I still think for my language implementation it should be possible to use a much more strict rule set and only allow dynamic code where necessary or wanted, not in thinks like loops or math, where we deal with numbers anyway. Lua is a great example of that, numbers are just numbers. They are always doubles, so there is no need to box or unbox them all the time, which is probably one of the reasons it can handle math better than other dynamic languages.

I also found this useful explanation of what IL the DLR emits for dynamic code, it could explain why stuff is slower than IL code, but still a lot faster than evaluating everything ever time. The DLR does a great job of caching and optimizing dynamic code, but as you can see the IL generated is quite long and verbose too, which is ok for dynamic code I guess. I'm just interested in ways to optimize this further to at least provide proof that the DLR is in fact just as good as a compiler for more static languages.

After the last section it is quite obvious what we have to do do generate statements that can be executed quicker: Get rid of all that boxing and unboxing, use specific data types and make sure all the assignments use the correct format too (else convert constants or other variables). This should not be too hard to do in an actual language once we figure out that we are only dealing with numbers (similar to Lua we could only allow one kind of number, like doubles). This is the updated code for GenerateStatements:

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(double), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Constant(0.0)
            )
        );
        Expression loops = program.Variable(typeof(double), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Constant(10000000.0)
            )
        );

        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    loops,
                    Expression.Constant(0.0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Add(counter,
                            Expression.Constant(1.0))),
                    Expression.Assign(loops,
                        Expression.Subtract(loops,
                            AstUtils.Constant(1.0)))
                ),
                AstUtils.Empty()
            )
        );

        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(double) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

BTW: To analyze the IL code I use the ScriptCode.SaveToAssembly method and use Reflector to show the IL code. As you can see the code has become shorter and we can't see any signs of boxing and unboxing anymore. The code is not even optimal because we use doubles for everything (remember that the C#, C++ and Assembler samples all used a simple integer for the while loop), but as the execution time with less than 100ms shows us, the performance is quite good with this approach.

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] float64 num,
    [2] float64 num2)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: stloc.1 
  L_0012: ldc.r8 10000000
  L_001b: stloc.2 
  L_001c: ldloc.2 
  L_001d: ldc.r8 0
  L_0026: cgt 
  L_0028: brfalse L_0032
  L_002d: br L_0037
  L_0032: br L_0054
  L_0037: ldloc.1 
  L_0038: ldc.r8 1
  L_0041: add 
  L_0042: stloc.1 
  L_0043: ldloc.2 
  L_0044: ldc.r8 1
  L_004d: sub 
  L_004e: stloc.2 
  L_004f: br L_001c
  L_0054: ldloc.1 
  L_0055: call void [mscorlib]System.Console::WriteLine(float64)
  L_005a: ldnull 
  L_005b: ret 
}

This is a more complex version of the above unit test from last section, the 10000000 is printed by the DoLoop DLR code:

runtime creation: 336ms
get engine: 16ms
create script: 4ms
Init code and first script startup time: 232ms
10000000
execute again (DoLoop code): 98ms
and shutdown: 0ms

Okay, I would say this blog post is long enough now and I wrote all day on it and the code samples presented here. Could even be the longest blog post on my blog (html is 150kb). Who knows .. it is probably a good idea to go to bed now.

Good fight, good night, till next time :)

I have been working quite a bit in Python and Lua last year and I also tried to use those languages in our current game project at exDream (will be announced shortly, so I'm finally allowed to talk about it ^^). While both Python and Lua are written in C (there are obviously other ports like the great IronPython project for .NET, but I'm not going to talk about that in this post), it was not easy at all to make Python run on the Xbox 360 or PS3. I was able in early testing to get some parts running on the Xbox 360, but too much was missed and commented out and the PS3 port looked like way too much work. So I decided to just use Lua, which is probably a better choice for our game anyway since I only use it to do some AI and physics code with it along with some settings. I'm not going to benefit from the Python libraries either way, they are not needed for our game and they won't run on those consoles without serious effort anyway.

Ok, so I was writing some Lua code and used mostly LuaInterface to make the Lua code run from .NET and of course to work with all my .NET code and even XNA. The second step was to make this run in our new game engine, which is written in native code (C++) since the whole engine is based on the Unreal 3 Engine. For that I used the normal Lua 5.1.4 release, which compiles just fine and works great. Since Lua is written completely in ANSI C I was able to quickly test some code on the Xbox 360 and PS3 too. This is important because our game will be a multi-platform title (PC, Xbox 360 and PS3), which is thanks to the Unreal 3 Engine a much simpler task than writing our own engine for all those platforms (it has obviously it own disadvantages, not only do we have to use native code, but we also have to work with a lot of code in the engine that we don't really need for our game).

So far so good. While it sounds like a simple task to make Lua run on all platforms, there is a lot of pain involved to make it run perfectly and behave the same way on all platforms. Since the Unreal 3 Engine is written in C++ and uses its own scripting Language UnrealScript, it does not help out a lot when you want to integrate your own code or libraries in there, so this was half of the work (just see my previous post about compile times and you see why this is a much lengthier process than writing some .NET code). Secondly there are many special rules on both the Xbox 360 platform and the PlayStation 3, some ANSI C methods are just missing or just don't do anything because of the platform design. For example on the Xbox 360 you can't use relative paths, you always have to specify a full path when opening a file, which is something that the Lua engine does not do out of the box. After figuring out where to store the Lua scripts on the consoles and how to load them, everything else just falls into place and works finally as expected.

To make things simpler for other people trying to run Lua code on the Xbox 360 or PS3 I wrote the following little solution:

• LuaXbox360PS3PC.zip (241.92 KB)
  

As you can see on the right the solution consists of 4 projects:

• LuaSupport, which is based on the current Lua 5.1.4 release from 22 Aug 2008 with my own additions to support the XBOX and PS3 platforms (uses XBOX and PS3 defines).
  
  
• TestLuaPC is a simple test project for the PC to make sure everything still runs as expected on the PC. Please note that on the PC Lua assumes that all files can be found in the current directory. I just change to the Lua script directory and then execute the Lua files there normally.
  
  
• TestLuaPS3 will probably only compile if you have the PS3 SDK installed, just unload this project if you don't have it installed yet. Also please test other SDK samples first to make sure everything is correctly setup (it uses a lot of extra include paths). In case you are not able to open the files, check for yourself if the /app_home/Lua/ directory contains the required files after building the project (use the ProDG Target Manager for that). Debugging can also be complicated, I guess it does not work on Windows 7 at all, but I was able to debug and run the app on a normal Windows Vista PC. In any case you can also just run the .elf file from the ProDG Target Manager and then see the output in the console.
  
  
• TestLuaXbox360 is the project to test Lua on the Xbox 360. You will also need the XDK (Xbox Development Kit SDK) installed and have a Xbox 360 DevKit in your network for testing. A TestKit also works, but you won't be able to debug, you can just run the app and check the console output yourself.
  
  

Some of the Lua functionality is not available on the Xbox 360 or PS3 like lua_popen or lua_pclose (among others), and there is also some code that will not do anything like os_execute, os_getenv and some other os methods because they won't make sense on a console anyway or could not be implemented. I did not use these methods anyway, but if you need those and want to write your own code for them, go ahead.

Lets focus on the 2 most important code changes. First of all the Lua default directories had to be adjusted in luaconf.h:83 to support Xbox 360, PS3, PC and Linux platforms:

#if defined(XBOX)
/* On the Xbox we just use the D:\ drive for everything, all lua files should be
   located there */
#define LUA_LDIR    "D:\\Lua\\"
#define LUA_CDIR    "D:\\"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(PS3)
/* On the PS3 we just use the /app_home/Lua/ sub directory for everything,
   all Lua files should be located there */
#define LUA_LDIR    "/app_home/Lua/"
#define LUA_CDIR    "/app_home/"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(_WIN32)
/*
** In Windows, any exclamation mark ('!') in the path is replaced by the
** path of the directory of the executable file of the current process.
** Note: We use the ..\Lua\ directory by default, but that is also the current
** directory, we change it in AIEngineInterface before running scripts!
*/
#define LUA_LDIR    "!\\..\\Lua\\"
#define LUA_CDIR    "!\\"
#define LUA_PATH_DEFAULT  \
        ".\\?.lua;"  LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua;" \
        LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#else
/* Default linux/mac directories */
#define LUA_ROOT    "/usr/local/"
#define LUA_LDIR    LUA_ROOT "share/lua/5.1/"
#define LUA_CDIR    LUA_ROOT "lib/lua/5.1/"
#define LUA_PATH_DEFAULT  \
        "./?.lua;"  LUA_LDIR"?.lua;"  LUA_LDIR"?/init.lua;" \
                    LUA_CDIR"?.lua;"  LUA_CDIR"?/init.lua"
#define LUA_CPATH_DEFAULT \
    "./?.so;"  LUA_CDIR"?.so;" LUA_CDIR"loadall.so"
#endif

And the most important code change is located in luaL_loadfile method, which is called by the lua_dofile macro. Please note that I did not change the code formating and tried to leave as much code as is. I have my own coding conventions, but it does not make sense to apply them when only changing <10% of the code of an existing library.

LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  char fullFilename[128];
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  lf.extraline = 0;
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    // Note: Always open as the requested file, Lua should not care about
    // our crazy directory remapping.
    lua_pushfstring(L, "@%s", filename);

#if (PS3)
    // On the PS3, check always the /app_home/Lua/ directory!
    // This works both for dev-testing via remote HDD and on the game disc.
    // These work probably too just for developing, but app_home is fine!
    // /host_root/Lua/ or /dev_bdvd/PS3_GAME/USRDIR/Lua/
    CheckForValidFilenameWithPath(fullFilename, filename, "/app_home/Lua/");
#elif XBOX
    // For the Xbox we have to make sure always to load files from D:\ because
    // fopen ALWAYS expects a full paths, there are no current directories on the
    // Xbox 360 and therefore no relative paths! Check always "D:\Lua\<file>"
    CheckForValidFilenameWithPath(fullFilename, filename, "D:\\Lua\\");
#else
    // On the PC we just use the default search logic (see luaconf.h) and we
    // don't care about directories since we will already be in the correct one!
    // In earlier versions we had a lot of extra checks here.
    strcpy(fullFilename, filename);
#endif

    // Rest of this code is untouched, we just use fullFilename now!
    lf.f = fopen(fullFilename, "r");
    if (lf.f == NULL)
      return errfile(L, "open", fnameindex);
  }
  c = getc(lf.f);
  if (c == '#') {  /* Unix exec. file? */
    lf.extraline = 1;
    while ((c = getc(lf.f)) != EOF && c != '\n') ;  /* skip first line */
    if (c == '\n') c = getc(lf.f);
  }
  if (c == LUA_SIGNATURE[0] && fullFilename) {  /* binary file? */
    lf.f = freopen(fullFilename, "rb", lf.f);  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, "reopen", fnameindex);
    /* skip eventual `#!...' */
   while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;
    lf.extraline = 0;
  }
  ungetc(c, lf.f);
  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (filename) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, "read", fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}

The code in luaL_loadfile will call the new method CheckForValidFilenameWithPath, which helps us to build a full path filename for the Xbox 360 and PS3 platforms. It will just add D:\Lua\ or /app_home/Lua/ in front of the filename if it does not exist yet. Here we can see another pain point of C code, it takes so freaking long to do very simple tasks (in .NET this would be a 2-liner).

static void CheckForValidFilenameWithPath(char* fullFilename,
  const char* filename, const char* platformPath)
{
  int i = 0;
  int pathLen = (int)strlen(platformPath);
  // Already have a filename with the correct path?
  if ((int)strlen(filename) > pathLen)
    for (i = 0; i < pathLen; i++)
      if (filename[i] != platformPath[i])
        break;

  // Found path?
  if (i == pathLen)
  {
    // Then just use the existing file, happens when loading libraries from Lua
    strcpy(fullFilename, filename);
  }
  else
  {
    // Copy path
    for (i = 0; i < pathLen; i++)
      fullFilename[i] = platformPath[i];
    // Add the relative filename
    for (i = 0; i < (int)strlen(filename); i++)
      fullFilename[i+pathLen] = filename[i];
    // And finally close the string
    fullFilename[(int)strlen(filename)+pathLen] = 0;
  } // else
} // CheckForValidFilenameWithPath(fullFilename, filename, platformPath)

And that are the most important code changes to make everything run on the Xbox 360, PS3 and PC. Here is the main method for all the TestLua projects (always pretty much the same, only the Xbox 360 uses a slightly different way to print to the console):

int _tmain(int argc, _TCHAR* argv[])
{
    printf("Initialize Lua\n");

    Lua = lua_open();
    luaL_openlibs(Lua);
    lua_register(Lua, "print", lua_print);

    // Simple print test from Lua:
    luaL_dostring(Lua, "print(\"Hi from Lua\")");

    // Now load a Lua file, which also loads another Lua file (LuaUnit.lua)
    if (luaL_dofile(Lua, "SomeClass.lua") != 0)
    {
        printf("Error: Unable to execute Lua file %s: %s\n", "SomeClass.lua",
            lua_tostring(Lua, 1));
    } // if (luaL_dofile)

    // Stop the Lua engine and cleanup
    printf("Closing Lua\n");
    lua_close(Lua);
    Lua = NULL;

    // Keep command window for debug mode (ctrl+f5 does not have this issue)
    char input[100];
    gets_s(input);

    return 0;
} // _tmain

Summary:

• Using Lua on other platforms is easier than Python or other dynamic languages I have checked because Lua is written completely in ANSI
• Making code run on multiple platforms is still hard in C++ because you not only end up with ugly #define code, but you also have to test a lot
• Building native applications is freaking time-intensive, it took me almost a full day to write this app and test it on every platform from serval PCs. In .NET it would take me less than an hour, but then again there is no .NET on the PS3 and our current game engine is using native code anyway.
• On the Xbox 360 you always have to use full paths like D:\Lua\SomeClass.lua, relative paths won't work. The SDK says you should use Game:\, but since our game engine uses D:\ too, I left it that way in this sample too.
• The PS3 is even more complicated with lots more possible mounted paths, but it seems /app_home did always work in both my samples and our game, so I'm sticking with it :)

If you are interested in Lua from .NET, check out the LuaInterface project. Someone also started NUA (Lua on the DLR) 2 years ago, but it seems this project (and many other attempts) have died a long time ago. There was also a port by the Lua guys (especially the great Fabio Mascarenhas, who has also started the LuaInterface project and is still involved) called LuaCLR, but it is still in alpha stage and pretty useless, LuaInterface on the other hand is a fully working library! The difference here is that LuaCLR emits CLR code like any other .NET language, while LuaInterface still relys on the Lua runtime. LuaInterface uses only 2 very simple C++ files, the rest is in C#, but because of that you currently cannot run it with XNA on the Xbox 360. IronPython, while fully written in C#, does not work on the Xbox 360 either because reflection is not supported by the Xbox 360 .NET framework.

And finally here is a link to Lua running on the iPhone: Lua v5.1.3 for iPhone/iPod, kinda cool, but since we can run C# code on the iPhone with help of the great Unity3D engine, I'm happy with that at the moment. The main problem for other languages on the iPhone is the missing libraries anyway (can't use Cocoa or any own UI) ..

Just in case you have not read about it yet: John Carmack has ported the good old Wolfenstein 3D game from 1992 to the iPhone as well.

Last week I finally bought a Intel's X25-M 80GB SSD and I have been using it for about a week now at work. A few weeks back I also posted my impressions about the cheap OCZ 64 GB SSD I used before and how bad it worked out with Visual Studio compiling for me (also some helpful links in there if you want to optimize your SSD).

My first impressions of the Intel X25-M 80GB are as follows:

• Initially very fast read, easily archived over 210 MB/s read performance with HDTune and other benchmarks.
• Blocks above 4kb were faster, smaller blocks were slightly slower. 4096 bytes also seems to be a good NFTS block size for them.
• Writing on it is slower with 70-80 MB/s, which is less than the OCZ SSD I had before in benchmarks. But real world usage tells another story as I was usually limited by the network, reading data from other disks and other factors. Other than the OCZ SSD the Intel one stayed very fast even with many files and programs open and doing crazy compiling in the background.
• Using Windows Vista is fine, but I remembered how much better Windows 7 was for SSD when I was using it a month ago. You can tweak Vista quite a bit with the helpful tips from the OCZ Foraum, but it is not the same thing as having the operating system handle all the dirty stuff for you in a much better way.
• So I'm back at Windows 7 again. Yes some apps and games do not work and I even have problems now doing some work related stuff (building content) for which I have to use another PC or use my secondary disk and boot Vista for that. But most tools I use everyday at work are just fine in Windows 7. Boot time is reduced, memory usage is way better, SSDs work great and the overall desktop experience is just so much better (I love the jiggle windows trick to show the desktop).
• Compile times for our game have gone way down (more on that in a second).
• When I'm going home and do some work there I clearly notice the difference. But even if I just open up FireFox or some other complex program, it seems like it takes forever on a normal hard disk. I was pretty happy with my system at home before, but I have not upgraded it for 2 years. It has never really bothered me before since I can still play all the games in Full-HD and everything is quick in Windows. I will probably wait until some cool new DirectX 11 cards will come out and of course some cool 8-core CPUs for some serious multicore action.
• I can sense a slight slowdown to last week, probably due the fact that I use the SSD each day heavily and it is pretty full. Hopefully it will not get worse over time, but even then I could quickly just copy everything over to my secondary disk, reformat and copy everything back.
• Do not believe most SSD reviews, it is really stupid that in the last 6 months almost no SSD review checked the usability of those things, even Linus Torvalds agrees with that (he is a Intel SSD user too ^^).
  
• Also check out this very good new article on AnandTech: The SSD Anthology: Understanding SSDs

I could probably go on and on about this and there are certainly advantages (yeah, reading is fast) and disadvantages (wtf, this is costly, and you will run out of disk space sooner than you can say "copy this over there"). Lets take a look at some random numbers of compile-time from our current game project (which I'm still not allowed to talk about, so just pretend it is a huge C++ project with many many files). Please note that all those times are not in any way accurate, both because I just checked a few times and estimated others and it is also very different from PC to PC anyway plus there are many other factors like having intermediate files already present or not, etc. I'm also not a hardware tester like Tom's Hardware Guide, go there if you want more and real benchmarks, this is just what a noticed in the last weeks. Keep in mind that I could only overclock my current Intel i7 setup a few days ago since the default CPU cooler by Intel for i7 is really crap, loud and can't cool very good. So I bought a Zalman 9900 instead and it allowed me to go to 3.75 Ghz without any problems (up from 2.66 Ghz default).

Older PCs we have at work are obviously slower anyway, but the point here is that the SSD gives an additional bonus. I also only included the PCs I have worked on, some interns have slower PCs, but they usually do never have to do a full recompile anyway. Keep in mind that I do not compare .NET compile times here (they are all below 2 seconds on all those systems even in the larger .NET projects I have). This is all unmanaged C++ code, but since we also do a lot of our code in scripts we do not have to compile C++ most of the time - and even when - we usually only change certain parts of the code and then a incremental build is used.



PS: I was watching the current South Park S13E02 Episode while writing this and I think it was pretty crappy (superheroes, no real story). Last weeks episode with Kenny's girlfriend was way better and a really good season starter.

Ferdi has posted a German Tutorial on how to use Visual Studio for Unity over at the MobileBits blog. This is kinda useful because Unity3d currently only works on the Mac and I'm pretty much unable to use the Mac for anything because I cannot use my own keyboard layout over there. I have been working a little via VNC, but having a familar IDE like Visual Studio is just so much better ;)

An English tutorial can be found at the UnifyCommunity site:
http://unifycommunity.com/wiki/index.php?title=Setting_up_Visual_Studio_for_Unity

Basically you create a console application in Visual Studio where you write all the c# scripts for Unity. Then a little FTP upload tool is used to upload those c# files to the FTP server on the Mac, where you can then compile the scripts in the Unity IDE. More on that next week. I have not worked much with Unity yet because we have been busy at exDream with our current project and we got many new interns this week plus marketing guys coming by and stuff like that.

I have updated my CR_Commenter plugin quite a lot last weekend and tested it all week. CR_Commenter v1.9 new features include full hotkey and feature customization, much better support for C++, Lua, Python and way better light-commenting features and more powerful and stable region features. Thanks to the many people writing me emails using the addin or even improving it themselfs and sending me updated code back (full thanks are listed below).

Download links for CR_Commenter v1.9 (2008-11-27):

• Click here to download the installer: CR_Commenter.exe v1.9 (98 kB)
• And here is the sourcecode for: CR_Commenter_src.zip v1.9 (90 kB)

If you do not know about the CR_Commenter plugin for CodeRush yet (I originally wrote it back in 2004 and still use it every single day), please read the article for version 1.8 from earlier this year where I also added a lot of features and this article of version 1.7 from 2006 or check out the original Commenter - A CodeRush plugin which helps you to comment your code article on CodeProject.net from 2004, which I have finally updated too (its about time).

To use CR_Commenter you need to install CodeRush 2.0 or later, which is a commercial product you can get from DevExpress here. You can also use the free DXCore component with a reduced feature set, but all the basic CR_Commenter features should work fine there too. Please note that there is also a recently released free CodeRush Xpress Edition, which should work too, but I have not tested it.

If you have used CR_Commenter before the first thing you will probably notice is the completely new options screen:


Let me go through all the new and improved features. Over the years so many things were added to CR_Commenter and since the options screen, hotkeys and feature set has changed so much all the time, it is hard to keep track what features are available in the latest version. To make it easier for new users to understand what you can do with CR_Commenter v1.9 I decided to explain every feature, which might also be useful for existing users as they might have missed some of the cool features before. All these features can now be enabled or disabled and you can set your own hotkey for most of them (if it makes sense). I will go through the same list as seen in the Options screen, which is also represented in the FeatureHotkey.cs class that has a lot of comments and explanations for each feature. As mentioned before all Hotkeys can be changed to whatever you like. Please note that the pictures might look a little different than your Visual Studio code, I just use a different theme and bigger font sizes, it is still Visual Studio 2008.

CR_Commenter v1.9 Features:

• Automatically add comments after closing blocks with }
  Hotkey: }
  This is the most basic feature of CR_Commenter. In v1.9 you can now enable or disable it as you like. Please also note that you can now specify to only generate comments for blocks greater than a certain size (by default 5), which reduces the amount of comments generated greatly and is also the new default setting! Note: Since this is only parsed after pressing } it only works for C like languages like C, C++, C#, fx, UnrealScript, etc. For some of the newly supported languages like Lua (or VB, which is still untested and more or less unsupported sadly) or even Python that use identation for blocks you have to use Ctrl+1
  
  
  
  
• Force Comment Block
  Hotkey: Ctrl+1
  Same as }, but allows us to comment whole classes and methods easier by positioning the text cursor on what you want to update. Use Ctrl+6 for always updating the whole file no matter where you are located. Can also be used for languages not using } for commenting or if you have disabled } commenting.
  
  
  
  
• Add namespace for the current word. (Disabled by default)
  Hotkey: Ctrl+2
  Used to support automatically adding a namespace in C# 3.0/VS 2008. Not longer needed since VS supports it on its own now! This feature could be used back in 2006 to add namespaces in VS 2005.
  
  
  
  
• Autogenerate region around the current block
  Hotkey: Ctrl+3
  Autogenerate region around the current block (method, property, constructor, etc.), for selections use Ctrl+R (default CodeRush hotkey). The most important thing about this feature is that it will automatically give your new region the method name and it will make sure all comments belonging to the method are put inside the region. Note: In the picture the #region SetNewValue and #endregion parts are hidden by CodeRush with lines to make the source code look cleaner.
  
  
  
  
• Autogenerate regions for the whole file (all methods)!
  Hotkey: Shift+Ctrl+3
  Same as Ctrl+3, but will go through the whole file and generate region blocks for every method that does not seem to have one yet! You can also just go to the class level and press Ctrl+3 to generate all regions for all class methods. Since v1.9 also works with C++ and will generate #prama regions for you. BTW: Parsing the whole file with Ctrl+1 will also generate a includes region automatically for you.
  
  
  
  
• Toggle collapsing and expanding current selection
  Hotkey: Ctrl+4
  Very useful to avoid having to use the mouse for these things. Please note that Visual Studio also supports collapsing sections with Ctrl+M+M, but that will only work on sections, not on regions, which is a pretty useless feature in C#, Ctrl+4 is much better :) Since v1.9 also supports regions if CodeRush does not parse them (e.g. #pragma region in C++), so you can now even collapse C++ regions.
  
  
  
  
• Toggle collapsing and expanding everything at top level
  Hotkey: Ctrl+5
  Will expand or collapse all regions in the current file at once. First time pressed it expands everything, second time everything is collapsed, good for getting a quick file overview. Note that again Visual Studio supports collapsing all sections with Ctrl+M+L, but this really sucks because all methods, xml comments and everything else is collapsed too and expanding all of that again is a pain. Again, Ctrl+5 is much better.
  
  
  
  
• Update all comments and xml blocks for the whole file.
  Hotkey: Ctrl+6
  Similar to Ctrl+1, but will not only process the current section, but the whole file. This generates and updates all comments in the whole file, updates the header comment and makes sure all Xml comment blocks are generated if missing (see below).
  
  
  
  
• Update just the header comment.
  Hotkey: Ctrl+8
  Force updating the header comment, can be useful to test new headers and to remove any old header data. Also updates the created and modified timestamps. Since v1.9 changing or having different header comments now works, but it will also remove any existing header comments to replace with the new header comment if the format is different. Also check out the new cool options in the Commenter options screen, which allows you to specify seperate header comments for each solution or project if you like.
  
  
  
  
• Generate xUnit or NUnit unit tests automatically for us
  Hotkey: Ctrl+9
  Generate xUnit unit tests automatically for us for all methods! All unit tests will have to be written and tested until everything is implemented correctly. Existing test methods will not be overwritten! You can either enable xUnit Tests (like in the screenshot below) or NUnit Tests.
  
  
  
  
• Remove All Comments, use this to cleanup a file
  Hotkey: Ctrl+0
  Remove All Comments, use this to cleanup a file. It will only remove generated comments (after } blocks). Also nice for testing and cleaning up other peoples code.
  
  
  
  
• Obfuscate whole file (removes ALL comments, xml comments, empty lines, headers, etc.)
  Hotkey: Ctrl+\
  Obfuscate whole file. All code will stay the same, but we will remove ALL comments, Xml Comments, empty lines, headers, etc. to reduce file size and make code harder to read. Not really a Commenter feature, but I implemented this for another app and it was easy to implement here too.
  
  
  
  
• Autogenerate XML Comments
  This feature will automatically add XML Comments to methods, properties, constructors, classes, etc. when the normal comments or generated. You can specify if you only want to generate XML comments for public methods or if you want to generate XML tags for all parameters (which was default on, but is default off now to produce much less empty xml parameter comments, which are usually not updated by the programmer anyway).
  
  
• Other Improvements in v1.9
  
  • Added cool new per-project and per-solution header comments
    
    
  • New shortcut Ctrl+6 to just update the header
    
    
  • Added lower limit for block size to reduce comments for small blocks (also works recursive)
    
    
  • Added new features and hotkey logic, everything now easier to customize
    
    
  • Easier visibility if plugin is installed and working (more visual events, better options screen)
    
    
  • Fix generating comments for new files (messes up the using directives region), seems to be related to generating header comments
    
    
  • Improve line length count for removing comments (good new standard: 5 lines)
    
    
  • Fixed several problems in Format-Strings or SQL code.
    
    
  • Added whole method identifier from the accessibility keyword to the closing parenthesis for region directives.
    
    
  • Improved the CodeRush-autogenerated regions (Ctrl+3), make sure comments are not messed up
    
    
  • New default: Do not generate Xml parameter comments (mostly not used for better documentation anyway, adding them yourself for more details is more comfortable)
    
    
  • Added C++ "pragma region - #pragma endregion" support (thanks for testing to Enrico)
    
    
  • C++: include includes into using directives region
    
    
  • C++: Fixed header comment (project name could not be found)
    
    
  • C++: Do not use xml comments, just leave normal comments!
    
    
  • Fixed wronly generated regions outside of our plugin
    
    
  • C++: Fixed #includes region, do not modify include lines
    
    
  • Fixed region generation when CodeRush generates region before us!
    
    
  • Fixed header in case some other header comment is used
    
    
  • Added multiline header comment support for // /* */ -- and # to support all kinds of languages and comment formats
    
    
  • Also force generate comments on properties (methods work, also test constructors)
    
    
  • C++ allow Ctrl+4 and Ctrl+5 for C++ region collapsing (using visual studios Edit.ToggleOutliningExpansion and Edit.ToggleAllOutlining as fallbacks)
    
    
  
  
• Older and Improved Features (from v1.8 and before):
  
  • Fixed '}' inside string or comment, even if CodeRush has not processed the whole file yet (double checking now). E.g. the following string will not longer produce any comments:
    SQL = String.Format("UPDATE GC SET Status = {0} WHERE ID = {1}", Msg, ID);
    
    
  • Supporting to generate regions again, but split up function names, also suppress CodeRush region generation with Ctrl+3, Ctrl+R (also suppressing VS hotkeys now) Better support for auto generated region blocks for big methods, just press Ctrl+3 to generate region (let CR_Commenter choose the name automatically, allow editing after pressing Ctrl+3).
    
    
  • Fixed #if DEBUG in Using directives (caused trouble, was removed)
    
    
  • Also added code to always remove #if DEBUG and #endif, especially at the beginning and end of the Using directives region
    
    
  • Fixed double line #using Using directives issue when using directives got mixed up.
    
    
  • Fixed long line commenter generation, will now still work, but not longer cut of any of your code. If you have long code lines (>100 letters), you can still generate comments for them and the Commenter will not longer cut anything off (you are responsible for formating the code).
    
    
  • Fixed auto-collapsing of regions, especially when closing blocks in methods, the method should not auto-collapse!
    
    
  • Checked support with other languages (which are at least similar to C#) like Lua, Python, etc.
    
    
  • In Python/Lua, etc. we had to disable } comment generation and also use other comments (# for python, -- for lua) when using Ctrl+1 for header generation.
    
    
  • Test all other hotkeys and make sure they do not affect the code badly in CodeRush unsupported languages (note: VB is still not supported since it is very different from c# code).
    
    
  • Supports now any unit test framework, actually does not care anymore what you use, only the unit test generation code will add Xunit, but you can easily change that code if you want to.
    
    
  • Options are now available directly into Editor tree of the CodeRush options!
    
    
  • Refactored all classes, added about 4 new classes and removed 1 unused class. The source code is now much easier to read, to follow and to change.
    
    
  • Remove commented out code, cleaned up the source code a lot.
    
    
  • And finally applied the commenter to the commenter source code :)
    
    

I would also like to thank all the people that wrote me emails, use the plugin, all my colleagues that use this tool everyday too (shoutout to Enrico Cieslik and Henning Thöle, and also Fabienne Khöle) and the following people who have contributed with ideas, code improvements and helpful bug reports (they are also mentioned in the TODO.txt file for their specific bug report or improvement idea): Andy (mangledmonster), Andrew Jones, Gil Yoder, Alexey Romanov, Enrico (Judge), Ed Blankenship, Sven Heitmann, Jan Diederich, Rob Heyes, Dan Avni, Andy Jacobs and everyone else :)

I watched quite a lot of PDC 2008 Talks this week and I really like that they put all that stuff online for everyone to watch. To learn more about the PDC, check out event photos or see all the sessions, directly go to http://www.microsoftpdc.com/. Another good list of all sessions can be found here: http://coolthingoftheday.blogspot.com/2008/10/pdc2008-quick-video-link-list.html

Here are my personal favorite session recordings (from what I have seen, I have not seen all talks):

The Future of C# (by Anders Hejlsberg)
http://channel9.msdn.com/pdc2008/TL16/
As always Anders is fun to watch and you learn a lot about C#, the new dynamic keywords and whats to come. Not really stuff you will probably use right now, but good to know what we can do and probably will do in a few years. In case you want to watch IronPython or IronRuby, this session is a good prerequisite too.

Deep Dive: Dynamic Languages in Microsoft .NET
http://channel9.msdn.com/pdc2008/TL10/
Jim Hugunin (the creator of IronPython and DLR) talks about IronPython and the DLR (Dynamic Language Runtime) in general. Fun to watch, interesting insights and generally always interesting to anyone interesting in dynamic languages.

Panel: The Future of Programming Languages
http://channel9.msdn.com/pdc2008/TL57/
I like Panels like this and they had really great guests: Gilad Bracha, Douglas Crockford, Anders Hejlsberg, Erik Meijer, Wolfram Schulte, Jeremy Siek. They get a little bit off-topic at the end, but overall an interesting and very fun to watch discussion with a lot of interesting arguments on all sides (lot of different languages are discussed).

IronRuby: The Right Language for the Right Job
http://channel9.msdn.com/pdc2008/TL44/
I have learned Ruby few months ago and used it for a few days, but I never really liked it and continued to use Python instead, but John Lam (the IronRuby inventor) is a cool dude and give a lot of tips and shows off a lot of cool demos, which are equally interesting even if you would use some other dynamic language than Ruby.

Microsoft Visual C# IDE: Tips and Tricks
http://channel9.msdn.com/pdc2008/TL46/
Interesting and useful Tricks, Dustin shows off a lot of shortcuts and the whole presentation is in Visual Studio, which is cool.

Visual Studio Debugger Tips & Tricks
http://channel9.msdn.com/pdc2008/TL59/
I like Tips & Tricks talks and this one is good too. Informative and fun to watch as well. I will probably watch it again some time soon, like I will with some of the other talks too (because I usually code or do something else while the videos are running on a secondary monitor ^^).

Mono and .NET (by Miguel de Icaza)
http://channel9.msdn.com/pdc2008/PC54/
One of the last talks I discovered after watching many other talks. At first I thought this might be not really interesting for me, since this will probably focus only on Linux and Mono development, but I could not be more wrong. This was a really great talk, Miguel is such a great presenter, even when things go wrong, he makes fun remarks and is always on the topic. He gives great insight of game programming on Mono for Linux, the Mac and even the iPhone, which sounded really cool to me. Other topics were also interesting and fascinating.

Managed Extensibility Framework: Overview
http://channel9.msdn.com/pdc2008/TL33/
If you were every interesting in extending Visual Studio or want to hear about good ideas on how to extend frameworks or applications, this talk is helpful and insightful.

Visual C++: 10 is the New 6
http://channel9.msdn.com/pdc2008/TL13/
I'm not doing much C++ coding anymore, but many years ago this was my main programming language and it is still the most used language for games. This talk is highly informative and Boris Jabes talks in detail about new C++ Features in VS2010 and what is possible, what will get better and what even features will come in the future after VS2010. Also a lot of interesting questions and answers at the end.

The two talks Parallel Programming for C++ and Parallel Programming for Managed Developers were also highly informative and gave us programmers some good ideas on how to make better use of Multi-core systems. I especially like the Talk Parallel Library (TPL), which is insanely easy to use:

Parallel.For(0, 100, i => a[i] = a[i] * a[i]);

Microsoft XNA Game Studio: An Overview
http://channel9.msdn.com/pdc2008/TL43/
Frank Savage from the XNA Team talks about XNA Game Studio 3.0, which is highly informative if you don't know much about XNA yet. For me there was not much new to learn in this talk, but Frank is such a cool guy and it is fun to watch how he presents his stuff, which is one more reason to watch this talk. He also shows off some XNA games and the recently XNA Dream Build Play contest winner games.

The keynotes from Day 1 (Azure, Infos) and Day 2 (Windows 7, Visual Studio 2010 and .NET 4) were also interesting to watch. Other sessions sounded also interested and from what I have seen Oslo, Azure, Silverlight, Unit Testing in VS2010 and other new stuff is cool too, but I have no plans to use that stuff anytime soon (maybe Oslo for parsing text, which is easy and cool).

I hope you enjoy these videos as much as I do, definitely a great idea to put them all online :)

Since I'm not posting many XNA articles lately, lets announce some stuff from Ziggyware :)

Ziggyware Fall 2008 XNA Article Contest


Start date: October 12'th 2008
Ending Date: November 30'th. (Midnight CST)

Contest Subjects

• Write an XNA related tutorial for XNA developers (see examples)

Important: Read How to create an article for Ziggyware - Entries are subject to approval and may be disqualified per disgression.

Prizes

• First Place
  • XBox 360 Elite (USA NTSC)
  • One Year subscription to the Microsoft XNA Creators Club
• Second Place
  • One Year subscription to the Microsoft XNA Creators Club

More information can found at Ziggyware.com:
http://www.ziggyware.com/news.php?readmore=905

Hi Folks, sorry for not updating my blog for such a long time. I basically had no time and not much interesting to report. Since I will focus more on game programming again in the near future, more blogging might happen again :)

Anyway, yesterday I was an interview partner on a German TV show called "neues", which can be seen on TV Sundays each week at 16:30 (4:30 PM). The topic is user generated content and I will answer some questions in the second half of the show. Since the show is not technical, it might not be of much interest for game developers, but hopefully the general public can learn a bit about XNA and the cool opportunities everyone has with developing games on their Xbox 360 or Windows PC.

It was a very interesting experience seeing how a show gets created, how many people are involved and how often it is rerecorded until everyone is happy. The people at ZDF/3sat were also very nice and it was fun to work with them.

The show will also be available on the 3sat video archive online (probably in the next few days):
http://www.3sat.de/mediathek/mediathek.php?query=*&red=neues&mode=search

Update 2008-09-14: Direct link to the video from the 3sat mediathek:
http://www.3sat.de/mediathek/?obj=9543

Some pics from my journey to Mainz where the TV show was recorded. It was quite a long trip taking about 5 hours reaching Mainz. At Frankfurt I moved from the ICE (fast train, but the power supply did not work so I could not even use my laptop, grr) to a slower train called RE (which does not supply any power to the train riders).


The weather was not very enjoyable Friday. In the morning it was raining very much.


Since I was not allowed to take any pictures inside the studio and due the bad weather I took not many photos. Hopefully I can post a small video once the 3sat video from the show is available.


Getting back home was the same ride in reverse. I did not spend much time in Mainz or Frankfurt due the bad weather and I was kinda tired so I played a few games I wanted to check out (like Spore and Space Siege) in the train. This ICE at Mainz was going in the opposite direction, seems like not many ICE trains drive from Mainz to Frankfurt, so I had to take a slow RE train again.


And finally in Frankfurt at the MUCH bigger train station. They have about 30 rail tracks there and you can see a lot of people wearing business suits. Probably all bankers and investors, hehe ^^

Okay, that's it. Maybe I will finish up some blog posts I have written in the last months, but never finished them due lack of time and motivation. Take care!

As you can read on the XNA Team Blog the first CTP (kinda early beta/alpha) for XNA Game Studio 3.0 is available for download now.
The major new features beside full VS 2008 (Express, Standard, Pro, etc., but not longer VS 2005) support you can now write and deploy games for the Zune. Luckily for me I own one (not many non-US people do since the Zune is ONLY available in the US). Please note that while the Zune is a powerful device it obviously will NOT support any advanced rendering techniques at all, but all the other parts of the XNA Framework are supported very well. This means 3D games or shader based games will be possible on the Zune and the display is just 320x240 pixel and there are other things you have to think about (controls and playing music), but other than that it should be pretty easy to start developing Zune games right away :)

While the new features are really cool and I definitely will check out the Zune development options (XNA MVPs already saw some cool samples and stuff at the MVP Summit about this ^^), the show stopper for me is the "no 64 bit support" issue. All my PCs (work, home, laptop and even the servers) run on 64 bit vista/win2003/win2008, it will not be easy to play around with XNA 3.0 ("Hey, go away from that PC, I wanna test something").

Anyway, check out XNA Game Studio 3.0, especially if you own a Zune and find this Zune-targeted game development cool. Other features worth checking out are better sound APIs in the new Media namespace, namely the SoundEffect and MediaPlayer APIs. They allow you to play sounds and music much easier and in more formats (yeah, mp3s).

Tuesday's Sessions were happening on the Microsoft Campus. Since the XNA MVP Party yesterday and waking up this morning (which is actually Wednesday) I had no time to prepare and publish this blog post. On the Microsoft Campus, especially in Building 85 (XNA Guys) there was no way I could upload this because Internet was just way too slow. So this post had to wait until I got back to the Hotel after the Attendee party.

This morning we could eat breakfast at the Hotel in this huge ballroom.	Walking out of the hotel to the bus, which will take us to the 15km distant Microsoft's Campus.	The weather is pretty much the same as on Monday or Sunday. Driving up north here until we reach the bridge over to Kirkland and Redmond.
After about a 40 minute drive we arrived at the Microsoft Conference Center (Building 33). Traffic is not going very fast over here ^^	This is how it looks inside the Microsoft Conference Center (short MSCC) at 9:00 in the morning when a conference is going on.	Instead of attending the first XNA session I decided to take a look at the Silverlight 2.0 talk by the master himself: Scott Guthrie
Until someone decided to throw down this wall, which causes quite a bit of noise and amusement.	And this was me uploading the post for Monday yesterday while listening to the sessions.	After eating some lunch and then some cake (see picture) I watched some StarCraft vods from GOM TV with the good old Nick 'Tasteless' commenting in English. This was the game with Tossgirl, which was very exciting to watch.
Shortly after that another session about ASP.NET MVC started with Scott Hanselman.	The MVC talk was very enjoyable to watch because of the many remarks Scott made.	As you can see, the MVC framework is still very much a work in progress.
Okay, let me take a look at that. Hmm, that should test the MVC routing engine. While Scott was talking about that I tried to write some MVC tests with Xunit, which worked just great (as advertised).	Setup-Code for some routes, looks complicated, but I'm not sure how to make them much easier.	I went to Building 85 now for more XNA sessions over there. It was raining quite a bit.
One of the buses traveling from building to building at the Microsoft Campus.	A few native DirectX MVPs, which probably make fun of us managed DirectX/XNA MVPs. But it is all in good fun, we all like DirectX somehow :)	Heading back to Seattle through all the traffic.
Instead of heading back to the Hotels all XNA/DirectX MVPs had a nice little dinner at Jillian's, which obviously did not have enough power to light up both parts of their logo.	Andy Dunn (The ZMan), Sean ?, Michael Klucher (from the XNA Team) and George Clingerman (Mr. www.XnaDevelopment.com) standing around and discussing the physics of billiard :)	Michael Klucher, Nick Gravelyn and Niko Sumi? sitting down and discussing more XNA.
And finally heading back to the Hotel after we were thrown out from Jillian's ^^

More pictures today from the second and last day at the Microsoft Campus. Cya!

After updating so much on my blog the last 2 days and posting all the unfinished posts I had prepared a few weeks ago I finally reach the last important one I wanted to make. But due the fact that I will fly to the MVP summit tomorrow I better not stay away too long. Hopefully this post still contains all the vital information. Let's get started.

First of all you need the latest version of the free NSIS (Nullsoft Installer System):

• http://nsis.sourceforge.net/Main_Page

There are TONS of information on that website and if you are new to the NSIS scripts be sure to read at least the quick start guide (called simple tutorials) and use the NSIS Forums search feature for more specific problems and questions.

Good, now just let us go quickly through the .nsi file I currently use for all my XNA Game Setups. Please note that this file has evolved over many years (I guess I started using NSIS in 2002) and parts might not be simple to read or even be how it is done today, but the script was always a very helpful and quick way to create Setups for me. While detecting all the requirements to make XNA work is not a piece of cake many NSIS tutorials helped me to figure all the parts out.

The following .nsi script is from the previous blog post "Loading Collada Models in XNA 2.0 and doing Skinning and Animation" and was used to build the XnaSkinningColladaModelsSetup.exe (2.2 MB) file. If you want to use it for your own XNA Game project just use a text-editor and replace "XnaSkinningColladaModels" with your own game name (it occurs many times in the file). Also create a .bmp file with the name of your game for the setup header (check MUI_HEADERIMAGE_BITMAP) or comment the 2 Header Image lines out with a ; or # if you do not want any header image. All files are grabbed from the /Release directory and will be installed to the target (make sure that you only have files in the /Release directory that you want to have deployed). This should be already enough information to make your setup work, the rest of this article will just go into the details of the script.

The XnaSkinningColladaModelsSetup.nsi file is organized in 6 parts (Initialization, onInit, File section, Optional sections, Helper functions and Uninstall):

• Initialization: Just sets all the names, the required modules, and how the setup is displayed to the user (all the pages, see MUI_PAGE). Version information and file information are also a vital part here and you should enter some meaningful values here. Also put any language strings for different languages (this installer is just using english text) at text part of the initialization code.
  

  !include "MUI.nsh"
  SetCompressor /SOLID lzma
  
  ;Name and file
  Name "XnaSkinningColladaModels"
  OutFile "XnaSkinningColladaModelsSetup.exe"
  
  ;Default installation folder
  InstallDir "$PROGRAMFILES\XnaSkinningColladaModels"
  
  ;Get installation folder from registry if available
  InstallDirRegKey HKLM "Software\XnaSkinningColladaModels" ""
  
  !define MUI_COMPONENTSPAGE_NODESC
  
  ;Interface Settings
  !define MUI_ICON "${NSISDIR}\Contrib\Graphics\Icons\modern-install-blue.ico"
  !define MUI_UNICON "${NSISDIR}\Contrib\Graphics\Icons\modern-uninstall-blue.ico"
  !define MUI_HEADERIMAGE
  !define MUI_HEADERIMAGE_BITMAP "XnaSkinningColladaModels.bmp"
  !define MUI_HEADERIMAGE_UNBITMAP "XnaSkinningColladaModels.bmp"
  !define MUI_ABORTWARNING
  
  ;Pages
  !insertmacro MUI_PAGE_WELCOME
  !insertmacro MUI_PAGE_COMPONENTS
  !insertmacro MUI_PAGE_DIRECTORY
  !insertmacro MUI_PAGE_INSTFILES
  
  !insertmacro MUI_UNPAGE_CONFIRM
  !insertmacro MUI_UNPAGE_INSTFILES
  
  ;Languages
  !insertmacro MUI_LANGUAGE "English"
  
  ;Version Information
  VIProductVersion "1.0.0.0"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "ProductName" "XnaSkinningColladaModels"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "Comments" "XnaSkinningColladaModels project, see http://BenjaminNitschke.com"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "CompanyName" "BenjaminNitschke.com"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "LegalCopyright" "© BenjaminNitschke.com 2007"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "FileDescription" "XnaSkinningColladaModels"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "FileVersion" "1.0.0.0"
  
  ; The text to prompt the user to enter a directory
  ComponentText "This will install XnaSkinningColladaModels for XNA 2.0."
  
  UninstallText "This will remove XnaSkinningColladaModels for XNA 2.0 from your computer."
  
  ; New .NET Check function, we want at least v2.0
  !define DOT_MAJOR "2"
  !define DOT_MINOR "0"
  
  ; The text to prompt the user to enter a directory
  DirText "Please select a directory for the installation:"
  
  AutoCloseWindow true
  

  
  
• .onInit: This method is executed before the installer is run and can perform any checks you like. I just try to find out if any c:\games directory exists in several languages, if it does, then we can use it, else just use the default c:\program files location.
  

  
  Function .onInit
      ;First try to find "Games" (eng), "Spiele" (ger), "jeux" (fr),
      ; "giochi" (ita), "jogos" (port) or "spelen" (nl) directory!
      StrCpy $R1 "C:\Games"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\Spiele"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\jeux"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\giochi"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\jogos"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\spelen"
      IfFileExists $R1 UseIt
      Goto DontUseIt
  UseIt:
      StrCpy $INSTDIR "$R1\XnaSkinningColladaModels"
  DontUseIt:
  FunctionEnd
  

  
  
• File section: This is the most important section and always required because it tells the setup where the files come from (in our case from the Release sub directory) and where they should be installed (in #INSTDIR).
  

  
  ; The stuff to install
  Section "XnaSkinningColladaModels"
    SectionIn RO
    ; Set output path to the installation directory.
    SetOutPath $INSTDIR
    ; Put file there
    File /r "Release\*.*"
   
    ; Write the installation path into the registry
    WriteRegStr HKLM "SOFTWARE\XnaSkinningColladaModels" "Install_Dir" "$INSTDIR"
    WriteRegStr HKLM "SOFTWARE\XnaSkinningColladaModels" "Version" "v1.0"
    ; Write the uninstall keys for Windows
    WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\XnaSkinningColladaModels" "DisplayName" "XnaSkinningColladaModels"
    WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\XnaSkinningColladaModels" "UninstallString" "$INSTDIR\uninstall.exe $INSTDIR"
    WriteUninstaller "uninstall.exe"
  SectionEnd
  

  
  
• Optional sections: These sections are optional and the user can decide not to create desktop shortcuts, not to install XNA 2.0 if it is not found, etc. I usually leave them all on by default because they make sense. Most important here are the DirectX and XNA prequisites sections, which make a lot of use of the helper functions.
  

  
  ; Optional sections
  Section "Create Start-Menu shortcuts"
    StrCpy $R1 "$SMPROGRAMS\XnaSkinningColladaModels"
    CreateDirectory "$R1"
    CreateShortCut "$R1\XnaSkinningColladaModels.lnk" "$INSTDIR\XnaSkinningColladaModels.exe" "" "$INSTDIR\XnaSkinningColladaModels.exe" 0 "" "" "XnaSkinningColladaModels"
    CreateShortCut "$R1\XnaSkinningColladaModels Uninstall.lnk" "$INSTDIR\uninstall.exe" "" "$INSTDIR\uninstall.exe" 0 "" "" "Uninstall XnaSkinningColladaModels."
    CreateShortCut "$R1\BenjaminNitschke.com.lnk" "http://BenjaminNitschke.com" "" "" 0 "" "" "BenjaminNitschke.com website from the creator of this game"
  SectionEnd
  
  Section "Create Desktop shortcuts"
    CreateShortCut "$DESKTOP\XnaSkinningColladaModels.lnk" "$INSTDIR\XnaSkinningColladaModels.exe" "" "$INSTDIR\XnaSkinningColladaModels.exe" 0
  SectionEnd
  
  Section "Check for .NET 2.0 Framework (required)"
    Call IsDotNETInstalled
    Pop $0
    StrCmp $0 1 found.NETFramework no.NETFramework
  no.NETFramework:
  ;MessageBox MB_OK ".NET was not found .."
    ; First of all check for correct IE version!
    Call CheckForCorrectIEVersion
    ; First check if DotNetFx.exe exists
    IfFileExists "$EXEDIR\DotNetFx.exe" 0 DotNetFileNotFound
    ReadRegStr $R0 HKLM "SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" "AutoAdminLogon"
    ExecWait '"$EXEDIR\DotNetFx.exe" /q'
    StrCmp $R0 1 RestoreAutoLogon End
  RestoreAutoLogon:
    WriteRegStr HKLM "SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" "AutoAdminLogon" "1"
    goto End
  
    ; If DotNetFx.exe was not found, try downloading online
  DotNetFileNotFound:
    MessageBox MB_YESNOCANCEL|MB_ICONQUESTION ".NET Framework 2.0 was not found. XnaSkinningColladaModels will not run without it! $\r$\nWould you like to automatically download and install it? $\r$\n $\r$\nChoose Yes for an automatic download and installation. $\r$\nChoose No, you want to select the download yourself (e.g. locally if already downloaded). $\r$\nMicrosoft's .NET Framework Download page will open up! $\r$\nChoose cancel to abort any action here, you will have to install the .NET Framework (2.0) $\r$\nyourself. That can be done for example automatically by the Windows Update $\r$\n(just go to http://windowsupdate.microsoft.com )" IDYES DL IDNO IndirectDL
    goto End
    IndirectDL:
      ExecShell open "http://www.microsoft.com/downloads/details.aspx?FamilyID=0856eacb-4362-4b0d-8edd-aab15c5e04f5" "" SW_SHOWNORMAL
      goto End
    DL:
      Call ConnectInternet ;Make an internet connection (if no connection available)
  
      StrCpy $2 "$TEMP\DotNetFX_Setup.exe"
      NSISdl::download http://download.microsoft.com/download/5/6/7/567758a3-759e-473e-bf8f-52154438565a/dotnetfx.exe $2
      Pop $0
      StrCmp $0 success success
        SetDetailsView show
        DetailPrint "download failed: $0"
        MessageBox MB_OK ".NET could not be installed, download was aborted. Please install .NET yourself! $\r$\nRest of the installation went ok, you just need .NET 2.0 to start XnaSkinningColladaModels!"
        goto End
      success:
        ; Install, in quiet mode (will do everything automatically)
        ExecWait '"$2" /q'
        Delete $2
    
      goto End
  found.NETFramework:
  ;MessageBox MB_OK ".NET is installed, very good ^^"
  End:
  SectionEnd
  
  Section "Check for DirectX 9.0c (November 2007) (required)"
    call IsDirectX9Installed
    pop $0
    StrCmp $R1 1 maybefound.DirectX9 notFound.DirectX9
  maybefound.DirectX9:
      ; Check if managed directx december 2006 is installed!
      IfFileExists "$WINDIR\Microsoft.NET\DirectX for Managed Code\1.0.2911.0" found.DirectX9  
  notFound.DirectX9:
  ;MessageBox MB_OK "DirectX9 or better was not found!"
    ; First check if a file "dx90update_redist.exe" exists
    IfFileExists "$EXEDIR\directx_nov2007_redist.exe" 0 DX9RedistFileNotFoundTest1
    ExecWait '"$EXEDIR\directx_nov2007_redist.exe" /Q /T:"$TEMP" /C'
    ; Now install DirectX9
    ExecWait "$TEMP\dxsetup.exe"
    goto End
  DX9RedistFileNotFoundTest1:
      ; Check again in temp folder
    IfFileExists "$TEMP\directx_nov2007_redist.exe" 0 DX9RedistFileNotFound
    ExecWait '"$TEMP\directx_nov2007_redist.exe" /Q /T:"$TEMP" /C'
    ; Now install DirectX9
    ExecWait "$TEMP\dxsetup.exe"
    goto End
    ; If that failed, ask user!
  DX9RedistFileNotFound:
    MessageBox MB_YESNOCANCEL|MB_ICONQUESTION "DirectX9.0c November 2007 or better was not found. XnaSkinningColladaModels needs DirectX9.0c and Managed DirectX runtimes to run. $\r$\nWould you like to automatically download and install it? $\r$\n $\r$\nChoose Yes for an automatic download and installation. $\r$\nChoose No, you want to select the download yourself (e.g. in another language than english). $\r$\nMicrosoft's DirectX Download page will open up! $\r$\nChoose cancel to abort any action here, you will have to install DirectX yourself." IDYES DL IDNO IndirectDL
    goto End
  IndirectDL:
    ExecShell open "http://www.microsoft.com/downloads/details.aspx?familyid=1A2393C0-1B2F-428E-BD79-02DF977D17B8" "" SW_SHOWNORMAL 
    goto End
  DL:
    Call ConnectInternet ;Make an internet connection (if no connection available)
  
    StrCpy $2 "$TEMP\directx_nov2007_redist.exe"
    NSISdl::download http://download.microsoft.com/download/c/6/a/c6af7ea3-f85c-4fef-a475-5810f82def1f/directx_nov2007_redist.exe $2
  
    Pop $0
    StrCmp $0 success success
      SetDetailsView show
      DetailPrint "download failed: $0"
      MessageBox MB_OK "DirectX could not be installed, download was aborted. Please install DirectX yourself! $\r$\nRest of the installation went ok, you just need DirectX 9.0c December 2006 to start XnaSkinningColladaModels!"
      goto End
    success:
      ; Install quietly into temp folder
      ExecWait '"$2" /Q /T:"$TEMP" /C'
      ; Now install DirectX9
      ExecWait "$TEMP\dxsetup.exe"
      ; Yeah, we made it!
      ;why delete, maybe user wants to keep it ^^ Delete $2
    goto End
  found.DirectX9:
  ;MessageBox MB_OK "Found DirectX9 or better .."
    goto End
  End:
  SectionEnd
  
  Section "Check for XNA 2.0 (December 2007) (required)"
    call IsXNAInstalled
    pop $0
    StrCmp $R1 1 found.XNA notFound.XNA
  notFound.XNA:
    MessageBox MB_YESNOCANCEL|MB_ICONQUESTION "XNA 2.0 (December 2007) or better was not found. The XnaSkinningColladaModels needs the XNA 2.0 runtimes to run. $\r$\nWould you like to automatically download and install it? $\r$\n $\r$\nChoose Yes for an automatic download and installation. $\r$\nChoose No, you want to select the download yourself (e.g. in another language than english). $\r$\nMicrosoft's XNA Download page will open up! $\r$\nChoose cancel to abort any action here, you will have to install XNA yourself." IDYES DL IDNO IndirectDL
    goto End
  IndirectDL:
    ExecShell open "http://www.microsoft.com/downloads/details.aspx?FamilyID=15fb9169-4a25-4dca-bf40-9c497568f102" "" SW_SHOWNORMAL 
    goto End
  DL:
    Call ConnectInternet ;Make an internet connection (if no connection available)
  
    StrCpy $2 "$TEMP\xnafx20_redist.msi"
    NSISdl::download "http://download.microsoft.com/download/b/8/a/b8a1f45a-63e6-4d1f-9c03-55a83d30ee3b/xnafx20_redist.msi" $2
  
    Pop $0
    StrCmp $0 success success
      SetDetailsView show
      DetailPrint "download failed: $0"
      MessageBox MB_OK "XNA 2.0 could not be installed, download was aborted. Please install XNA 2.0 yourself! $\r$\nRest of the installation went ok, you just need XNA 2.0 (December 2007) to start the XnaSkinningColladaModels!"
      goto End
    success:
      ; Install XNA, quietly does not work on vista for some reason ...
      ExecWait 'msiexec /i "$2" /q'
      ; Yeah, we made it!
      ;why delete, maybe user wants to keep it ^^ Delete $2
    goto End
  found.XNA:
  ;MessageBox MB_OK "Found XNA 2.0 or better .."
    goto End
  End:
  SectionEnd
  
  Section "-Ask some questions"
    MessageBox MB_YESNO|MB_ICONQUESTION "Start the XnaSkinningColladaModels now?" IDYES StartRC
    goto DontStart
  StartRC:
    Exec "$INSTDIR\XnaSkinningColladaModels.exe"
  DontStart:
  SectionEnd
  

  
  
• Helper functions: These little helpers are the utitilies to make all the complex XNA installation steps work. They go through all the steps from making sure .NET is installed, installing it automatically for you if you like, even opening up an internet connection if you are on dialup, checking all the prequisites for installing .NET (like having at least IE 5) and then finally install all the XNA 2.0 and DirectX stuff that is required to run your game on top of that. All without ever leaving the installer and tested on many platforms. I usually do not touch this code unless I have to update some links or include a newer version of .NET, DirectX or XNA. I tried to comment my work, but understanding this script part will not be easy. Just do not expect that any changes you make here will be safe. You need to do a lot of testing in order to add additional functionality here.
  

  ; IsDotNETInstalled
  ;
  ; Usage:
  ;   Call IsDotNETInstalled
  ;   Pop $0
  ;   StrCmp $0 1 found.NETFramework no.NETFramework
  ;Function IsDotNETInstalled
  ;   Push $0
  ;   Push $1
  ;   Push $2
  ;   Push $3
  ;   Push $4
  ;
  ;   ReadRegStr $4 HKEY_LOCAL_MACHINE \
  ;     "Software\Microsoft\.NETFramework" "InstallRoot"
  ;   # remove trailing back slash
  ;   Push $4
  ;   Exch $EXEDIR
  ;   Exch $EXEDIR
  ;   Pop $4
  ;   # if the root directory doesn't exist .NET is not installed
  ;   IfFileExists $4 0 noDotNET
  ;
  ;   StrCpy $0 0
  ;
  ;   EnumStart:
  ;
  ;     EnumRegKey $2 HKEY_LOCAL_MACHINE \
  ;       "Software\Microsoft\.NETFramework\Policy"  $0
  ;     IntOp $0 $0 + 1
  ;     StrCmp $2 "" noDotNET
  ;
  ;     StrCpy $1 0
  ;
  ;     EnumPolicy:
  ;
  ;       EnumRegValue $3 HKEY_LOCAL_MACHINE \
  ;         "Software\Microsoft\.NETFramework\Policy\$2" $1
  ;       IntOp $1 $1 + 1
  ;        StrCmp $3 "" EnumStart
  ;         IfFileExists "$4\$2.$3" foundDotNET EnumPolicy
  ;
  ;   noDotNET:
  ;     StrCpy $0 0
  ;     Goto done
  ;
  ;   foundDotNET:
  ;     StrCpy $0 1
  ;
  ;   done:
  ;     Pop $4
  ;     Pop $3
  ;     Pop $2
  ;     Pop $1
  ;     Exch $0
  ;FunctionEnd
  
  ; Call IsDotNetInstalled
  ; This function will abort the installation if the required version 
  ; or higher version of the .NETFramework is not installed.  Place it in
  ; either your .onInit function or your first install section before 
  ; other code.
  Function IsDotNETInstalled
    Push $0
    Push $1
    Push $2
    Push $3
    Push $4
  
    StrCpy $0 "0"
    StrCpy $1 "SOFTWARE\Microsoft\.NETFramework" ;registry entry to look in.
    StrCpy $2 0
  
    StartEnum:
      ;Enumerate the versions installed.
      EnumRegKey $3 HKLM "$1\policy" $2
  
  ;MessageBox MB_OK "EnumRegKey $3"
      ;If we don't find any versions installed, it's not here.
      StrCmp $3 "" noDotNet notEmpty
  
      ;We found something.
      notEmpty:
        ;Find out if the RegKey starts with 'v'.  
        ;If it doesn't, goto the next key.
        StrCpy $4 $3 1 0
        StrCmp $4 "v" +1 goNext
        StrCpy $4 $3 1 1
        
        ;It starts with 'v'.  Now check to see how the installed major version
        ;relates to our required major version.
        ;If it's equal check the minor version, if it's greater, 
        ;we found a good RegKey.
        IntCmp $4 ${DOT_MAJOR} +1 goNext yesDotNetReg
        ;Check the minor version.  If it's equal or greater to our requested 
        ;version then we're good.
        StrCpy $4 $3 1 3
        IntCmp $4 ${DOT_MINOR} yesDotNetReg goNext yesDotNetReg
  
      goNext:
        ;Go to the next RegKey.
        IntOp $2 $2 + 1
        goto StartEnum
  
    yesDotNetReg:
  ;MessageBox MB_OK "DotNet Reg stuff was found!"
      ;Now that we've found a good RegKey, let's make sure it's actually
      ;installed by getting the install path and checking to see if the 
      ;mscorlib.dll exists.
      EnumRegValue $2 HKLM "$1\policy\$3" 0
      ;$2 should equal whatever comes after the major and minor versions 
      ;(ie, v1.1.4322)
      StrCmp $2 "" noDotNet
      ReadRegStr $4 HKLM $1 "InstallRoot"
      ;Hopefully the install root isn't empty.
      StrCmp $4 "" noDotNet
      ;build the actuall directory path to mscorlib.dll.
      StrCpy $4 "$4$3.$2\mscorlib.dll"
  ;MessageBox MB_OK "Does $4 exists?"
      IfFileExists $4 yesDotNet noDotNet
  
    noDotNet:
  ;MessageBox MB_OK "DotNet was not found .."
      StrCpy $0 0
      Goto done
  
    yesDotNet:
  ;MessageBox MB_OK "DotNet stuff was found!"
      StrCpy $0 1
    done:
      Pop $4
      Pop $3
      Pop $2
      Pop $1
      Exch $0
  FunctionEnd
  
  Function ConnectInternet
    ClearErrors
    Dialer::AttemptConnect
    IfErrors noie3
  
    Pop $R0
    StrCmp $R0 "online" connected
      Strcpy $R0 "Cannot connect to the internet."
      MessageBox MB_OK|MB_ICONSTOP $R0
      Quit
  
    noie3:
  
    ; IE3 not installed
    MessageBox MB_OK|MB_ICONINFORMATION "Please connect to the internet now."
  
    connected:
  FunctionEnd
  
  ;GetIEVersion
  ; Based on Yazno's function, [url]http://yazno.tripod.com/powerpimpit/[/url]
  ; Returns on top of stack
  ; 1-6 (Installed IE Version)
  ; or
  ; '' (IE is not installed)
  ;
  ; Usage:
  ;   Call GetIEVersion
  ;   Pop $R0
  ;   ; at this point $R0 is "5" or whatnot
  Function GetIEVersion
     Push $R0
     ClearErrors
     ReadRegStr $R0 HKLM "Software\Microsoft\Internet Explorer" "Version"
     IfErrors lbl_123 lbl_456
  
     lbl_456: ; ie 4+
        ReadRegStr $R0 HKLM "Software\Microsoft\Internet Explorer\Version Vector" "IE"   
        Strcpy $R0 $R0 4
        Goto lbl_done
  
     lbl_123: ; older ie version
        ClearErrors
        ReadRegStr $R0 HKLM "Software\Microsoft\Internet Explorer" "IVer"
        IfErrors lbl_error
  
        StrCpy $R0 $R0 3
        StrCmp $R0 '100' lbl_ie1
        StrCmp $R0 '101' lbl_ie2
        StrCmp $R0 '102' lbl_ie2
  
        StrCpy $R0 '3' ; default to ie3 if not 100, 101, or 102.
        Goto lbl_done
        lbl_ie1:
           StrCpy $R0 '1'
           Goto lbl_done
        lbl_ie2:
           StrCpy $R0 '2'
           Goto lbl_done
        lbl_error:
           StrCpy $R0 ''
        lbl_done:
     Exch $R0
  FunctionEnd
  
  ; Checks for current IE version and will try to call
  ; IE6Setup.exe if it exists, otherwise prompt the user
  ; to install it for himself (at least IE5.01 is required for .NET)
  Function CheckForCorrectIEVersion
     Call GetIEVersion
     Pop $R0
     Push $R1
     StrCpy $R1 $R0 1
     StrCmp $R1 '6' lbl_IEOK
     StrCmp $R1 '5' lbl_IE5
     StrCmp $R1 '4' lbl_IEKO
     StrCmp $R1 '2' lbl_IEKO
     StrCmp $R1 '3' lbl_IEKO
     StrCmp $R1 '1' lbl_IEKO
  
     lbl_IE5:
        StrCmp $R0 '5.00' lbl_IEKO
        Goto lbl_IEOK
     lbl_IEKO:
  ;MessageBox MB_OK "IE5.01 or better was not found!"
        ; First check if ie6setup.exe exists
        IfFileExists "$EXEDIR\Setups\ie6sp1\ie6setup.exe" 0 IE6FileNotFound
        ExecWait '"$EXEDIR\Setups\ie6sp1\ie6setup.exe"'
        goto lbl_IEOK
        ; If DotNetFx.exe was not found, try downloading online
      IE6FileNotFound:
        MessageBox MB_YESNOCANCEL|MB_ICONQUESTION "IE5.01 or better was not found. .NET can't be installed without it! $\r$\nWould you like to automatically download and install it? $\r$\n $\r$\nChoose Yes for an automatic download and installation. $\r$\nChoose No, you want to select the download yourself (e.g. in another language than english). $\r$\nMicrosoft's .NET Framework Download page will open up! $\r$\nChoose cancel to abort any action here, you will have to install IE5.01 or better$\r$\nyourself, that can be done for example automatically by the Windows Update $\r$\n(just go to http://windowsupdate.microsoft.com )" IDYES DL IDNO IndirectDL
        goto lbl_IEOK
       IndirectDL:
        ExecShell open "http://www.microsoft.com/downloads/details.aspx?FamilyID=1e1550cb-5e5d-48f5-b02b-20b602228de6" "" SW_SHOWNORMAL
        goto lbl_IEOK
       DL:
        Call ConnectInternet ;Make an internet connection (if no connection available)
        StrCpy $2 "$TEMP\IE6Setup.exe"
        NSISdl::download http://download.microsoft.com/download/ie6sp1/finrel/6_sp1/W98NT42KMeXP/EN-US/ie6setup.exe $2
        Pop $0
        StrCmp $0 success success
        SetDetailsView show
        DetailPrint "download failed: $0"
        MessageBox MB_OK "IE6Setup could not be installed, download was aborted. Please install IE and .NET yourself! $\r$\nRest of the installation went ok, you just need IE for .NET to start XnaSkinningColladaModels!"
        goto lbl_IEOK
      success:
        ; Install, in quiet mode (will do everything automatically)
        ExecWait '"$2"'
        Delete $2
     lbl_IEOK:
  ;MessageBox MB_OK "IE5.01 or better was found .."
  FunctionEnd
  
  ; Check if DirectX9 or better is installed,
  ; will return $0 1 then, else $0 is 0.
  Function IsDirectX9Installed
    Push $0
  
    ReadRegStr $0 HKLM "SOFTWARE\Microsoft\DirectX" "Version"
    StrCpy $1 $0 4 3
    IntCmp $1 9 found.DirectX9 notFound.DirectX9 found.DirectX9
  notFound.DirectX9:
  ;MessageBox MB_OK "can't find direct9"
    StrCpy $0 0
    Goto done
  
  found.DirectX9:
  ;MessageBox MB_OK "found direct9 or better"
    StrCpy $0 1
  
  done:
  ;MessageBox MB_OK "direct9 found value: $0"
    StrCpy $R1 $0
    Exec $0
  FunctionEnd
  
  ; Check if XNA 2.0 or better is installed,
  ; will return $0 1 then, else $0 is 0.
  Function IsXNAInstalled
    Push $0
  
    ;Enumerate the versions installed.
    EnumRegKey $0 HKCU "SOFTWARE\Microsoft\XNA\Game Studio" "v2.0"
      ;if the root directory doesn't exist XNA is not installed
    StrCmp $0 "" notFound.XNA found.XNA
  
  notFound.XNA:
  ;MessageBox MB_OK "can't find XNA"
    StrCpy $0 0
    Goto done
  
  found.XNA:
  ;MessageBox MB_OK "found XNA or better"
    StrCpy $0 1
  
  done:
  ;MessageBox MB_OK "XNA found value: $0"
    StrCpy $R1 $0
    Exec $0
  FunctionEnd
  

  
  
  
• And finally the uninstall section: This section will deinstall your game and get rid of all files in the directory that belong to the game, if there is more, it will not kill any other sub directories or even delete the game directory if there is other data present. If you want to be totally sure that only your game files are deleted, only include files that have been installed in this section via the Delete command. I used 2 RMDir /r commands to delete the Content and ColladaModels directories, which makes things much easier. But if for example the user made screenshots, they will be saved in a /Screenshots directory, which is not deleted here and will stay alive even after the uninstall is complete. There is also another check to make sure the game is still installed at this location by making sure the .exe file still exists. If the .exe file is missing, the uninstaller assumes the game is already uninstalled and it will only remove registry keys, but not any files.
  

  
  ; special uninstall section.
  Section "Uninstall"
    ; REMOVE UNINSTALLER
    Delete $INSTDIR\uninstall.exe
  
    ; remove registry keys
    DeleteRegKey HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\XnaSkinningColladaModels"
    DeleteRegKey HKLM "SOFTWARE\XnaSkinningColladaModels"
  
    ; remove shortcuts, if any.
    Delete "$SMPROGRAMS\XnaSkinningColladaModels\*.*"
    Delete "$DESKTOP\XnaSkinningColladaModels.lnk"
    ; remove directories used.
    RMDir "$SMPROGRAMS\XnaSkinningColladaModels"
  
    ; First check if rocket commander is located here, if not, dont kill stuff!
    IfFileExists "$INSTDIR\XnaSkinningColladaModels.exe" 0 RCNotFound
    ; remove files
    Delete "$INSTDIR\XnaSkinningColladaModels.*"
    RMDir /r "$INSTDIR\Content"
    RMDir /r "$INSTDIR\ColladaModels"
    
    ; Try to kill main folder, but this will fail if there are still some files in it
    RMDir "$INSTDIR"
    goto End
    RCNotFound:
    MessageBox MB_OK "XnaSkinningColladaModels.exe was not found in $INSTDIR, won't delete any files, $\r$\nplease remove directory and files yourself (all registry keys were killed)."
    End:
  SectionEnd
  

  
  

Okay, we have now covered all code of the XnaSkinningColladaModelsSetup.nsi file, download it and check it out for yourself if you want to learn more about it or even use it in your projects. Good luck!

This article is very similar to the original "Skeletal Bone Animation and Skinning with Collada Models in XNA", which is one of the most read articles on my blog. The old article and the SkinningWithColladaModelsInXna project did only work on XNA 1.0, this is the updated version for XNA 2.0. Most of the content of this post stay the same, but the source code and downloads have changed now.

Content

• Video, Screenshot and Downloads
• Introduction and why Collada?
• Class Overview
• Loading Collada files
• Problems with the animation data
• Optimizing vertices
• Optimizing the Optimization
• Adjusting the shaders for skinned meshes
• Post screen shaders for the final result
• References and Links

Video, Screenshot and Downloads

First of all, here is a video for the project I'm talking about here:

Screenshot of the same scene (better quality since youtube videos just suck quality wise):

Before you start reading this article I suggest you download the executable and source code first in case you want to try it out directly while reading. The whole project is based on the RocketCommanderXNA engine and just adds the ColladaModel.cs class, which does all the amazing stuff you see in the video The video above shows shadowing from the XNA Shooter engine, which is not yet released and their fore not included in the downloads. The rest of the code is the same, just the shadow map rendering was removed. More details about shadow mapping in XNA can also be found in my book (plug plug plug ^^).

• The source code is about 2.6 MB and contains projects for XNA Game Studio (which works in Visual Studio 2005 too thanks to XNA 2.0). It was tested on Windows XP and Vista 32 and 64 bit as well as on the Xbox 360. Please not the PostScreenShaders are untested and need some tweaking to work correctly, but all the other things like the model loading and rendering work just fine.
  XnaSkinningColladaModelsSourceCode.zip (2.6 MB)
• The executable is available too, it will install all necessary files like the XNA 2.0 Framework and DirectX files. You will however need a Shader Model 2.0 capable graphic card. You can read the next blog post for more information on how to create such setups :)
  XnaSkinningColladaModelsSetup.exe (2.2 MB)
  
  

Introduction and why Collada?

Note: The rest of this article is from Feb 2007, I have just corrected some errors and updated the source code.

OK, let's get started. I talked a lot about the XNA Content Pipeline and its problems on my blog, and in several recent interviews and also in my upcoming XNA book. For projects like Rocket Commander or even the Racing Game it was sometimes a little bit annoying, but I could do everything I needed by loading X files and adding some features to it to fix the tangents, load the correct shader techniques, etc.
However XNA does not support loading animation data or gives you a way to display them. You have to do all that work by yourself. This includes static mesh animations (like it was used in Rocket Commander, I just left it out in Rocket Commander XNA), but also skeletal animation with bones and skinned meshes.

There is a nice project on CodePlex called XNA Animation Component Library, but it only support FBX and ASCII X files (and recently BVH, ASF and AMC formats) and I could not get any of my test models to work with this library. The main thing missing here is good shader support and it also has a lot of problems with complex animated models. Other than that exporting 3D Models as X files is really a pain in the ass, no matter which exporter you use (Panda Exporter for 3dsMax was good a year ago, I still use it, now kiloWatt X file Exporter for 3dsMax is better, and the Microsoft X File Exporter for Max always sucked). Simple models might work, but the more complex 3D models get and the more meshes and animation data is stored in a scene, the more problems you will have. Sometimes it is not possible to reconstruct everything correctly on your importer side.

Anyway, at the very beginning of the Racing Game development there was no content pipeline in XNA (it was Beta 1) and I implemented loading 3D Models with help of the Collada file format, which is basically just XML and very easy to read. For that reason it was relatively simple to get some 3D data loaded and displayed in the early XNA versions with help of vertex buffers. There were some problems with shader settings and I had to try many different exporters and ended up with the one from Feeling Software. Back then it had still some problems loading shaders and using the correct techniques, but the recent version (3.0.2) is much better and works like a charm.

After Microsoft had implemented the content pipeline and made it possible to load X and FBX files indirectly by going over the content pipeline I had to remove most of my Collada code and re implement the model loading with the new framework. Loading and displaying 3D models was much simpler this way and especially some early unit tests were really simple, but as soon I tried shaders and loading tangents there were a bunch of problems. I reported many bugs back then and it has gotten a lot better, but I still had to fix several issues in the Racing Game and Rocket Commander XNA myself like finding out the correct shader technique and fixing tangent data with help of a custom model processor.

Some things like the level loading in the Racing Game just do not work with X or FBX files because they use splines, which are not exported at all in these formats. Collada came as the rescue again because it is really no big problem for this format. Later versions of the Racing Game removed the Collada level loading and introduced a binary format for the levels, but the importer still accepts Collada files.

Recently I wanted to test a couple of animated models and use skinning since it can often be more useful than static animations and it usually looks much better, especially for organic 3D models. As I said above I could not get anything working with the XNA Animation Component Library and I especially do not like the way they still use the content pipeline and the project is too complicated for me anyway (I just do not need 6 different file formats, I just want one and it should work perfectly with all the features I need).

After some searching I saw some guy called remi from the Collada forum was working on importing Collada models too and I posted some thoughts there too (maybe this was a mistake, I got many emails asking me about tips ^^). Here is the thread about that in the Collada forum. He has provided a test project with some models and it works nice for static meshes without shader information, but that was not really what I was searching for.

This was a month ago and I had not much time working on any of these issues, but after restarting my blog earlier this week I thought this would be a nice topic to talk about. I'm still pretty busy with Arena Wars Reloaded, but I worked a couple of hours every day on this little test program for the past few days. The rest of this article explains the project and into which problems I ran.

Class Overview

Before we go into the details here is a little class overview of the project. As I said before the Rocket Commander XNA engine was used to get up and running with the project without having to re implement the basics. Most of the classes were already in place and had not to be changed. ColladaModel and SkinnedTangentVertex are the 2 new classes and to help us out with the XML loading of the Collada files the XmlHelper class was also brought into the project.

I wrote the ColladaModel file from scratch, but I could reuse some of the static mesh loading code I had done last year. All of the bone and animation loading code was just try and error and I only used the Collada specifications as a source of help, but most stuff had to be tested with the unit tests at the end of the class many many times.

The main program just calls the unit tests TestGoblinColladaModelScene or TestShowBones, there is no real game here, its just a test program. The unit test then calls several Standard Engine classes for doing all the post screen processing, rendering the ground plane, etc. More importantly the ColladaModel class itself basically just provides a constructor and a Render method, everything else is private and will be handled automatically for you. Most people might ignore this, but this is always the most important thing about my classes, the use should be as simple as possible and when I look at the projects mentioned above I really ask myself why people sometimes think so complicated.

The internal Bone class inside ColladaModel is used to store all the bones in a flat list, but each entry has a parent and a list old children bones. This way the list can be used both in a simple for loop, but you can also go through it recursively (which is obviously slower and often more complicated). We will talk about the loading process in a minute.

All the mesh data is stored in vertices, which is just a list of SkinnedTangentVertex structs. The SkinnedTangentVertex struct is very much like the standard TangentVertex struct used in Rocket Commander XNA, but it has 2 new members: blendIndices and blendWeights. Both are in the form of Vector3 and their fore can hold 3 values allowing us to interpolate up to 3 bone influences for each vertex in the shader. More is often not required and we have to re normalize all bone weights anyway, so skipping the least important bone weights is not a big deal. My test models use mostly max. 2-3 influences. Please also note that the vertex shader has now a lot more work to do with all that skinning and you should really optimize it as much as possible. Both the number of vertices we have to process is important (we will talk about optimizing that in the optimizing part of this article) and also the number of instructions the vertex shader has, both numbers should be as low as possible. The GPU is really fast processing this data, but if you do not have animated geometry with bones, there is no reason to let it process all that data (which can make the vertex shader 2-3 times longer and slower). The ground of our test scene does not use a skinning shader as an example.

And finally there are some additions to the ShaderEffect class. First of all we got a new shader called "SkinnedNormalMapping", which does the same thing as the normal mapping (or parallax mapping) shader, but it has an array of 80 bone matrices we can use for skinning. These matrices are set with help of the SetBoneMatrices method in the Set Parameter region of ShaderEffect.cs.

Loading Collada files

Before we go into the details of the loading process, lets make sure we read the summary of the class first because it clearly states what we can do and can't do with this class. This is just a test project and I wanted to make things as simple as possible for both you as the reader and for my requirements.

   /// <summary>
   /// Collada model. Supports bones and animation for Collada (.dae) exported
   /// 3D Models from 3D Studio Max (8 or 9).
   /// This class is just for testing and it will only display one single mesh
   /// with bone and skinning support, the mesh also can only have one single
   /// material. Bones can be either in matrix mode or stored with transform
   /// and rotation values. SkinnedTangentVertex is used to store the vertices.
   /// </summary>
   class ColladaModel : IDisposable

OK, with that said let's go directly into the loading code, which is located in the constructor of this class. All variables used in these class are just for internal use, all you need to know are the vertices and bone lists, which I have already mentioned, and the vertex and index buffers, which are used for rendering. All the rest of the variables are just there to help us loading the Collada file (don't worry, there are not many variables anyway and most methods are short too).

    #region Constructor
    /// <summary>
    /// Create a model from a Collada file
    /// </summary>
    /// <param name="setName">Set name</param>
    public ColladaModel(string setName)
    {
        // Set name to identify this model and build the filename
        name = setName;
        string filename = Path.Combine(ColladaDirectory,
            StringHelper.ExtractFilename(name, true) + "." +
            ColladaExtension);

        // Load file
        Stream file = File.OpenRead(filename);
        string colladaXml = new StreamReader(file).ReadToEnd();
        XmlNode colladaFile = XmlHelper.LoadXmlFromText(colladaXml);

        // Load material (we only support one)
        LoadMaterial(colladaFile);
        
        // Load bones
        LoadBones(colladaFile);

        // Load mesh (vertices data, combine with bone weights)
        LoadMesh(colladaFile);

        // And finally load bone animation data
        LoadAnimation(colladaFile);

        // Close file, we are done.
        file.Close();
    } // ColladaModel(setFilename)
    #endregion

As you can see first of all the filename is constructed and we just load the file as a text file and throw it to the XmlHelper.LoadXmlFromText helper method (which just uses the existing XmlDocument functionality to load XML from a string). We now get the main Collada node, which contains all the children nodes we need for loading the materials, bones, meshes, etc.