Buyaa, I'm Microsoft MVP once again in the XNA/DirectX category for one more year (2009/2010). I have been an XNA MVP since 2006 and I'm still very proud of it :)

I was pretty busy this year with our current game project "Fireburst" at exDream (a racing game for Xbox 360, PS3 and PC using the Unreal3 engine, more about it soon) and theirfore I did not do many other things (except writing some iPhone games, starting to develop my own dynamic language and some tools). But once that project is done in 1-2 months, I will do lots more XNA fun stuff and hopefully XNA Community Games (now called Indie Games on the Xbox 360, I really hate that term) will be available in Germany so I can finally review and submit some games myself. I'm still pretty fit in XNA 3.1 and DirectX 11 (played around with it a lot in March), but for XNA I'm still waiting for availablilty in Germany and for DirectX 11 I'm really waiting for some cool hardware at the end of this year.

At exDream we recently also had some interesting discussions about using .NET for PS3 (recently possible thanks to Novell), Xbox 360 (hello XNA) and PC or even more dynamic script languages for upcoming projects. Maybe also including some other promising platforms such as iPhone (also .NET able thanks to Unity), Android, PSP Go, WII, whatever, but that all should depend more on the game and if we are able to manage so many platforms. Working with XNA was almost no extra work to make a game run on the Xbox 360 plus the PC, the game just has to fit and you obviously should allow control with the Xbox 360 controller. Our multiplatform game Fireburst is also not that bad to develop for because most issues are handled by the great Unreal3 engine (except the fact of course that it is all ugly C++ and UnrealScript code, which just looks like C++ anyway and even has to be compiled), but it is still quite a lot more work than just doing a PC only game, especially because of optimizations and testing required for all platforms.

Enough rambling, I'm going to celebrate this day by installing Windows 2008 server (omg) because our pre-release version we had on there just ran out (warg) .. stupid thing has to be completely reinstalled, no upgrade option ..

I want to quote from this great article I just read: Singletasking: The Next Trend in Web Working?

"Singletasking is just what it sounds like: approaching and tackling one task at a time, sequentially, instead of trying to do a whole bunch of things at once, as has become de rigeur in our modern multitasking age. If you’re like me, the thought is probably at least a little refreshing, and maybe more than a little appealing right off the bat.

The principle is sound. Take on one task at a time, and don’t begin another until the one you’ve already started is complete. It sounds simple, but you know as well as I do that actually implementing that kind of thing in real life will take a lot more effort than you might first think. For one, it means ignoring any urge to procrastinate, and making sure that you prioritize very carefully in advance, lest you realize too late that what you thought was most urgent actually could’ve taken a back seat to something else."

The article goes on with tools used to track work tasks and even ideas on only use one Tab in Firefox and only using one monitor. I can hardly agree with that, I currently have 10 tabs open (after reading all emails and stuff on the web) because I don't have time right now to finish reading those sites or because some things are still pending (waiting for a response and keeping the tab open as a reminder). I also currently look at 3 monitors. Well not really, I only write this on my right monitor, but I have my main task (some optimization work) on my main screen in the center in Visual Studio. I really hate putting anything above Visual Studio because then I won't focus on writing code, but do other stuff instead. I only do it if I have to write some document, have to do some web research or reading some documentation.

While I totally agree with doing (and finishing) one task at a time (that's actually how my TODO list works that I use in every single project for the last 6+ years), I just can't agree with just having one monitor (or just using one at a time like the writer of the article proposes). For example when opening an editor like for our Unreal3 game, you absolutely need 2+ monitors because else you just have 5 overlapping windows and spend half of your time moving them around trying to see whats behind them.

In Visual Studio I only have the Solution Explorer sometimes and some Find Results or Console Windows on my secondary screen, but all the screens are filled with useful stuff anyway. I also have basically 2 screens at once at my primary screen anyway because I always look at 2 concurrent tabs in Visual Studio (really can't live without it). That does not mean that I really multitask. I ALWAYS only work on one tab, but I can quickly look over, see all the variables and methods that I need and I can code much faster that way.

If you want to have some distractions when working I suggest only checking them out after completing a task. I usually do it as a reward kinda thing, like playing a game I wanted to test out only after I finished my current task. And then continue with the next task.

As Jamie Cansdale posted in my comments ^^:

There's now a version of TestDriven.Net what works with VS 2010:
http://weblogs.asp.net/nunitaddin/archive/2009/06/03/testdriven-net-2-22-support-for-visual-studio-2010-beta-1.aspx

Finally time to use VS2010 a little bit more, without TestDriven.Net it was just unusable before.

I worked all weekend on AI and game logic scripts in Lua and UnrealScript .. would like to talk about it, but since our game project is still not announced yet, I'm not allowed to .. grr ..

Also reading (or hearing) right now: http://www.brainrules.net/
And working through the great new release of the DLR and the new example script language Sympl: http://www.codeplex.com/dlr

Download link for CR_RandomBackgroundChanger now at CodePlex.

I used several tools over the years to change my background image automatically every few minutes. I even collected over 50k desktop images over the years ^^ While this is great and always fun to watch at changing backgrounds every few minutes, most of the time I do not even notice because all my screens are completely covered by windows.

As a programmer I have Visual Studio open most of the time, especially at work, where distractions like changing background images would have the biggest impact. Luckily Visual Studio has a boring white (or black for me) background and will not distract us that much. It would be terrible if Visual Studio would have changing background images that even make it harder to look at code and could be distracting having a background image changing every 5 minutes. Let's explore that!

First of all: How to even change or set a background image directly in the Visual Studio code window?
There is a great collection of gadgets from the SlickEdit guys, called Free SlickEdit Gadgets, which are completely free to use and include a feature to set a background image in the code editor. Searching for this on a search engine like Google isn't that easy, it is easier to find if you search for "Visual Studio Dancing Banana" :)

After you have installed the SlickEdit Gadgets (which currently only work on VS2005/VS2008, not on VS2010 yet), you should see a SlickEdit menu item. From there you can select Editor Gadgets, which just opens the Visual Studio Options dialog. There you can go to the Editor graphic tab and select a background image to be displayed in the editor. I use the following settings:

• Display an image: Checked
• Image textbox: The selected image
• Tile Radiobutton checked (use bottom left if you just want a small transparent image there, IMO not distracting enough!)
• Lock the image location checked
• Transparency checked and set to 75% (which means 25% visibility, 75% of your background color remains).

Now your code window might look like the following:


While this is kinda fun, it is not distracting enough yet. We need changing background images and preferably a way to quickly change the editor background image with a click at anytime in case it doesn't fit. First of all we need to find out where SlickEdit saves this background image. After searching for some SlickEdit setting file and not able to find any, I checked the registry and found this key: HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\SlickEdit\EditorGadgets\ImageFilename This is where the image name is saved. Just for fun I changed it to "C:\Users\Public\Pictures\Sample Pictures\Penguins.jpg", but obviously nothing happend because there was no reason for Visual Studio to reload and apply any new settings from the Options dialog. So how to we force that to happen?

Well, after trying to find some solution for that for a while, I gave up. There is no event I can trigger to update this and there is simply no functionality in the SlickEdit Gadgets to get this update working.

Since I would need to write something to set a new background image every few minutes anyway and this won't be easy or even impossible with my little Wallpaper Changer app I decided to write a VS addin myself. I started a new CodeRush plugin (which also runs on the free DXCore framework) and called it CR_RandomBackgroundChanger (a little bit too long for CodePlex, so there it is just called VSBackgroundChanger). Next I used the EditorPaint event and wrote the following code to display an image in the code editor:

 private void BackgroundChangerPlugIn_EditorPaint(EditorPaintEventArgs ea)
 {
     //Log.Write("BackgroundChangerPlugIn_EditorPaint");
     //tst: ea.DrawLine(1, 1, 50, Color.Red);
     if (currentBackgroundImage == null)
         currentBackgroundImage = Bitmap.FromFile(
             @"C:\Users\Public\Pictures\Sample Pictures\Desert.jpg");
     ea.Graphics.DrawImage(backgroundImage, 0, 0);
 } // BackgroundChangerPlugIn_EditorPaint(ea)
 

This produced terrible results. The image got displayed for a fraction of a second and then was replaced by the default background color. Using other events from the StandardPlugIn class in the CodeRush framework like EditorPaintBackground, EditorPaintForeground, EditorPaintLanguageElement, EditorvalidateClipRegion, etc. also did not help much, the background image was usually not displayed and even if it was flickering terribly and disappearing all the time.

So CodeRush was not very helpful in this particular instance, but no reason to give up. Actually I gave up that day after trying out so many things for many frustrating hours. The next day I tried something different: Use the NativeWindow class to intercept any event that is send to the Visual Studio code editor window (called View in CodeRush btw). Basically you derive from NativeWindow and then call the AssignHandle method to set the window handle (I just do it in the constructor). From then on you will get all WndProc events and you can decide whether to pass it on to the original WndProc or do something yourself.

public class ActiveViewBackgroundHelper : NativeWindow
{
    #region Constructor
    /// <summary>
    /// Create active view background helper class, each document view
    /// will get an instance of this class to handle the background drawing!
    /// </summary>
    public ActiveViewBackgroundHelper(IntPtr windowHandle)
    {
        base.AssignHandle(windowHandle);
    } // ActiveViewBackgroundHelper(windowHandle)
    #endregion

    #region WndProc
    /// <summary>
    /// WndProc, we are only interested at the WM_PAINT event!
    /// If a paint happens and we got a valid  we will call DrawBackgroundImage
    /// </summary>
    /// <param name="m"></param>
    protected override void WndProc(ref Message m)
    {
        // Your own code goes here
        base.WndProc(ref m);
    } // WndProc()
    #endregion
} // class ActiveViewBackgroundHelper

Now I was able to intercept all WM_ERASEBKGND and WM_PAINT messages. WM_ERASEBKGND turned out to be non-relevant because all the rendering happens in WM_PAINT. Otherr events like WM_NCPAINT and all the code rush events happened between a single WM_PAINT event anyway. So I had to dig deeper into WM_PAINT. I could not render a background image at the very start of WM_PAINT and if I aborted rendered right there, it would be displayed in the editor. But there was no text anymore making VS just useless. I had to figure out how to render text on top of my image by copying all the VS editor content to a bitmap and then later drawing it on top of my background image. To make all the flickering go away by drawing into a targetBitmap and then displaying that at once to the window (everything else flickers). There is also a lot of other optimizations and tricks in there. We will get back to the code in a little bit.

After working on this for the better part of the last weekend, I finally got it working and put some extra features in this CR_RandomBackgroundChanger addin, which you can see in this nice option screen for it. Note that I have a black background in Visual Studio, if you have a white one the preview image would be white with black text and it would have 25% of the background image mixed in.



You can download the addin and the source code at http://vsbackgroundchanger.codeplex.com/

I'm trying out CodePlex for this addin. I will probably improve it a little in the future and upgrade it to VS2010. I will also put up other projects on CodePlex like CR_Commenter and maybe some of my XNA games. I also plan to work a little bit more on my language and then put it on CodePlex too when I got a alpha version working with all basic features.

In the option screen of CR_RandomBackgroundChanger you can specify 5 settings:

• Enabled: Whether to use this addin at all. This allows you to quickly disable the addin and keeping all your previous settings once you need more distractions again.
• Directory: All images in this directory and all images in all the sub directories will be used for displaying random background images in your Visual Studio code editor. Please note that searching for 100 000+ images can take a few seconds (I tested it with that many), but the subsequent VS startups and every background image change will be very fast because of all the caching involved. The disadvantage of all the caching is that if you add new images or even delete the whole images folder, you need to update this directory to get all those changes reflected by the addin by either deleting the cached ImageFilenames.txt in your %TEMP% directory or by just selecting the same directory again.
• Transparency (default: 25% opacity): This is the most important setting because having colorful background images at 100% opacity can often make the already colorful editor text unreadable. If you know that your background images have mostly the same color as your code background you can use higher settings (for example I used some black only images with some stuff on the right in the beginning, while this was cool at first, it got boring after a while because I'm used to changing background images every few minutes). If you have photos or other images that do not really fit to having text on top of them use even lower opacity settings (e.g. 10%) to make Visual Studio text easier to read. You can mostly still see the background image if your code window is big enough and you have some empty lines in there :)
• Change image every x seconds: Use this setting to allow the addin to change the background images every 5 minutes (or use whatever number of minutes you like). Note that using 0 minutes will result in constantly changing backgrounds while you are typing and scrolling, not really sure if this is useful, but was good for testing ^^
• And finally the last settings is to use different background images for different document windows. If you have 20 documents opened in Visual Studio, every one of them will get a different background image (which can then change every 5 minutes too). This even makes it easier to figure out where you are, I probably need to improve the Tab-Rendering too to make it more useful like the ColorfulTabs addin for Firefox. And this is of course great fun, usually I get annoyed by having too many documents open so I often close them all after I get 50, but now I really like having many different background images open.

The option screen also features the Preview box, which lets you see your selected background images and tweak the transparency setting until it looks right for you. Once you close the Options screen all settings will be applied to your VS code editor. Please note that this addin only works whenever a WM_PAINT event happens. This is usually only in the active document you are working on. Most of the time the background image will stay even if you change to another tab on the other side (if you use 2 tabs at once, else you will never notice this anyway). Sometimes you see some black or white background lines drawn on top of your background image because just some text was updated, but this happens very rarely. I hope you enjoy the addin, I will probably use it for a long time and I will port it to VS2010 once CodeRush runs there or I'm using VS2010 (still waiting for a working TestDriven.net version). Hopefully implementing this addin with the MEF (Managed Extensibility Framework) is not hard.

Let's go back to the code. Last time I stopped talking about the WndProc method, which is most important for catching and handling the WM_PAINT event. From here we can call our DrawBackgroundImage method, which does all the magic. Please note that the source code for the addin is a lot more complex than the code presented here. This is mostly because of heavy optimizations and a lot of caching to make the code run as fast as possible (one of the most important things about this addin, it would be unusable if it makes VS slow).

#region WndProc
/// <summary>
/// WndProc, we are only interested at the WM_PAINT event!
/// If a paint happens and we got a valid  we will call DrawBackgroundImage
/// </summary>
/// <param name="m"></param>
protected override void WndProc(ref Message m)
{
    // Just make sure we mark the erase background message as handled.
    // Will not do anything anyway (all painting done in WM_PAINT).
    if (m.Msg == (int)WindowsMessages.WM_ERASEBKGND)
    {
        // Mark event is already handled
        m.Result = (IntPtr)1;
    } // if

    // Handle the paint event
    if (m.Msg == (int)WindowsMessages.WM_PAINT &&
        // Are we currently drawing our own background? Then make sure we do
        // not handle this and use the default message handling instead!
        drawingBackground == false &&
        // Make sure we got a view, else we don't have a windows handle!
        CodeRush.TextViews.Active != null &&
        // Check for invalid hwnd, then we can't paint backgroundImage!
        (int)CodeRush.TextViews.Active.Handle != 0)
    {
        TextView view = CodeRush.TextViews.Active;
        IntPtr activeWindowHandle = view.Handle;
        Rectangle rect = new Rectangle();
        Win32.GetUpdateRect(activeWindowHandle, ref rect, false);
        // Only proceed if we have a valid rect
        if (Win32.IsRectEmpty(ref rect) == false)
        {
            // Make sure we mark this flag so subsequent calls to WM_PAINT
            // will actually just paint the normal stuff, not just our
            // background rendering!
            drawingBackground = true;
            DrawBackgroundImage(activeWindowHandle, rect);
            drawingBackground = false;

            // Mark event is already handled
            m.Result = (IntPtr)1;

            // Do not call base.WndProc, we don't want to process it here!
            // Instead we invalidate inside DrawBackgroundImage and force
            // a new WM_PAINT event inside there, which will be executed
            // because drawingBackground is still true while in there.
            return;
        } // if (rect)
    } // if (WM_PAINT)

    base.WndProc(ref m);
} // WndProc()
#endregion
    

And finally the DrawBackgroundImage method, which copies the current editor text as an image, applies transparency to it. Then we draw our background image and the now transparent editor text on top into a helper targetBitmap, which is finally displayed on the screen at once (else we would get flickering issues). Please note that in the source code you can download there is a some testing code, e.g. for drawing bitmaps with transparency using ImageAttributes and SetColorMatrix, which works great, but is just too slow. Our approach is to pre-calculate the opacity in the backgroundImage and use that over and over again (3-4 times faster). There is still a performance penalty for all this painting, but any future optimizations will be hard. I recommend using 2 tabs, rendering is twice as fast (because only one side is updated when you type or scroll) and you should have a fast PC :)

#region DrawBackgroundImage
/// <summary>
/// This method draws the background and is called from WndProc whenever
/// it intercepts a WM_PAINT message. Again, some caching and confusing
/// optimized code is in here too, again for getting good performance.
/// </summary>
private void DrawBackgroundImage(IntPtr activeWindowHandle,
    Rectangle rect)
{
    // Create an image for storing the orginal editor screen.
    Bitmap sourceImage = new Bitmap(rect.Width, rect.Height);
    // Always create new graphics object, else we won't have current data
    Graphics sourceImageGraphics = Graphics.FromImage(sourceImage);

    // And grab current editor window content and copy it to it!
    IntPtr hdc = sourceImageGraphics.GetHdc();
    Win32.PrintWindow(activeWindowHandle, hdc, 1);
    sourceImageGraphics.ReleaseHdc(hdc);

    // Next find the color on the bottom right and use it as the
    // transparent color! Note: Antialasing will cause artifacts, make sure
    // the backgroundImage fits to the background color, e.g. by using a
    // lot of alpha transparency (<25% visibility)!
    backgroundColorPixel =
        sourceImage.GetPixel(rect.Width - 1, rect.Height - 1);
    sourceImage.MakeTransparent(backgroundColorPixel);

    // Create target image where we wanna paint to, this is important
    // because drawing directly to the VS window will produce flickering!
    Bitmap targetBitmap = new Bitmap(rect.Width, rect.Height);
    Graphics targetBitmapGraphics = Graphics.FromImage(targetBitmap);

    // Clear background with pixel color
    //not required, we draw solid image now (use this for transparent drawing):
    //targetBitmapGraphics.Clear(backgroundColorPixel);

    // Draw background image (tiled and transparent if specified)
    float transparency = Options.transparency / 100.0f;
    Image backgroundImage = GetBackgroundImage((int)activeWindowHandle,
        backgroundColorPixel, transparency);
    
    // Do the drawing as many times as we need to tile to fill everything!
    for (int y = 0; y < rect.Height; y+=backgroundImage.Height)
        for (int x = 0; x < rect.Width; x += backgroundImage.Width)
        {
            targetBitmapGraphics.DrawImage(backgroundImage, x, y);
        } // for for

    // Now draw source image on top (text and foreground stuff), else we
    // would only see our distracting background image and while that is
    // fun, we sometimes still need to be productive and see the editor
    // text ^^
    targetBitmapGraphics.DrawImage(sourceImage, 0, 0);

    // Finally draw on the VS window, but only do one single draw here
    // to make sure we do not have any flickering!
    Graphics graphics = Graphics.FromHwnd(activeWindowHandle);
    graphics.DrawImage(targetBitmap, 0, 0);

    // This is important: Validate the rect so all this can now be
    // displayed! All WM_PAINT calls during this method will
    // be handled normally (without our background painting), which is
    // used for normal updates because we mess everything up ^^
    Win32.ValidateRect(activeWindowHandle, ref rect);

    // Dispose everything we do not need anymore
    // Note: We always have to create new graphics and dispose them here,
    // else updating sourceImage, targetBitmap, etc. does not work!
    if (graphics != null)
        graphics.Dispose();
    if (sourceImageGraphics != null)
        sourceImageGraphics.Dispose();
    if (targetBitmapGraphics != null)
        targetBitmapGraphics.Dispose();
} // DrawBackgroundImage(activeWindowHandle, rect)
#endregion

For more information check out the source code, the important code for the addin is in ActiveViewBackgroundHelper.cs and the Options and PlugIn classes. The rest of the classes are just helpers for logging, string operations, random methods and Win32 helpers for some calls.

Download this addin at http://vsbackgroundchanger.codeplex.com/

I hope you will enjoy this addin. Here are 2 more screens from using this addin with different background colors and images:

White background theme (25% opacity):


Black background theme (25% opacity):

Last week when I installed VS2010 I played around with it for a few minutes, but I've been very busy at work because our game is at its final stage and we got a lot of heat going on (this is not a hint, no no, no hint from me, I'm not allowed to talk about it. Damn it, this could be a hint).

Currently the most interesting new feature besides the cool VS2010 IDE is the Parallel Extensions for me. Sadly the IDE still unusable for real work IMO because all the addins just don't work, there isn't even a fix for TestDriven.net yet, Jamie Cansdale is probably busy too ^^

VS2010 support for parallel programming goes beyond just adding a few extra classes in .NET 4.0: You also got a great IDE implementation which lots of useful features and new tool debugging and profiling windows for checking out parallel tasks, threads and the scheduling. Next there are native C++ libraries that work with Parallel Extensions too (using lambda functions) and work good together with STL. You can check out all the new features at the official VS2010 page!

Let's take a quick look on how to use these Parallel Extensions. I wrote this in a few minutes last week, I was just too lazy (I mean busy of course) to post it yet, it is obviously very simple stuff, but was still useful to test out some of the new parallel IDE features and check out some new .NET 4.0 classes. First of all let's do a boring foreach loop, which displays numbers from 0 to 9, which get added to expectedTotalNum. This should obviously be 45 because sum(0..9)=45. Later we will do some parallel foreach adding and check if we got the same result. Since it does not matter in which order we add these numbers, it is also a good test to just start a bunch of parallel tasks and let them do their work. Obviously you would never write parallel code just to add some numbers, but this should illustrate the point and it does not hurt your performance much anyway (as long as you have a few lines of code executing that take more than a few instructions).

 // Initialize a list for some parallel testing :)
 List<int> someInts = new List<int>();
 for (int num = 0; num < 10; num++)
   someInts.Add(num);

 // Print out numbers sequentially
 int expectedTotalNum = 0;
 foreach (int num in someInts)
 {
   Console.WriteLine("sequential num=" + num);
   expectedTotalNum += num;
 }

 Console.WriteLine("expectedTotalNum=" + expectedTotalNum);

This outputs the obvious sequential adding of 0 to 9 to expectedTotalNum, which is 45 at the end of the loop:

sequential num=0
sequential num=1
sequential num=2
sequential num=3
sequential num=4
sequential num=5
sequential num=6
sequential num=7
sequential num=8
sequential num=9
expectedTotalNum=45

Now let's do the same in parallel, just by replacing foreach with Parallel.ForEach:

 // And do it with the new Parallel Programming classes in .NET 4
 int totalNum = 0;
 System.Threading.Parallel.ForEach(someInts, num =>
   {
     Console.WriteLine("parallel num=" + num);
     totalNum += num;
   });
 Console.WriteLine("totalNum=" + totalNum);

While the result is still the same because totalNum is 45 at the end of this loop, we get there in a different way. As you can see this is a little bit confusing at first because the adding does not happen sequentially anymore, but in parallel instead. Also note that this output can change when you execute it again and can even be much different on different platforms with different number of CPUs:

parallel num=0
parallel num=2
parallel num=5
parallel num=7
parallel num=8
parallel num=9
parallel num=6
parallel num=3
parallel num=4
parallel num=1
totalNum=45

Okay, good stuff so far, but you might not always have a foreach loop and you might not want to wait for it to complete anyway. Maybe you just have some work tasks that need to be executed, no matter in what order and you might not even care if they complete their work right away or in a little bit. A method could just add some work that has to be done and then return while the work tasks are executed in the background. This is when you would have used the Thread or ThreadPool classes in the past, which are great on their own, but you always have some setup code and it was hard to test and you could not use them all over the place because setting up threads is a costly operation and if your work is just few lines of code, it was always a better idea to execute it right away. But fear no more, now you can just create new tasks with the new Task class in the System.Threading namespace. This is a much easier job and has many advantages because .NET 4.0 handles all the thread creation for you, will reuse threads once they are completed with their task and all this works in a very performant way. Setting up tasks is a little bit more work than just executing Parallel.ForEach, but it allows much greater flexibility and you can add more tasks from whereever you are. Tasks can even have children and you can have a lot of complex code using all these tasks. Testing multi-threaded code is obviously harder than just writing sequential code, but with all the great additions to the IDE, it is now easier than ever with the new Parallel Tasks window and by checking out the Parallel Stacks, which shows you all the running threads and were all the task code is.

The following code will create 10 tasks and do the same thing as above, but with several additions. To be able to wait for all the tasks to finish we will add all 10 tasks to the allTasks list. We also have to make sure that our local foreach variable does not change while we are executing tasks because the foreach loop will quickly create all tasks, but might not execute them right away theirfore using num can cause problems. Instead we just create a local copy of num and use that instead, which can't change because we never increase it like we do with num. Finally we add some boring thread sleeping to make it easier to debug this code and check out whats going on by adding breakpoints. Without the sleep the code still works, but checking out the Parallel Tasks and Parallel Stacks windows will most likely give us no results or only the last few tasks that are still being executed at the end of the foreach loop because the tasks are so simple and executed very quickly. We even wait a little after the foreach to make sure all the tasks have been added and are being executed right now or are scheduled (waiting for execution).

 // And finally some tasks, yeah!
 totalNum = 0;
 var allTasks = new List<Task>();
 foreach (int num in someInts)
 {
   // We need a local variable for num because num itself can change
   // at the end of this loop before the task might even be executed!
   int numToBeAdded = num;
   allTasks.Add(Task.Factory.StartNew(delegate
   {
     totalNum += numToBeAdded;
     Console.WriteLine("Adding " + numToBeAdded + " in task with id=" + Task.Current.Id);
     // Wait a little for checking out the tasks in the new Tasks window in VS2010!
     Thread.Sleep(1000);
   }));
 } // foreach
 Console.WriteLine("Done with foreach loop. Tasks might still be pending");

 // Wait a little for all tasks to start
 Thread.Sleep(100);
 Task.WaitAll(allTasks.ToArray());

 // And finally return the result (45 again if everything worked)
 Console.WriteLine("totalNum=" + totalNum);

First of all the results again, which is 45 again. This only work with numToBeAdded, if you use num instead it will sometimes give you different results because at the time you execute a task num already might have changed, especially if you have more tasks than CPUs used for execution and time consuming code like Console.Writeline or even a Thread.Sleep is in there!

Adding 1 in task with id=1
Adding 7 in task with id=6
Adding 8 in task with id=7
Adding 6 in task with id=5
Adding 4 in task with id=3
Adding 3 in task with id=4
Done with foreach loop. Tasks might still be pending
Adding 9 in task with id=8
Adding 0 in task with id=10
Adding 5 in task with id=2
Adding 2 in task with id=9
totalNum=45

As you can see this even looks more confusing than the way Parallel.ForEach added those numbers because while we might create those tasks sequentially (number 1-10) it does not mean there are also executed in exactly that way. To make it easier to check these things out there is the Parallel Tasks window, which shows the following after starting all 10 tasks by adding a breakpoint after Thread.Sleep(100) just before we wait for all tasks to complete:



Not only do we see all over our 10 tasks here, we also see right away that 8 of them are currently being executed (because I have 8 CPUs with my hyper-threaded i7) and all of them are waiting because of our stupid Thread.Sleep(1000) we added for each of those tasks. A second later those tasks are done and the last 2 are also executed, most likely with 2 thread ids already created earlier. You can click on each task and see where it is currently executing in the source code and you can also check out all the thread information in the normal Threads window. But even more useful than the Parallel Tasks window is the Parallel Stacks, which shows how all this code is related, which task or thread created which new task and so on:



All good stuff. While I already have some ideas how to use this on some of my current projects, I have not ported anything to .NET 4.0 / VS2010 yet because of the many issues I have with the IDE (no addins, color theme not really working, I always have to reset it when starting VS2010, also I don't like that the project and solution formats have changed so I cannot easily switch back to VS2008, etc.). But hopefully more and more addins will work for VS2010 and some of the issues are fixed, then it will be great to use all this new .NET 4.0 stuff (dynamics, parallel extensions, MEF, etc.).

VS2010 beta came out a few hours ago, installing it took way too long (over an hour, VS2008 takes less than 10 min for me) and it seems some addins don't install or just don't work, but other than that everything else seems to work great:

• TestDriven.NET provides an VS2010 option and it will appear in Addins, but the Menu Items will not appear anywhere. Just use keyboard shortcuts and everything still works :) .. at least it worked for a short while (see below). Hopefully Jamie Cansdale fixes this soon because without TestDriven.NET I can't really use VS2010 right now. MS provides its own unit testing framework, which is implemented into VS2010 and even available in the Pro edition now (was only in more advanced versions before), but I still don't like it. It does not output anything to the console except exceptions; it is not possible to just start some Ad-Hoc unit test and for all my functional tests it is also useless. For just "normal" unit test projects VS test integration is quite ok, but I would suggest trying out NUnit or xUnit together with TestDriven.NET, which is just more flexible, easier to use and improves productivity IMO.
  
• Completely unable to install any CodeRush version (neither 2, 3, 9.1, Xpress), so I'm also unable to use my CR_Commenter :( hopefully DevExpress provides a version soon, they always were very quick with early VS2005 and VS2008 support!
• Most other addins I tried also did not install or just do not appear anywhere (only in VS2008)
• One of my own addins I wrote 2 years ago for VS2008 also does not appear, it probably needs to be configured to appear in the VisualStudio/10 reg key!

Most annoying:

• Even though Rico Mariani (the .NET performance god) is helping out the VS team to give VS2010 better performance, it currently lags all over the place. I had 10 second delays in Options, Add References and Renaming dialogs already. Seems to be always an issue with opening up stuff for the first time. After a while everything seems to be fast, but I would not say this is faster than VS2008 yet.
• Only tried this at my home PC till now, but everytime I close VS2010 I cannot open it up again until I restart (or at least log off/log on) again, which is REALLY annoying. I will just keep VS2010 open now all the time and hope it never crashes ^^ The start splash-screen is also messed up and completely black. Maybe this is only on my home PC because I tried to install so many addins, will try this tomorrow at work too.
  Update 2009-05-18: I just found out what was causing this. Some addins tried to load, but failed for some reasons and then the addin loading code locks up or whatever and VS never finishes starting up (the process devenv.exe is still there, but you just don't see any IDE window, they are never created). It seems to be related to the TestDriven.NET-2.21.2448 version, I also tried TestDriven.NET-2.20.2438 - same problem :( (Jamie Cansdale, the creator of TestDriven.NET was also recently blogging about this). Then I used , which also already has VS2010 support and that seems to work fine (no menus, but keyboard shortcuts work fine). I posted more details on Jamie's blog!
  

Every project I have opened, had to be converted to the new VS 10 format, which is kinda stupid because it seems nothing has changed. All the code works just fine and no changes were required. I will start using some .NET 4.0 features and test out stuff in the next days, especially using Parallel Programming and MEF (Managed Extensibility Framework). Probably will play around with the C# 4.0 dynamic keyword a bit also, but can't think of anything yet where I would really need it. Maybe to talk easier to IronPython or my own DLR language ..

Cool features:

• Start page is pretty good, easily customizable, but I will probably never see it as I setup my VS to always load the last solution (like my Firefox ^^)
• It is easy to drag tabs around and move them to different screens. Finally I can use some multi-monitoring.
• Old exported VS2008 settings also work just fine in VS2010, even coloring :) For some strange reason the background color is always white when I start VS2010 for the first time, then after going to options and closing it, it will be the correct color (black for me).
• Overall the IDE is very nice and fits perfectly into my color theme. Its also a lot easier to see whats going on, which tab is selected and positioning tool windows all over the place is better than ever before.

Go check out VS2010 yourself, available on MSDN right now and on MS site on Wednesday for everyone else.

While I'm waiting for VS2010 beta today (refreshing blogs and MSDN every few minutes :D), I just found out that you can have your Visual Studio Tabs be automatically ordered by adding the following registry key in HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0:

• UseMRUDocOrdering: 1 (DWORD)

Now when you select any tab in Visual Studio it will automatically move to the very front thus keeping your recent documents at the start. It is kinda weird at first, but it seems to be a very useful feature. I like it already :)



In case you don't want to create the registry key yourself and want me to help you out modifying your registry (always a safe idea), just execute this file to do the job:

• VS9UseMRUDocOrdering.reg (.26 KB)
• Update: Does not seem to work yet in VS2010, maybe use /10.0_Config key instead?: VS10UseMRUDocOrdering.reg (.26 KB)

PS: Since my brother is running a TOR exit node server Google does not like when I do searches anymore because it thinks I search too much stuff now and might be infected with viruses and trojans (yeah, sure ..). I could use Google Custom Search sites like Blackle.com, which return the exact same results as Google, but for now I'm trying out Yahoo once again (was using it a lot almost 15 years ago, when there was no Google). Sometimes Yahoo is stupid and does not give good answers, e.g. searching for "year yahoo was launched" gives you random answers on Yahoo, but google returns the Yahoo Questions link with the correct answer as the first result, wtf?

However, most of the time you do not get stupid SEO optimized search results and totally messed up product searches you get when using Google or Live Search. As an Microsoft MVP I also get asked a lot why I would not use Live Search and Microsoft is constantly trying to get more people (and MVPs) using Live Search, but the results are very close to Google most of the time anyway and the interface is just strange if you are used to Google for 10 years (the English Live Search is actually nice with a new image every day, but the German version is totally messed up, just white boxes??). Yahoo at least gives different results, which are sometimes better, sometimes worse .. Maybe we all should switch search engines from time to time, its really stupid Google has 95% market share here in Germany almost and it returns so crappy results so often (finding drivers has become 10 times as hard in the last years, searching files or torrents is impossible, because you only get SEO sites, no real results, product searches are totally messed up, Google groups is not used much anymore, etc.). Okay, enough ranting about search engines .. time to refresh MSDN again.

Visual Studio 2010 beta is apparently coming soon. Since I used VS2005 beta and VS2008 beta when they came out, I will be an early adopter once again. I really want to use the cool Parallel APIs in VS2010 and C# 4.0 dynamic stuff :)

Another topic: I try to synchronize several of my servers at different locations every day. This way it does not matter where I download or create a file, I still can use it the next day wherever I am (home, work, different work ^^). I use SyncBackSE for that job, great tool, but the problem with it is that on slow internet connections (e.g. slow DSL at work right now) it takes forever to even scan the directories. Its about 30000 directories that have to be synchronized and this takes a lot of hours, especially if the internet connection is already doing something else. Obviously uploading and downloading a few GB can also take a lot of time, but in 95% of the cases nothing or just a few MB change every day.

Because searching sub directories is so slow I tried to make it faster by removing directories when not necessary (just compressing 100 directories into a zip file, removing empty directories or old files, etc.). But this is obviously a lot of work for this many directories and I can easily overlook that in sub directory 3592 there is 1500 empty folders of some crap from some years ago no one will ever need again. For this reason I quickly wrote this little tool called KillEmptyDirectories, you can see a picture on the right! It only took me an hour to write, so there is no rocket science here and it is very straight forward. I did not even unit test it, I just tested it by executing on several directories. It also can remove hidden and read-only files by changing the file attributes (otherwise File.Delete will always throw UnauthorizedAccessException). I tested it on around 30000 directories and was able to kill around 8000 of those (deleting a lot of old stuff) ^^ with some additional compression of unused older directories I was able to get it down to around 3000 directories (from 30000), so the scanning process is now about 10 times faster.

If you want to try it out, you can download it here. But use with care, you can obviously kill directories with it and by using the "Always kill these directories" options you can even delete directories with files in it. An extra confirmation message will appear if you try to use that feature.

• KillEmptyDirectories.exe (12.5 KB)
• KillEmptyDirectories.zip (11.83 KB)

And this is the main function that does all the directory killing:

/// <summary>
/// Recursively kill directories, will be called as many times as we have
/// sub directories (can be many thousand times). Will always go through
/// all subdirectories first in case we can remove them, which makes it
/// much easier to delete directories with just empty sub directories.
/// </summary>
private int RecursivelyKillDirectories(string directory,
    string[] includeFiles, string[] alwaysKillDirectories,
    bool alwaysKillThisDirectory)
{
    int directoriesKilled = 0;
    string[] subDirectories = Directory.GetDirectories(directory);

    // Only delete this directory if there are no useful files in here!
    // Note: GetFileName will give us the last part of the directory! 
    if (alwaysKillDirectories.Contains(Path.GetFileName(directory)))
        alwaysKillThisDirectory = true;

    // Handle subdirectories always first (maybe we can kill them too)
    foreach (string subDir in subDirectories)
        directoriesKilled += RecursivelyKillDirectories(subDir, includeFiles,
            alwaysKillDirectories, alwaysKillThisDirectory);

    // Get all files here and count how many of those we can ignore
    string[] files = Directory.GetFiles(directory);
    int ignoreFileCount = 0;
    foreach (string file in files)
        if (includeFiles.Contains(Path.GetFileName(file)))
            ignoreFileCount++;

    // Only found ignored files (or no files at all) or do we want to kill
    // this directory anyway?
    if (files.Length == ignoreFileCount ||
        alwaysKillThisDirectory)
    {
        // Check again if we don't have any sub directories left here
        // Maybe the subdirectories above were already killed now!
        subDirectories = Directory.GetDirectories(directory);
        if (subDirectories.Length == 0 ||
            alwaysKillThisDirectory)
        {
            try
            {
                // Kill all files in it (only ignored files anyway)
                foreach (string file in files)
                {
                    // Make sure we can delete hidden and readonly files
                    FileAttributes attributes = File.GetAttributes(file);
                    if ((attributes & FileAttributes.ReadOnly) != 0 ||
                        (attributes & FileAttributes.Hidden) != 0)
                        File.SetAttributes(file, FileAttributes.Normal);
                    File.Delete(file);
                } // foreach
                Directory.Delete(directory);

                // We killed something, yeah
                directoriesKilled++;
            } // try
            catch (Exception ex)
            {
                MessageBox.Show("Failed to delete " + directory + ": " +
                    ex.ToString());
            } // catch
        } // if
    } // if

    // Most of the time nothing was killed
    return directoriesKilled;
} // RecursivelyKillDirectories(directory, includeFiles)

Hi Guys, sorry for not updating much here last week. I have been quite busy with all the iPhone projects going on (at mobile-bits.de) and of course our big game project at exDream (btw: we made a new cool team photo today with 15 people from exDream, it will be up soon at http://www.exdream.com/company.html). I also worked a lot on the DLR language I started a month ago and I'm quite happy with the current state, everything works very well right now and the very basic language functionality (only some very simple commands) is already there and very fast and easy to use. I will improve the language hopefully this month to a useable state, but it will probably take a long time (maybe a year) until everything is done for this language. More on that in a few weeks maybe ..

Because we also lost one of our main website servers at exDream last week because the old publisher didn't want to pay for it anymore, I had to move a lot of websites to a much slower server today (sites for ArenaWars, Rocket Commander, Mods, EuroVernichter, Xna Racing Game, other Xna Projects, some internal sites, etc.). We will get a better internet connection in the next few weeks for that, but for now it is very slow and not very enjoyable to surf on there. The most annoying thing while moving all the wwwroot directories to the new slow server was the fact that the ACL permissions were totally screwed up. I had this happen to me before, but today it was very annoying. Dunno exactly why this happens, but it seems when you extract files into wwwroot your existing directory permissions are more or less ignored. This only happend to me on that Win2003 server, I tried reproducing the same thing with Windows 7 at home, but directory permissions are correctly set when extracting more sub directories there. I could just set those permissions again for some folders, which is kinda strange because the base directory already had the permissions for the exact same user. But for other directories I could just see that the web user permissions are set (so I could not add them again), but still once accessing the website I would just get:

HTTP Error 401 401.3 Unauthorized: Unauthorized due to ACL on resource

or this other error once I allowed my user to access the aspx files, which is pretty confusing too:

Parser Error Message: An error occurred loading a configuration file: Failed to start monitoring changes to 'C:\inetpub\wwwroot\dummywebapp\web.config' because access is denied.

Since I had set the ACL permissions and they looked correctly I was very confused and tried a lot of other stuff like changing the Web Application Pool, recreating the web apps, using all kinds of different users for those web apps, etc. But all that did not really help. Only after I just made sure all permissions for all files plus all directories for this web app were set correctly, it finally worked. I would say this problem should be fixed and the error messages should be better, but in Windows 2008 and Windows 7 it is already so much better, I was easily able to extract the exact same .rar file on my Windows 7 and everything just worked as expected as opposed to the Win2003 server ..

This week I wanted to test using a Ramdisk (Ramdrive using just your main RAM) for compiling Visual Studio projects. Playing games or doing other disk intensive stuff would be great too, but most games are just way too big and smaller older games load pretty quick anyway.

While copying files and benchmarking a Ramdisk is incredibly fast with 4-6 GB/s (theoretically my ram should almost reach 10 GB/s, but well that's already way fast enough). Since I use Windows 7 RC since Friday when it came out on MSDN I had a lot of problems finding Ramdisk programs that actually work and do not crash every 2 seconds. Currently I use RamdiskVE by Cenatek, but the company does not exist anymore because it was bought by Dataram, which provides RamDisk on their own now. I was however unable to run Dataram's Ramdisk on Windows 7, it constantly crashes and has also many other limitations. Here is also a discussion on the Ocz Forum about Ramdisks if you want to check out some of the products for yourself.

The Ramdisk is also useful for Temp files, IE Temp files and other Scratch Disk functionality (e.g. Photoshop), but you should only use those for programs to stupid to use more of your Ram (e.g. because a program is 32bit and you have way more memory in your 64bit system). It also will increase the lifetime of your hard disks or SSDs in case you write a lot of stuff on your Ramdisk because the Ramdisk will only be loaded once at start up and saved once when shutting down your PC. But you should be aware that in case of a crash, you will obviously lose all the changed content on your Ramdisk.

For testing I used my new rig with a i7 920 D0 CPU and 12 GB ram, which I overclocked from 2.66Ghz to ~4.4Ghz. This is pretty fast, for example the For vs Foreach Performance application from the last blog post is twice as fast as my 6600 CPU from last week (all times cut in half basically, which is more than I expected).

I also tested quite a lot of hard disks and SSDs, which was kinda interesting because I learned that it does not matter if you mix totally different hard drives into a Raid 0 as long both are similarly fast. For example I tried using 2 older 160 GB disks in a Raid 0 and was only able to get around 60mb/s, which is slower than a single Raptor hard disk, but putting a new fast Samsung 250 GB and a Raptor 150 GB together into a Raid 0 gave me around 160mb/s. To keep things crazy I also added another Intel x25-M SSD to the one I already had and put them into a Raid 0 too, which is amazingly fast (>450mb/s). For highest speed you should always make sure to Enable Write-Through Cache on the Raid controller (in my case a Intel software raid with the ICH10R controller) AND to enable write caching and finally turn off the Windows write-cache buffer flushing (both can be found in hardware->disk->policies) for your disks. Please note that I do not care about data redundancy since everything I do is saved on a server with a Raid 5 anyway. If one of my Raid 0s would fail it would just be annoying to reinstall everything, but I would not lose any of my work or files.

So lets see how much benefit you actually get from compiling several different projects on a fast PC with those different disk configurations. The compile time is obviously heavily dependent on your CPU speed, but I tried to measure how much the total time changes just by using different disks. The following example shows a full recompile of one of my bigger solutions:



All compiles are full recompiles from scratch (no immediate files yet). For testing the DLR (change set 23173, about 25 MB in 20 C# projects), one of my solutions with 5 C# projects, 1 C++ project (~43 MB) and finally the good old Quake3 v1.32 source code (5 MB of c code). Since I do not compile much C++ at home I was to lazy to test bigger C++ projects, but I would guess most times would just scale and the conclusion would be the same.

4 Tests were executed:

• Loading Visual Studio 2008 and opening the each solution (average time),
• Compiling the DLR and running the ToyConsole sample. This will generate around 25 MB of files (17 MB of those are .pdb), ~60 files.
• Compiling and running my own solution (~43 MB, 5 C# projects, 1 C++ project). Generates ~47 MB (lots of copying, >300 files).
• and finally compiling Quake3 (~500 c files).

Most of the tests were done several times, but I stopped all of them with this cool freeware stopwatch called PC Chrono. Keep that in mind, the results will not be very accurate. Each test was done on the following drives:

• Good old Raptor Hdd (one of the fastest desktop disks you can get, almost empty for better testing)
• Single Ocz Ssd with 150mb/s read/write (remember that I complained about the bad JMicron controller on it way back in February)
• Intel X25-M Ssd Raid 0 Array with 450mb/s read, 140mb/s write (nice ^^)
• And finally the Ramdisk with 4500mb/s read+write (at least)

Disk/Compile times	Loading VS+solution	Compiling DLR+start ToyConsole	Compile&Run own solution	Quake3 Compile
Good old Raptor Hdd	3.75s	9.01s	2.95s	17.57s
Single Ocz Ssd 150mb/s	2.96s	10.47s	3.24s	22.67s
Intel X25-M Sdd Raid 0 Array	1.26s	7.79s	2.88s	17.53s
Ramdisk with 4500mb/s	1.43s	7.70s	2.51s	15.89s

With this data the following fancy graph was build. It shows that there are some improvements in several areas, but as long as you are not limited by your disk speed or IOs (amount of input/output operations you can do per second), you do not get much benefit from way faster drives (the ram disk is at least 45 times faster than the hard disk I used, but the performance improvement is maybe 10 or 20%):



Since it is so much fun using the Intel SSD Raid 0, I will keep using it. It is also big enough to use for all my programs and games. For example launching Left 4 Dead levels is 2-3 times faster than before. But I will probably not continue to use the Ramdisk for compiling. I can just keep everything on the SSD raid, which will not run out of space as soon as my small Ramdisk. Maybe when I do some file-intensive stuff in the future I will try out the Ramdisk idea again.

For now my advise for getting the fastest Visual Studio experience would be getting the fastest CPU you can (i7 920 is pretty nice, overclocking is important too, without it my tests would be 40% slower, 8 hyper-threads might also be useful in the future), enough Ram (4-6 GB), a fast disk like the Intel X25-M and of course using Windows 7 (since everything responses much faster). Then you can really have a lot of fun compiling big and small projects, and playing games, and doing other stuff on your PC.

BTW: My brother also just blogged about Windows 7 RC and using RamdiskVE. He also noticed that the network layer in Windows 7 is way faster than before in Windows Vista for him, in Vista he had 30-60mb/s max, now it is like 110mb/s copying files over the network:

Today I had a discussion with a few other programmers (Hi ViperDK and Timo ^^) about foreach and for loop performance. Everyone had his own ideas and experiences, but we talked mostly about XNA on the Xbox 360 (using the compact framework, that does not like foreach, just use for all the time in tight render loops for your XNA games, foreach will create too much IEnumerator garbage that will not be collected) or other special cases (huge arrays or lists).

Anyway, just guessing around is obviously not as good as actually writing a performance test application and see whats going on (see end of this article for download link). Again, we need a disclaimer: You will probably never write loops with 1 billion iterations, these numbers are similar for less iterations, but you should always try to write clean code and ONLY optimize when it is required after finding performance issues! Lets go directly to the results for 1 billion for or foreach iterations in release mode (will loop through a 10k array or list for 100k times, see below for code):

As you can see there are some major differences between using foreach with an array or using a generic list (boxing/unboxing is not the issue here, you would not be able to archive this performance with an ArrayList). Another thing you can notice right away is that using simple arrays is always faster than using dynamic lists, even converting them from a list to an array via .ToArray() and then executing the foreach is much faster.

It also makes a HUGE difference executing this in debug mode (I execute my code almost 99% in debug mode), everything is 4 times slower for this special performance test. Except for Foreach+List, which is in comparison not that bad in debug mode. What is really bad is For+List, which almost takes 16 seconds to execute 1 billion times (again using my Core 2 Duo 6600 CPU with 4 GB Ram).



Thanks to some helper methods the code for this performance test is pretty short and easy (check end of this article for the source code download link):

 Stopwatch timer = new Stopwatch();
 // Run through all tests
 for (int test = 0; test < 8; test++)
 {
     // Find out which test we wanna make
     bool rememberLength = (test / 4) % 2 == 1;
     bool useForeach = (test / 2) % 2 == 1;
     bool useList = test % 2 == 1;

     // Start timer for test 
     timer.Start();

     // Execute
     int result = RunForOrForeachLoops(rememberLength, useForeach, useList);

     // Stop timer
     timer.Stop();

     // Skip this test if we got no result!
     if (result == 0)
         continue;

     // Report results!
     resultText = ...
     Console.WriteLine(resultText + ": " + timer.ElapsedMilliseconds + "ms");

     // Reset timer for next run
     timer.Reset();
 } // for

The interesting code obviously happens in RunForOrForeachLoops. It basically executes the 1 billion iterations in several different ways (8 ways, as you can guess by the for main test for loop). All of those produce the same result, half of the tests use int[] intArray, the other half uses List<int> intList. And again half of the tests uses just for loops, the other half uses foreach for the iterations. Lets examine the useForeach=true and rememberLength=false cases (btw: Iterations is 100000 and the intList and intArrays have a size of 10000 elements):

if (useForeach)
{
    if (useList)
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intList)
                result += val;
        } // for
    } // if
    else
    {
        for (int iter = 0; iter < Iterations; iter++)
        {
            result = 0;
            foreach (int val in intArray)
                result += val;
        } // for
    } // else
} // if
else // useForeach == false
...

To figure out the differences between those few lines of code in RunForOrForeachLoops, that almost look the same, we have go to IL code once again. First lets check out what the fastest code block Foreach+Array is being compiled to in IL:

result = 0;
foreach (int val in intArray)
    result += val;

This becomes in IL:

 L_0007: ldsfld int32[] TestSimpleForeachApp.Program::intArray
 L_000c: stloc.2 
 L_000d: ldc.i4.0 
 L_000e: stloc.3 
 L_000f: br.s L_001d
 L_0011: ldloc.2 
 L_0012: ldloc.3 
 L_0013: ldelem.i4 
 L_0014: stloc.1 
 L_0015: ldloc.0 
 L_0016: ldloc.1 
 L_0017: add 
 L_0018: stloc.0 
 L_0019: ldloc.3 
 L_001a: ldc.i4.1 
 L_001b: add 
 L_001c: stloc.3 
 L_001d: ldloc.3 
 L_001e: ldloc.2 
 L_001f: ldlen 
 L_0020: conv.i4 
 L_0021: blt.s L_0011

In L_0021 we jump up to L_0011 as long as we are not done with the loop yet. Inside all the result += val happens, but we do not see any slow code, there are no calls to any methods, there is no unboxing and this all can probably be executed quickly after being complied to assembler by the JIT (just-in-time) compiler.

On the other hand, if we look at pretty much the same code, just with intList instead of intArray, the whole story changes:

result = 0;
foreach (int val in intList)
    result += val;

becomes much more complex IL code with lots of function calls, the creation of an Enumerator and there is even a dispose at the end (hello GC):

  L_0007: ldsfld class [mscorlib]System.Collections.Generic.List`1<int32> TestSimpleForeachApp.Program::intList
  L_000c: callvirt instance valuetype [mscorlib]System.Collections.Generic.List`1/Enumerator<!0>
[mscorlib]System.Collections.Generic.List`1<int32>::GetEnumerator()
  L_0011: stloc.2 
  L_0012: br.s L_0020
  L_0014: ldloca.s CS$5$0000
  L_0016: call instance !0 [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::get_Current()
  L_001b: stloc.1 
  L_001c: ldloc.0 
  L_001d: ldloc.1 
  L_001e: add 
  L_001f: stloc.0 
  L_0020: ldloca.s CS$5$0000
  L_0022: call instance bool [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::MoveNext()
  L_0027: brtrue.s L_0014
  L_0029: leave.s L_0039
  L_002b: ldloca.s CS$5$0000
  L_002d: constrained [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>
  L_0033: callvirt instance void [mscorlib]System.IDisposable::Dispose()
  L_0038: endfinally 

What even looks worst too me is the code that is executed in MoveNext (get_Current just returns this.current, no extra instructions here). Since the IL code is very long and confusing (31 IL instructions), here is the C# code of MoveNext:

public bool MoveNext()
{
  List<T> list = this.list;
  if ((this.version == list._version) && (this.index < list._size))
  {
    this.current = list._items[this.index];
    this.index++;
    return true;
  }
  return this.MoveNextRare();
}

And MoveNextRare could also be called at the end of MoveNext if something changes for the list (not the case in our example). But we have to keep in mind that this is just IL code, not actual code that will be executed on the CPU. The JIT compiler can still optimize things out, which we won't see here. But something must be going on since Foreach+List is definiately slower than using all the other approaches to get through all the values in intList. I recommend Foreach+Array converted from List because the IL for Foreach+Array is really fast and does not use any extra methods or Enumerators, especially if you know that the array won't change much and if you do not have to call .ToArray every time when you do a foreach.

And here is the sample project for all this performance testing fun:

• TestForEachPerformance.zip (4.56 KB) project (C#)
  

The last few days I was working a little bit with the DLR and got to a point today where I was wondering about the performance of a very simple while loop in the DLR, specifically in the ToyScript DLR sample project and of course IronPython. I am totally aware that the following comparison DOES NOT really provide accurate information about the actual performance of the languages and techniques used. Please keep this in mind when reading this article, it only covers a very certain aspect of all used languages. More specially how they perform when adding numbers in a while loop a lot of times! Some of it reflects that compiled languages are faster than dynamic languages, but you should not compare the absolute values, Python or Lua is not really 100 times slower, in many cases you can reach almost similar performance as long as you implement or call performance-critical code in a clever way (e.g. calling C++ code, using frameworks, etc.).

I was only interested about very ruff comparisons to figure out which ways are fast and which languages or techniques are kinda slow (click image on the right for a quick overview). Some code written in Assembler or direct IL code execution was really fast (duh, no wonder), but all execution times of C++, C#, Assembler or IL are way faster than all other dynamic languages or DLR approaches. The important thing for me was to figure out why dynamic languages like IronPython or ToyScript on top of the DLR were like 100 times slower for this specific code. At the end of writing this article I was able to provide pretty good performance with the DLR, which is just 2-3 times slower than the fastest execution time. That's pretty good and could probably be even improved more.

Basically I used the following code sample, which adds 1.0 to counter 10 million times while decreasing the loops variable until it reaches 0. The following code block is not valid syntax in any of the presented languages, but it is very similar to Python. In languages that use compile time types, counter would be a declared as a double (64bit floating point number), most dynamic languages use double as the run-time type for floating point numbers also.

loops = 10000000
counter = 0.0
while loops > 0
  loops = loops - 1
  counter = counter + 1.0

print counter

Languages and techniques used and tested (all with detailed code examples and explanations):

• C# 3.0
• Native C++
• Assembler
• IronPython 2.6
• ToyScript
• Python 2.6
• Python 3.1
• Lua 5.1.4
• Script.NET
• IL Emit
• Irony + DLR (dummy)
• DLR AST with objects
• DLR AST with doubles

Before boring you with all the different implementation details, here is the fancy Code Generation Performance comparison. This article became very long; I just wanted to post a few sentences about the DLR and this is what happened. The following graphs were created with Excel with some help from my good old friend ViperDK. The second graph (the first one was already shown above) shows just the execution time of the while-loop code block in each of the used language or technique.


And this graph also shows the other times like compile & build time where it makes sense or script or DLR start-up times to see how much this could affect the overall time from coding to seeing the result on the screen. This is important for my approach to my own language since I want a extremely low time for the coding to see result loop. Start-up time can be ignored in those cases, but it is still important since it is still kinda slow with the DLR (but will hopefully get better in the future).



Here are the values used to generate those graphs. Tested on my almost 3 years old 2.4 Ghz 6600 CPU; this is probably way faster on my i7@3.6 Ghz at work:

Language or Technique	Code lines	Compile&Build	Script Startup	Parsing&AST	Execution
C# 3.0	10	~500ms	-	-	~37ms
Native C++	8	1500ms	-	-	~31ms
Assembler	15+	~1100ms	-	-	~15ms
IronPython 2.6	6	-	~300ms DLR	160ms	~9000ms (~1250ms for files)
ToyScript	8	-	~430ms DLR	103ms	~8400ms
Python 2.6	6	-	~50ms	~100ms	~3500ms
Python 3.1	6	-	~50ms	~100ms	~4500ms
Lua 5.1.4	7	-	~25ms	~150ms	~1400ms
Script.NET	8	-	~100ms	-	~39752ms
IL Emit	30+	-	~100ms	~1ms	~16ms
Irony + DLR (dummy)	30 + 6	-	265ms Irony	87ms	~138ms (dummy)
DLR AST with objects	68+	-	433ms DLR	4ms	~459ms
DLR AST with doubles	60+	-	434ms DLR	5ms	~98ms

Implementing and testing the following code was not too bad for me since I was learning how to generate AST statements in the DLR, generating System.Reflection.Emit IL code, doing stuff with the ToyScript DLR sample, etc. anyway. I was of course interested in how to get the best performance for my code generation, but it is useful to understand why each approach is slower or faster. Sorry if the comparison presents your project or technology you might use in a bad light, but for example Script.NET is slow in comparison (this is explained below in more detail), but still fast enough for probably 99% of its uses. It still took much longer than I expected, writing this article did take some time too, but most of the time was spend writing and testing code in each language and technique and then gathering the execution time results. But performance evaluation is always fun :) Again: This is not a representative performance comparison of the languages and techniques involved, it only applies to my very short and stupid test code and should only give you a general idea.

C# 3.0

First of all the very simple implementation of the while loop:

public static double DoLoop(int loops)
{
    double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0;
    }
    return counter;
} // DoLoop(loops)

And now some code to measure the execution time and to print out the result to see if the loop was executed as many times as we expected:

    long startTime = WindowsHelper.GetPerformanceCounter();
    
    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    // Note: We did printing in other tests too (does not matter anyway)!
    Console.WriteLine("DoLoop result=" + result);
    
    long endTime = WindowsHelper.GetPerformanceCounter();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + WindowsHelper.ConvertToTime(endTime - startTime));

Note that the WindowsHelper class is just providing a little bit more accurate information than just using a StopWatch (which is used in most other samples here), which is fine too. With System.Diagnostics.StopWatch this code looks like the following (actually results in the same results on Windows 7 and Vista):

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    //Note: We did printing in other tests too (does not matter much anyway)!
    Console.WriteLine("DoLoop result=" + result);

    timer.Stop();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + timer.ElapsedMilliseconds + "ms");

The following output is generated:

DoLoop result=10000000
Executing the while loop 10000000 times took: 37ms

Native C++

Here is the whole program used for testing. This has nothing to do with the DLR or .NET anymore, I just wanted to know how my little while loop would perform in native C++ and Assembler. Sometimes when using for loops the C++ compiler will optimize the code away and just replace it with something static that results in the same value as if the for loop would have been executed. Please also note that it did not make any difference to use a 64bit platform and doubles or just 32bit floats, but since most script languages and especially .NET compile to x64 too, we should not use 64bit numbers when compiling a 32bit application here in C++. Switching to x64 is also not too hard, you just need the correct 64bit libraries and set the linker target platform to X64.

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    // C++ version
    int loops = 10000000;
    float counter = 0.0f;
    //64bit code: double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0f;
    }
    
    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    // Wait for user input (not required for Debug.StartWithoutDebugging)
    //char input[128];
    //scanf(input);

    return 0;
}

And after compiling (usually a lot faster than 1s when just incrementally compiling) and running this code, it generates this output:

counter=10000000.000000
This took 31ms

The result is not very accurate, I either get 31ms or 47ms, but not values in between, the accuracy of GetTickCount pretty much sucks and I should use similar code as the GetPerformanceCounter() stuff above from C#, but I do not really care about more accurate values here, C++ is more than fast enough and we do not need to discuss its runtime performance!

Assembler

Based on the C++ solution I was able to write some Assembler instructions. It has been a very long time since I used assembler code at all, probably in 2002 when optimizing some path finding algorithms for ArenaWars, which were called from C++ code, which was called from C# code. Before that in the C++ days I used it now and then, but never for more than some performance critical inner loops. Please also note that sometimes you make non-optimal assembler code (as I probably did in this example) and just having your C++ compiler generate assembler instructions can be better, so always profile your code and only optimize to Assembler if you have no other option left and if you are too lazy to rethink the problem :D

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();
    
    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
startLoop:
        // counter += 1.0f
        fld counter
        fadd counterInc
        fstp counter

        // loops--
        dec loops

        // loops > 0? 
        cmp loops, 0
        // Then goto startLoop (else we are done)
        jg startLoop
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

I don't have much to say about this ugly assembler code, as you can see it is a lot more lines of code and this was the first idea not even using registers as you should in assembler. After running this code and getting 31-47ms too (like the C++ version), I added some minor optimization by using registers more explicitly and this is the resulting code, it ran about 2-3 faster than the above code:

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
        // Copy loop counter to eax register (32bit)
        mov eax, loops
        // Load both float values
        fld counterInc
        fld counter
        
startLoop:
        // counter += 1.0f
        fadd st(0), st(1)
        //st(0) has still the counter value! no need to store: fst st(0)

        // loops--
        dec eax

        // loops > 0? 
        cmp eax, 0
        // Then goto startLoop (else we are done)
        jg startLoop

        // We are done, copy counter value back!
        fstp counter
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

This program does produce the following output:

counter=10000000.000000
This took 15ms

This is pretty good and I did not want to optimize it further, especially not when the timer is so inaccurate. Running the code 10 times resulted in 140ms. Again, fast enough, no reason to argue the superior runtime performance. But this comes at a high price since the code is in no way easy to understand, its hard to write and read.

IronPython 2.6

Speaking of code, I would say the implementation in Python is the shortest and very beautiful since it is so easy to read and write:

loops = 10000000
counter = 0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

I just used my ScriptManager from my .NET libraries, which can handle IronPython (and other DLR languages) as well as Lua and wrote this short unit test to see how IronPython performs:

    public void TestPythonDoLoop()
    {
        // Create script manager without any output redirection or special directories
        ScriptManager manager = new ScriptManager(null, null);
        
        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Simply execute line by line as we would in a console window
        manager.ExecuteCode("loops = 10000000", ScriptType.Python);
        manager.ExecuteCode("counter = 0", ScriptType.Python);
        manager.ExecuteCode(@"while loops > 0:
loops = loops - 1
counter = counter + 1", ScriptType.Python);
        manager.ExecuteCode("print counter", ScriptType.Python);

        timer.Stop();
        Console.WriteLine(
            "Executing the while loop 10000000 times in python took: " +
            timer.ElapsedMilliseconds + "ms");
    } // TestPythonDoLoop()

And this results in the following output:

Executing the while loop 10000000 times in python took: 9151ms

Update 2009-04-21: As Simon Davy correctly states in the comments of this post, ipy.exe (IronPython) is actually a lot faster when executing a file with the code above (e.g. use import time). This results in an execution time of just 1250ms, so a lot faster! I have tested it too with ipy, but I entered the code line by line (same way as I called my ScriptManager code in the unit test above) and it still takes 9 seconds. But Simon is right, once I execute everything at once (with ipy test.py) it only takes 1.25s (can also be done with my ScriptManager, but I have not thought of that). So it is probably just an issue with REPL (entering statements one by one .. less optimizations that way). While this certainly does make IronPython a much faster dynamic language than all the other ones, it still is slower than the compiled languages or the DLR code at the end of this article.

Thanks for noticing!This is certainly not good. The same thing happend after I tried using the IronPython 2.6 console or IronPython 2.0, it always took about 9 seconds to complete the big while loop on my PC. I wanted to figure out why this is so slow compared to C#, C++ or Assembler runtime, but then again we have to remember that Python is a dynamic language. The most important reason for the long runtime here is probably just all the boxing and unboxing of values, if you can say that for dynamic languages. What I mean is figuring out which type we got, and then performing the add/sub operation, which can be slow if you do it 10 million times. Please note again that this is probably not affecting any real Python program and performance for more complex programs will be way better. More on the type-performance and this issue in general at the end of this article!

ToyScript

To learn more about the DLR I played around with the DLR ToyScript sample quite a bit the last days. Since a lot of its code is based on the same ideas as IronPython and IronRuby, you can't expect much different performance results. This is the ToyScript code for the while loop:

loops = 10000000
counter = 0
while (loops > 0)
{
    loops = loops - 1
    counter = counter + 1
}
print counter

Similar to the ScriptManager above I wrote some helper classes for the ToyScript language allowing me just to execute code, viewing the AST (abstract syntax tree) and write a lot of unit tests. The following code is such a unit test for the ToyScript class to check the performance for these 10 million while loop iterations:

    public void TestBigLoop()
    {
        const int NumberOfLoops = 10000000;

        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Initialize the language first (takes its own time, around 400ms)
        ToyLanguage language = new ToyLanguage();
        // Execute something else (takes maybe 100ms)
        Assert.Equal(3.0, language.Execute("print 1+2"));

        timer.Stop();
        Console.WriteLine("Startup time: " + timer.ElapsedMilliseconds + "ms");
        timer.Reset();
        timer.Start();

        Assert.Equal((double)NumberOfLoops,
            language.Execute(
@"loops = " + NumberOfLoops + @"
counter = 0
while (loops > 0)
{
loops = loops - 1
counter = counter + 1
}
print counter
"));

        timer.Stop();
        // Show total execution time
        Console.WriteLine("Executing the while loop " + NumberOfLoops +
            " times took: " + timer.ElapsedMilliseconds+"ms");
    } // TestBigLoop()

This code runs for about the same time as IronPython, maybe a little faster since ToyScript has a lot less features than IronPython, but in general it uses the same approach. This is the output from the unit test:

Startup time: 650ms
Executing the while loop 10000000 times took: 8914ms

Investigating the Lambda Expression AST for the code block shows that quite a lot of statements were created in order to make this ToyScript code run on the DLR:

 // Expression: Expression`1
 
 .lambda (<toyblock>$2 Microsoft.Func`1[System.Object] #1)
 
 // LambdaExpression: <toyblock>$2(1)
 .lambda System.Object <toyblock>$2 ()(
 ) {
   .label 0x03d11d12:
   .comma  {
     .block  {
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)10000000)
       ,
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)0)
       ,
       .loop break:.label 0x03fe2d21  {
         .block  {
           .if (
             .site (Boolean) CallSiteBinder(ConvertTo to System.Boolean) (
               .extension CodeContextExpression ( )
               ,
               .site (Object) CallSiteBinder(DoOperation GreaterThan) (
                 .extension CodeContextExpression ( )
                 ,
                 .extension GlobalVariableExpression ( )
                 ,
                 0
               )
             )
           ) {
             /*empty*/
           } .else {
             .block  {
               /*empty*/,
               .break .label 0x03fe2d21
             }
           }
           ,
           .block  {
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             .site (Object) CallSiteBinder(DoOperation Subtract) (
               .extension CodeContextExpression ( )
               ,
               .extension GlobalVariableExpression ( )
               ,
               1
             ))
             ,
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             'ToyHelpers.Add'(
               (Object).extension GlobalVariableExpression ( )
               ,
               (Object)1
             ))
             ,
             /*empty*/
           },
           /*empty*/,
           /*empty*/
         }
       },
       'ToyHelpers.Print'(
         .extension CodeContextExpression ( )
         ,
         .extension GlobalVariableExpression ( )
       ),
       /*empty*/
     },
     .null
   }
 }

The IL code was even more confusing so I only tried to figure out the simple "print 1+2" expression IL code that can be produced by calling ScriptCode.SaveToAssembly(..), more DLR IL code analysis will happen at the end of this article:

  .method public specialname static object <toyblock>$1$1(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope $scope,
 class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext $language) cil managed
  {
    .maxstack 3
    .locals init (
      [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context)
    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
    L_0007: stloc.0 
    L_0008: ldloc.0 
    L_0009: ldc.r8 1
    L_0012: box float64
    L_0017: ldc.r8 2
    L_0020: box float64
    L_0025: call object ToyScript.ToyHelpers::Add(object, object)
    L_002a: call void ToyScript.ToyHelpers::Print(class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext, object)
    L_002f: ldnull 
    L_0030: ret 
  }

There is a lot of extra stuff in there, but you can clearly see the 1+2 in L_0009 to L_00020 with all its boxing fun. Then those two are added with help of a method Add that is defined in the ToyHelpers class (unboxing the doubles again, adding and boxing them to an object again, which is returned). Then the result is passed along to the ToyHelpers.Print method, which just calls Console.WriteLine in my case (usually it is used by the ConsoleHost class for the normal REPL console interaction).

I am not a DLR expert and there is probably a lot of other things going on, but to me all this boxing, unboxing and extra code just to add some numbers is clearly too much for the big loop. At the end of this article I will try to write some DLR code that does less boxing/unboxing.

Python 2.6

After testing the DLR it was time to check out the native Python implementations to see if this is releated to the dynamic nature of dynamic languages like Python or Lua. Well it obviously is related, but more specifically I wanted to see if certain implementaions are maybe more clever or optimized.

The Python code is the same as above for IronPython:

loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

Executing this takes about 3.5 seconds, we could cheat a little and use the following code to execute this loop a little faster:

counter = 0.0
for loops in range(0, 10000000):
    counter = counter + 1.0
print counter

This runs about 2 times faster (less than 2 seconds) because Python is clever and optimizes for loops. The same actually applies to IronPython, a for loop is much faster here too (5.5s instead of 9.1s). But this defeats the whole point of this exercise, I wanted to know what is slow about the DLR and implementations like IronPython or ToyScript and why. It is nice that Python executes for loops quicker, but the whole counter = counter + 1 thing is still very slow compared to C#, C++, etc.

Python 3.1

Executing Python 3.1 is pretty much the same as Python 2.6. It is slighly slower, but that can also be because I use a alpha build (3.1 alpha 2 from 2009-04-04). Again, executing the for-loop version with range was twice as fast (2.1s instead of 4.3s). In general is is quite known that Python is not one of the fastest languages, there are many dynamic languages that can be faster (for example Ruby or Lua), but there are also much slower ones like PHP. If you are more interested about comparing IronPython with Python read the performance comparisons at the IronPython project at codeplex: http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Performance

Lua 5.1.4

The lua code is pretty similar to Python, but it executes much faster:

loops = 10000000
counter = 0.0
while (loops > 0) do
    loops = loops-1
    counter = counter+1
end
print(counter)

This executes in less than 1.5 seconds and returns the expected 10000000 (not 10000000.0 as Python however since Lua treats everything as a 64bit double anyway). The for-loop trick works here too and reduces the time in half (0.7s instead of 1.4s):

counter = 0.0
for i=1,10000000 do
    counter = counter+1.0
end
print(counter)

The reason why Lua is much faster than all the other dynamic languages I have tested is probably because Lua has much simpler data types and is optimized a lot for math, for loops and stuff like that. I use Lua more than Python these days and the performance is quite good I have to say (using it for AI, some physics and some game logic). As you can see here in many benchmarks Lua is several times faster than Python (but Python is still a lot more powerful with the huge amount of libraries available for it).

Script.NET

Script.NET is a great project that uses the Irony compiler toolkit to parse and evaluate simple C# code. It does not generate any IL or immediate code, it just evaluates the AST directly. This is a very nice approach and easy to understand because all the code is C#, but there is a slight problem with it: This approach is it VERY slow. It was optimized many times already and runs much faster than a year ago, but I would not recommend it for anything other than some scripting in C# (guess what, that's the whole name of the project). If you do not need high performance Script.NET is just fine for executing a couple lines of code, even for games as long as you do not do any physics or rendering code with it ^^

loops = 10000000;
counter = 0;
while (loops > 0)
{
    loops = loops-1;
    counter = counter+1;
}
Console.WriteLine(counter);

The output appears after almost 40 seconds and displays correctly:

10000000

At least I now know why generating some IL code is very important, with or without the DLR. Next I will investigate IL code generation with Reflection.Emit, but I can still remember the pain from the last time I tried to do something with it.

IL Emit

So let's get right to it, the following code will generate the required IL for us:

    // Create a dynamic method called DoLoop where we put all our IL code
    DynamicMethod DoLoop = new DynamicMethod("DoLoop",
        typeof(double), null, typeof(string).Module);

    // Start a timer once again to measure creation time and execution time!
    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Get an ILGenerator and emit a body for our dynamic method
    ILGenerator il = DoLoop.GetILGenerator();
    
    // Create counter (double)
    LocalBuilder counter = il.DeclareLocal(typeof(double));
    il.Emit(OpCodes.Ldc_R8, 0.0);
    il.Emit(OpCodes.Stloc, counter);
    // Assign loops variable to 10000000 (10 million) for the loop!
    LocalBuilder loops = il.DeclareLocal(typeof(int));
    il.Emit(OpCodes.Ldc_I4, 10000000);
    il.Emit(OpCodes.Stloc, loops);
    
    // Start loop label
    Label startLabel = il.DefineLabel();
    il.MarkLabel(startLabel);
    
    // Do this 10000000 times:    
    // Increase counter by 1.0
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ldc_R8, 1.0);
    il.Emit(OpCodes.Add);
    il.Emit(OpCodes.Stloc, counter);

    // Decrease loops by 1
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 1);
    il.Emit(OpCodes.Sub);
    il.Emit(OpCodes.Stloc, loops);

    // Check if we still have to loop (as long as loops > 0)
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 0);
    il.Emit(OpCodes.Bgt, startLabel);

    // End loop and return counter!
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ret);

    // Create a delegate that represents the dynamic method. This
    // action completes the method, and any further attempts to
    // change the method will cause an exception.
    DoLoopDelegate loop = (DoLoopDelegate)
        DoLoop.CreateDelegate(typeof(DoLoopDelegate));

    timer.Stop();
    Console.WriteLine("DoLoop IL creation time: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();

This creates the dynamic DoLoop method, which we can call with loop() from here on, which will also return the result (counter) once we call it. This all takes almost no time at all:

DoLoop IL creation time: 0ms

Next, we need to invoke this dynamic method, which is rather easy:

    timer.Start();
    Console.WriteLine("Invoking DoLoop() returned: " + loop());
    timer.Stop();
    Console.WriteLine("DoLoop IL execution time: " + timer.ElapsedMilliseconds + "ms");

And this only takes about 16ms and also returns the correct value:

Invoking DoLoop() returned: 10000000
DoLoop IL execution time: 16ms

So IL code is rather fast (faster than everything else here except the fine-tuned assembler, which is similar to this), but don't ask in how many problems I ran into just writing these few lines of code. Let's just say a lot of times I used wrong opcodes, did not know how to implement stuff and finally when stuff compiled I got syntax errors when executing, not really telling me what I did wrong, so I had to comment out code until it worked again and then slowly added code until I could figure out what had to be changed. Let's just say not a pleasant thing to write a compiler this way.

Irony + DLR (dummy)

As mentioned before Script.NET actually uses the Irony compiler toolkit for parsing and building the AST (abstract syntax tree). Irony itself is rather fast (can parse 15000 lines of code or more per second), only the execution part of Script.NET wasn't so good because no code is generated, everything is evaluated on the fly.

While I have started using Irony just a week ago after playing around with ANTLR first (see earlier blog post here), but I still like it very much and I hope Roman Ivantsov (the guy behind Irony) will soon release a long promised update with some cool new features I can sink my teeth into. As you will see in my sample below I have added some own functionality to Irony already to make it easier to test code and I will probably extend it even more when I write more of my language grammar.

If you don't know how Irony works and you are interested in building compiler grammars directly in C# you should check out the project and the Lang.NET talks Roman did this and last year about it, plus the great Irony article on CodeProject of course. Basically you define some non-terminals (variables, numbers) and terminals (logical blocks of your code) in the constructor of a Irony.Grammar derived class and then define some rules for it (in BNF = Backus-Naur Form). Then you set the root node and you are ready to parse some code. Please note that this grammar is very simple and will only handle this sample code and very similar code blocks. I'm still just playing around with grammars, there is no concrete implementation of my language yet, just a lot of ideas .. Writing some grammars is a good training, I have written one in ANTLR and 4 other more complex ones with Irony yet.

    public TestDoLoopGrammar()
    {
        // 1. Terminals
        var variable = new IdentifierTerminal("variable");
        var number = new NumberLiteral("number");

        // 2. Non-terminals
        var program = new NonTerminal("program");
        //var commandList = new NonTerminal("commandList");
        var command = new NonTerminal("command");
        var assignmentOperator = new NonTerminal("assignmentOperator");
        var assignment = new NonTerminal("assignment");
        var whileLoop = new NonTerminal("whileLoop");
        var compareOperator = new NonTerminal("compareOperator");
        var condition = new NonTerminal("condition");
        var binaryOperator = new NonTerminal("binaryOperator");
        var operation = new NonTerminal("operation");

        // 3. BNF rules
        // Please note that this program grammar is rather limited and not really
        // useful, it is only used for testing and learning Irony ..
        // We can have many commands in case we want to go crazy
        program.Rule = MakePlusRule(program, null, command);
        assignmentOperator.Rule = Symbol("=");
        assignment.Rule = variable + assignmentOperator + number;
        whileLoop.Rule = Symbol("while") + condition + ":" +
            program + "end";
        // We only allow some simple compare conditions
        compareOperator.Rule = Symbol("<") | "==" | ">";
        condition.Rule = variable + compareOperator + number;
        // We can either use + or -
        binaryOperator.Rule = Symbol("+") | "-";
        operation.Rule =
            variable + assignmentOperator + variable + binaryOperator + number;
        // Commands can be assignments, while loops or operations
        command.Rule = assignment | whileLoop | operation;

        // 4. Operators precedence and punctuation
        RegisterPunctuation(":", "end");

        // 5. Global settings
        this.CaseSensitive = false;
        this.ThrowGrammarExceptionsOnError = true;

        // 6. Set grammar root
        this.Root = program;
    } // TestDoLoopGrammar()

Note that ThrowGrammarExceptionsOnError is not a property of Irony, I added it to make sure all Grammar errors immediatly cause an exception to be thrown, which makes it much easier for unit tests and figuring out what when wrong. Not really sure why Irony does not do that, I can understand not throwing exceptions for syntax errors when parsing some source code (because that might be slower than just reporting errors in a list form and more helpful because its easier to continue after the error). But grammar errors mean usually that you cannot use the grammar for anything anyway. Some things also just produce a warning, but I throw Exceptions then too because I don't want hidden fixes of my grammar, which would make it hard to fix real problems later.

Anyway, with the TestDoLoopGrammar class we can now test and parse some code. Irony usually expects you to use its Grammar Explorer program to do this job, which is a pretty nice application, but it is not unit test and we all know I'm all about unit testing (don't we?). I still use the Irony Grammar Explorer from time to time and I recommend it very much if you want to test out grammars and visualize the AST a little better than just throwing some text into a console window. The following simple unit test will parse our while-loop program source code and produce a nice AST for us, which it kindly prints out:

    public void GenerateDoLoopAST()
    {
        AstNode program = IronyHelpers.ParseWithErrorHandling(
            new TestDoLoopGrammar(),
@"loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
end");

        IronyHelpers.PrintAst(program);
    }

This will quickly print out the AST in the following way, which might seem a little complex and verbose at first, but you have to remember that we do not have to generate code for every single node. Often several nodes together will just be one single instruction. Other nodes like program are just used to hold code blocks together and allow several child nodes.

 program
   assignment
     loops [variable]
     assignmentOperator
       = [Symbol]
     10000000 [number]
   assignment
     counter [variable]
     assignmentOperator
       = [Symbol]
     0 [number]
   whileLoop
     while [variable]
     condition
       loops [variable]
       compareOperator
         > [Symbol]
       0 [number]
     program
       operation
         loops [variable]
         assignmentOperator
           = [Symbol]
         loops [variable]
         binaryOperator
           - [Symbol]
         1 [number]
       operation
         counter [variable]
         assignmentOperator
           = [Symbol]
         counter [variable]
         binaryOperator
           + [Symbol]
         1 [number]
     end [variable]

Now this tree has to be translated to some code. We could use Reflection.Emit for this, but we just saw how much work it was for this very simple sample. Thats the reason I'm using the DLR instead, which also gives my language some cool dynamic capabilities for free (plus the other benefits like interacting with other DLR languages). But since the DLR is not a piece of cake and I'm not done exploring it yet (just started last two week with it, gimme more time), I will not present a full project with this language here yet (totally out of the scope of this article). I used a modified version of ToyScript and some other sample projects (following next) to generate some DLR code here, but it is still hacked up very much. I will probably write another long article when I'm done with that and have some cool results to show.

DLR AST with objects

Okay, so let's dig deeper into the DLR. While I got some parts of the above Irony AST working in the DLR with help of the ToyScript sample, the whole DLR framework, the ToyScript sample and especially IronPython is still very hard to understand and its also kinda annoying that recent code drops do not work at all with ToyScript. They compile, but running ToyScript will just produce funny errors like "node not reducible", which is not very helpful if you just want to learn about the DLR. A lot of stuff in the DLR still constantly changes (just see the huge amount of code drops at dlr.codeplex.com). I totally like that the DLR team releases all this wonderful code and does it so often, but without any working documentation (almost everything you find on the internet is based on DLR versions from a year ago, when stuff worked in a totally different way) and not even the included samples working all the time is kinda painful. So what does a programmer do in such a case? Yes, he writes tests and figures it out on his own. If there were not this many classes that all do almost nothing in functionality (what the hell, in my projects each class has many hundred lines of code, but in the DLR it seems like 50% of all classes are just 10-30 lines long) it would a much easier job. Debugging is also much harder because stack-traces become 2 miles long after just calling one of two actual functions, that do not just call other functions. Okay, enough ranting, let's just hope these things get better in the final 1.0 release since Microsoft is not allowing any contributions from the community to the DLR and I certainly do not want to write my own. It is pretty amazing what the DLR can do and how well it performs, it just has to become easier to use. Someone had to say it :)

So the bare minimum you need for the DLR to compile some AST for you is a LanguageContext and a DefaultBinder, define one class of each and derive it accordingly. Then define a constructor for your language context, set the binder and while you are at it add the default System (mscorlib) assembly to the script domain manager. Note this code uses DLR change set 21259 (from Apr 5 2009), but I could also use the more recent 22459 version (from today), but I reverted to make the ToyScript sample work again.

    public YourLanguageContext(ScriptDomainManager manager,
        IDictionary<string, object> options)
        : base(manager)
    {
        Binder = new YourBinder(manager);
        manager.LoadAssembly(typeof(string).Assembly);
    } // YourLanguageContext(manager, options)

Next you need to provide an implementation of ParseSourceCode to make the compiler happy and to actually allow your program to parse some source code in your language:

    protected override ScriptCode CompileSourceCode(SourceUnit sourceUnit,
        CompilerOptions options, ErrorSink errorSink)
    {
        // Create lambda expression
        LambdaBuilder program = AstUtils.Lambda(typeof(object), "YourLanguage");
        program.Body = Expression.Block(GenerateStatements(program, sourceUnit.GetCode()));
        LambdaExpression ast = program.MakeLambda();
    
        Expression<DlrMainCallTarget> globalAst =
            new GlobalLookupRewriter().RewriteLambda(ast);
    
        return new LegacyScriptCode(globalAst, sourceUnit);
} // ParseSourceCode(context)

The interesting thing here is how we build an DLR AST from statements (Expression.Block) with help of the LambdaBuilder. The statements do not exist yet, that's what the GenerateStatements method will be used for. We pass the source code to this method and let its do its magic (this is a reduced version, the actual unit test uses a more complex version with some extra code paths and statements to make sure we can initialize the DLR first and to generate code for the next technique too):

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(object), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Convert(
                    Expression.Constant(0.0), typeof(object))
            )
        );
        Expression loops = program.Variable(typeof(object), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Convert(
                    Expression.Constant((int)10000000), typeof(object))
            )
        );
        
        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    Expression.Convert(loops, typeof(int)),
                    Expression.Constant(0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Convert(
                            Expression.Add(
                                Expression.Convert(counter, typeof(double)),
                                Expression.Constant(1.0)
                            ), typeof(object))),
                    Expression.Assign(loops,
                        Expression.Convert(
                            Expression.Subtract(
                                Expression.Convert(loops, typeof(int)),
                                AstUtils.Constant(1)
                            ), typeof(object)))
                ),
                AstUtils.Empty()
            )
        );
        
        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(object) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

This is obviously not dynamic code, it just adds a bunch of expression statements to a list and returns it. Please also note that I tried to use object for counter and loops to demonstrate this in a similar way ToyScript or IronPython will generate code (thats how I figured out some of these expressions). But since .NET or the DLR does not know how to add two objects for example and does not even allow to assign a number to an object, we have to do a lot of casting, which is in essense just boxing and unboxing.

This code is then executed by a unit test, which does basically create a ScriptRuntime for us, makes sure it uses this language implementation and then calls engine.CreateScriptSourceFromString with the code to be executed. Please note that GenerateStatements completely ignores the source code, it will always generate the same statements right now.

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Create runtime
    ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
    Type provider = typeof(YourNamespace.YourLanguageContext);
    setup.LanguageSetups.Add(
        new LanguageSetup(provider.AssemblyQualifiedName, provider.Name));
    ScriptRuntime runtime = new ScriptRuntime(setup);

    // Load Engine
    ScriptEngine engine =
        runtime.GetEngineByTypeName(provider.AssemblyQualifiedName);
    timer.Stop();
    Console.WriteLine("Create DLR runtime: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();
    timer.Start();

    // Execute command
    ScriptSource code = engine.CreateScriptSourceFromString("Objects");
    code.Execute();

    timer.Stop();
    Console.WriteLine("Execute code: " + timer.ElapsedMilliseconds + "ms");

This test will spend about 430ms in the DLR runtime creation (which is kinda slow and will hopefully be faster in the future) and that is the reason it is checked separately. Then it will spend a few ms for the statement creation (4ms as mentioned in the table above) and finally it will execute the code in about 459ms, which is a lot faster than the other DLR languages. But then again, we cheated quite a bit by providing the exact statements we need. The only thing we can optimize is to get rid of all those boxing and unboxing actions. While it is nice to be under half a second for these 10 million while loop iterations with one add to counter and one subtract from loops statement each, it is still a far cry from the performance we have seen in the compiled languages. Let's check out the IL code generated from the DLR AST to see whats going on here:

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] object obj2,
    [2] object obj3)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: box float64
  L_0016: stloc.1 
  L_0017: ldc.i4 0x989680
  L_001c: box int32
  L_0021: stloc.2 
  L_0022: ldloc.2 
  L_0023: unbox.any int32
  L_0028: ldc.i4.0 
  L_0029: cgt 
  L_002b: brfalse L_0035
  L_0030: br L_003a
  L_0035: br L_0063
  L_003a: ldloc.1 
  L_003b: unbox.any float64
  L_0040: ldc.r8 1
  L_0049: add 
  L_004a: box float64
  L_004f: stloc.1 
  L_0050: ldloc.2 
  L_0051: unbox.any int32
  L_0056: ldc.i4.1 
  L_0057: sub 
  L_0058: box int32
  L_005d: stloc.2 
  L_005e: br L_0022
  L_0063: ldloc.1 
  L_0064: call void [mscorlib]System.Console::WriteLine(object)
  L_0069: ldnull 
  L_006a: ret 
}

It might be a little hard to figure out which line does what, but once you found the constants like 0x989680 (which is just 10000000) and the add and sub operations, you can get an idea how the statements were translated. The code between L_0035 and L_0063 is the inner loop and the performance critical code. There are 2 unboxing and 2 boxing operations in that code and probably some other unnecessary assignments.

DLR AST with doubles

While playing around with the code even more and getting to a version like the one that will be shown in this section, I tried to understand why the DLR did certain things in the way that it does. A lot of dynamic behavior is only possible when you use very generic types, but I still think for my language implementation it should be possible to use a much more strict rule set and only allow dynamic code where necessary or wanted, not in thinks like loops or math, where we deal with numbers anyway. Lua is a great example of that, numbers are just numbers. They are always doubles, so there is no need to box or unbox them all the time, which is probably one of the reasons it can handle math better than other dynamic languages.

I also found this useful explanation of what IL the DLR emits for dynamic code, it could explain why stuff is slower than IL code, but still a lot faster than evaluating everything ever time. The DLR does a great job of caching and optimizing dynamic code, but as you can see the IL generated is quite long and verbose too, which is ok for dynamic code I guess. I'm just interested in ways to optimize this further to at least provide proof that the DLR is in fact just as good as a compiler for more static languages.

After the last section it is quite obvious what we have to do do generate statements that can be executed quicker: Get rid of all that boxing and unboxing, use specific data types and make sure all the assignments use the correct format too (else convert constants or other variables). This should not be too hard to do in an actual language once we figure out that we are only dealing with numbers (similar to Lua we could only allow one kind of number, like doubles). This is the updated code for GenerateStatements:

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(double), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Constant(0.0)
            )
        );
        Expression loops = program.Variable(typeof(double), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Constant(10000000.0)
            )
        );

        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    loops,
                    Expression.Constant(0.0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Add(counter,
                            Expression.Constant(1.0))),
                    Expression.Assign(loops,
                        Expression.Subtract(loops,
                            AstUtils.Constant(1.0)))
                ),
                AstUtils.Empty()
            )
        );

        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(double) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

BTW: To analyze the IL code I use the ScriptCode.SaveToAssembly method and use Reflector to show the IL code. As you can see the code has become shorter and we can't see any signs of boxing and unboxing anymore. The code is not even optimal because we use doubles for everything (remember that the C#, C++ and Assembler samples all used a simple integer for the while loop), but as the execution time with less than 100ms shows us, the performance is quite good with this approach.

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] float64 num,
    [2] float64 num2)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: stloc.1 
  L_0012: ldc.r8 10000000
  L_001b: stloc.2 
  L_001c: ldloc.2 
  L_001d: ldc.r8 0
  L_0026: cgt 
  L_0028: brfalse L_0032
  L_002d: br L_0037
  L_0032: br L_0054
  L_0037: ldloc.1 
  L_0038: ldc.r8 1
  L_0041: add 
  L_0042: stloc.1 
  L_0043: ldloc.2 
  L_0044: ldc.r8 1
  L_004d: sub 
  L_004e: stloc.2 
  L_004f: br L_001c
  L_0054: ldloc.1 
  L_0055: call void [mscorlib]System.Console::WriteLine(float64)
  L_005a: ldnull 
  L_005b: ret 
}

This is a more complex version of the above unit test from last section, the 10000000 is printed by the DoLoop DLR code:

runtime creation: 336ms
get engine: 16ms
create script: 4ms
Init code and first script startup time: 232ms
10000000
execute again (DoLoop code): 98ms
and shutdown: 0ms

Okay, I would say this blog post is long enough now and I wrote all day on it and the code samples presented here. Could even be the longest blog post on my blog (html is 150kb). Who knows .. it is probably a good idea to go to bed now.

Good fight, good night, till next time :)

I've been waiting for the latest Irony release the last few days, but luck was not on my side because the good old Roman Ivantsov, who is fearlessly leading the Irony project, was too busy and not able to drop a build yet. But he had a talk at the Lang.NET Symposium 2009 that was going on the last 3 days.

Since I'm starting to write my own DLR language the last few days. I already posted about the DLR and Antlr here, but I switched using Irony together with DLR now, in case you are interested in more links, check out DlrCalculator from the Script.NET guy and the Irony articles on CodeProject.

I will write more about this project when it matures a little bit more, right now I just have a couple of sample projects, lots of unit tests to try stuff out and a very long TODO list and document about what I want to implement. The project itself could go on for a very long time (>1 year). I already attempted to write a language service few years back and got pretty far with the VSSDK, but writing the language itself, the parser, AST generator, etc. was too much work back then. Thankfully with the new cool tools like Irony and the DLR these tasks have become much easier.

Anyway, time to watch some talks from the Lang.NET Symposium 2009, the first batch of videos was just released today, more talks will hopefully follow soon. Thank you very much Microsoft making these available :)

• Direct Link to the Lang.NET Symposium 2009 Talks!

I will start watching the Keynote by Jason Zander - Microsoft,

• then watch C# 4.0 Dynamic by Mads Torgersen - Microsoft,
• Intro to a Visual Studio Language Service by Ted Neward,
• Irony by Roman Ivantsov and
• IronPython Scripting in MS Dynamics by Roman Ivantsov,
• The Visual Studio 2010 Editor by Jack Tilford,
• The Second Life Cloud by Jim Purbrick - Linden Lab and
• then check out the rest of the talks :)

Update: I just heard in the Keynote by Jason Zander that a VS2010 beta release is imminent, very exciting stuff. Finally I can use a beta VS again, VS2008 is getting boring :D

I was going nuts yesterday night after I wrote a couple of unit tests that involved the C# HttpWebRequest class. For some strange reason it always takes 20-30 seconds to create a HttpWebRequest with an url and calling the GetResponse method on Windows 7. I was thinking maybe the server I'm contacting is kinda slow, but it was really fast in the browser. So maybe my SSD drive is messed up? Ok, tried on a normal HDD, same thing. Okay, then maybe TestDriven.net is doing something strange, lets write a simple console application. Wtf, still the same issue! Then I tried it on a different Windows 7 PC, same issue.

I gave up (was very late anyway) and went to work to test it again in the morning on my work PC (also Windows 7). Still the same freaking issue, it takes forever to get anything connected with HttpWebRequest and HttpWebResponse. Getting the actual data is like 30 times faster (and should not matter anyway), still slower than I remembered, but the issue was just in the following two lines (first line took 2-5s, second line 20-25s):

    HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
    HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();

Maybe this isn't the right comic, but its always good to make fun of not-perfectly-working operating systems:




Back to the Problem: I rewrote my console test application again (from last night) and compiled it on my work PC:

using System;
using System.Net;

namespace TestHttpRequest
{
    class Program
    {
        static void Main(string[] args)
        {
            //WTF is going on here? Takes 30s on Windows 7, <100ms on Windows Vista
            string url = "http://www.google.com";

            Console.WriteLine("Connecting: " + url);
            HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
            HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();

            Console.WriteLine("Connected ..");
            if (httpResponse.StatusCode == HttpStatusCode.OK)
            {
                // Dummy
                Console.WriteLine("Getting data ..");
                httpResponse.GetResponseStream();
            } // if
            else
                Console.WriteLine("Failed to get any data ..");

            Console.WriteLine("Done ..");
        }
    }
}

Guess what, still the same issue, takes around 30 seconds to do this very simple job in Windows 7. Could not really figure out the real cause of this (I'm not really that interested, I just wanted to write a couple of unit tests to make some things easier), but it seems like the Conhost.exe file (Console Window Host) of Windows 7 is doing a lot while this is going on (I even have different Windows 7 versions installed, it is the same issue on all of them). Could not see any networking going on (maybe 0.01 kb every 5 seconds) either. If I completely disable networking the GetResponse immediately crashes (it does not take 2-3s for the WebRequest.Create to do its work, and the exception occurs instantly), but that is not really helpful because I wanted to test the response ..

Instead of guessing what this could be about I just wanted to make sure this is a problem with my Windows 7, so I gave my test console application to my colleague that is running Windows Vista and the thing ran in about 20ms, there is nothing wrong with the code, it runs perfectly fine on a non-Windows 7 machine!

So I tried the same stuff in different ways, like using the WebClient class, same result, still 30 seconds for the request.

            // Same stuff with webclient
            WebClient webClient = new WebClient();
            Console.WriteLine("Downloading: " + url);
            byte[] data = webClient.DownloadData(url);
            Console.WriteLine("Done ..");
            

Next I implemented the same stuff with sockets (more specifically TcpClient, which is based on Sockets, but a little easier to use), which is kinda ugly and more work, but at least I finally got the response time I was looking for (<100ms). Microsoft should maybe look into this Windows 7 issue .. for my unit tests I can now use my own HttpWebResponse class and all problems are gone :)

            // Another try, this time with TcpClient ..
            TcpClient client = new TcpClient("www.google.com", 80);

            // Get the stream from the tcp client (allows us to read and write)
            NetworkStream stream = client.GetStream();

            // Write the http request ourselfs (Note: Needs 2 empty lines at the end)
            byte[] requestBytes = Encoding.ASCII.GetBytes(
@"GET / HTTP/1.1
Host: www.google.com
Connection: Close

");
            // Send this ound
            stream.Write(requestBytes, 0, requestBytes.Length);
            stream.Flush();

            // We can't use DataAvailable here because we don't know when the data
            // will arrive. So just call ReadByte until it returns -1 and then
            // we know we are done (Note: This will block our application,
            // if you really care about performance write it asyncronly).
            Console.WriteLine("Getting data ..");
            string result = "";
            int aByte = 0;
            do
            {
                aByte = stream.ReadByte();
                if (aByte >= 0)
                    result += (char)aByte;
            } while (aByte >= 0);
            Console.WriteLine("Result=" + result);
            client.Close();

Pretty good blog post from Jeff Atwood at CodingHorror.com:
http://www.codinghorror.com/blog/archives/001248.html

Lots of interesting comments too and a nice comic:

I have been working quite a bit in Python and Lua last year and I also tried to use those languages in our current game project at exDream (will be announced shortly, so I'm finally allowed to talk about it ^^). While both Python and Lua are written in C (there are obviously other ports like the great IronPython project for .NET, but I'm not going to talk about that in this post), it was not easy at all to make Python run on the Xbox 360 or PS3. I was able in early testing to get some parts running on the Xbox 360, but too much was missed and commented out and the PS3 port looked like way too much work. So I decided to just use Lua, which is probably a better choice for our game anyway since I only use it to do some AI and physics code with it along with some settings. I'm not going to benefit from the Python libraries either way, they are not needed for our game and they won't run on those consoles without serious effort anyway.

Ok, so I was writing some Lua code and used mostly LuaInterface to make the Lua code run from .NET and of course to work with all my .NET code and even XNA. The second step was to make this run in our new game engine, which is written in native code (C++) since the whole engine is based on the Unreal 3 Engine. For that I used the normal Lua 5.1.4 release, which compiles just fine and works great. Since Lua is written completely in ANSI C I was able to quickly test some code on the Xbox 360 and PS3 too. This is important because our game will be a multi-platform title (PC, Xbox 360 and PS3), which is thanks to the Unreal 3 Engine a much simpler task than writing our own engine for all those platforms (it has obviously it own disadvantages, not only do we have to use native code, but we also have to work with a lot of code in the engine that we don't really need for our game).

So far so good. While it sounds like a simple task to make Lua run on all platforms, there is a lot of pain involved to make it run perfectly and behave the same way on all platforms. Since the Unreal 3 Engine is written in C++ and uses its own scripting Language UnrealScript, it does not help out a lot when you want to integrate your own code or libraries in there, so this was half of the work (just see my previous post about compile times and you see why this is a much lengthier process than writing some .NET code). Secondly there are many special rules on both the Xbox 360 platform and the PlayStation 3, some ANSI C methods are just missing or just don't do anything because of the platform design. For example on the Xbox 360 you can't use relative paths, you always have to specify a full path when opening a file, which is something that the Lua engine does not do out of the box. After figuring out where to store the Lua scripts on the consoles and how to load them, everything else just falls into place and works finally as expected.

To make things simpler for other people trying to run Lua code on the Xbox 360 or PS3 I wrote the following little solution:

• LuaXbox360PS3PC.zip (241.92 KB)
  

As you can see on the right the solution consists of 4 projects:

• LuaSupport, which is based on the current Lua 5.1.4 release from 22 Aug 2008 with my own additions to support the XBOX and PS3 platforms (uses XBOX and PS3 defines).
  
  
• TestLuaPC is a simple test project for the PC to make sure everything still runs as expected on the PC. Please note that on the PC Lua assumes that all files can be found in the current directory. I just change to the Lua script directory and then execute the Lua files there normally.
  
  
• TestLuaPS3 will probably only compile if you have the PS3 SDK installed, just unload this project if you don't have it installed yet. Also please test other SDK samples first to make sure everything is correctly setup (it uses a lot of extra include paths). In case you are not able to open the files, check for yourself if the /app_home/Lua/ directory contains the required files after building the project (use the ProDG Target Manager for that). Debugging can also be complicated, I guess it does not work on Windows 7 at all, but I was able to debug and run the app on a normal Windows Vista PC. In any case you can also just run the .elf file from the ProDG Target Manager and then see the output in the console.
  
  
• TestLuaXbox360 is the project to test Lua on the Xbox 360. You will also need the XDK (Xbox Development Kit SDK) installed and have a Xbox 360 DevKit in your network for testing. A TestKit also works, but you won't be able to debug, you can just run the app and check the console output yourself.
  
  

Some of the Lua functionality is not available on the Xbox 360 or PS3 like lua_popen or lua_pclose (among others), and there is also some code that will not do anything like os_execute, os_getenv and some other os methods because they won't make sense on a console anyway or could not be implemented. I did not use these methods anyway, but if you need those and want to write your own code for them, go ahead.

Lets focus on the 2 most important code changes. First of all the Lua default directories had to be adjusted in luaconf.h:83 to support Xbox 360, PS3, PC and Linux platforms:

#if defined(XBOX)
/* On the Xbox we just use the D:\ drive for everything, all lua files should be
   located there */
#define LUA_LDIR    "D:\\Lua\\"
#define LUA_CDIR    "D:\\"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(PS3)
/* On the PS3 we just use the /app_home/Lua/ sub directory for everything,
   all Lua files should be located there */
#define LUA_LDIR    "/app_home/Lua/"
#define LUA_CDIR    "/app_home/"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(_WIN32)
/*
** In Windows, any exclamation mark ('!') in the path is replaced by the
** path of the directory of the executable file of the current process.
** Note: We use the ..\Lua\ directory by default, but that is also the current
** directory, we change it in AIEngineInterface before running scripts!
*/
#define LUA_LDIR    "!\\..\\Lua\\"
#define LUA_CDIR    "!\\"
#define LUA_PATH_DEFAULT  \
        ".\\?.lua;"  LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua;" \
        LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#else
/* Default linux/mac directories */
#define LUA_ROOT    "/usr/local/"
#define LUA_LDIR    LUA_ROOT "share/lua/5.1/"
#define LUA_CDIR    LUA_ROOT "lib/lua/5.1/"
#define LUA_PATH_DEFAULT  \
        "./?.lua;"  LUA_LDIR"?.lua;"  LUA_LDIR"?/init.lua;" \
                    LUA_CDIR"?.lua;"  LUA_CDIR"?/init.lua"
#define LUA_CPATH_DEFAULT \
    "./?.so;"  LUA_CDIR"?.so;" LUA_CDIR"loadall.so"
#endif

And the most important code change is located in luaL_loadfile method, which is called by the lua_dofile macro. Please note that I did not change the code formating and tried to leave as much code as is. I have my own coding conventions, but it does not make sense to apply them when only changing <10% of the code of an existing library.

LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  char fullFilename[128];
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  lf.extraline = 0;
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    // Note: Always open as the requested file, Lua should not care about
    // our crazy directory remapping.
    lua_pushfstring(L, "@%s", filename);

#if (PS3)
    // On the PS3, check always the /app_home/Lua/ directory!
    // This works both for dev-testing via remote HDD and on the game disc.
    // These work probably too just for developing, but app_home is fine!
    // /host_root/Lua/ or /dev_bdvd/PS3_GAME/USRDIR/Lua/
    CheckForValidFilenameWithPath(fullFilename, filename, "/app_home/Lua/");
#elif XBOX
    // For the Xbox we have to make sure always to load files from D:\ because
    // fopen ALWAYS expects a full paths, there are no current directories on the
    // Xbox 360 and therefore no relative paths! Check always "D:\Lua\<file>"
    CheckForValidFilenameWithPath(fullFilename, filename, "D:\\Lua\\");
#else
    // On the PC we just use the default search logic (see luaconf.h) and we
    // don't care about directories since we will already be in the correct one!
    // In earlier versions we had a lot of extra checks here.
    strcpy(fullFilename, filename);
#endif

    // Rest of this code is untouched, we just use fullFilename now!
    lf.f = fopen(fullFilename, "r");
    if (lf.f == NULL)
      return errfile(L, "open", fnameindex);
  }
  c = getc(lf.f);
  if (c == '#') {  /* Unix exec. file? */
    lf.extraline = 1;
    while ((c = getc(lf.f)) != EOF && c != '\n') ;  /* skip first line */
    if (c == '\n') c = getc(lf.f);
  }
  if (c == LUA_SIGNATURE[0] && fullFilename) {  /* binary file? */
    lf.f = freopen(fullFilename, "rb", lf.f);  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, "reopen", fnameindex);
    /* skip eventual `#!...' */
   while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;
    lf.extraline = 0;
  }
  ungetc(c, lf.f);
  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (filename) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, "read", fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}

The code in luaL_loadfile will call the new method CheckForValidFilenameWithPath, which helps us to build a full path filename for the Xbox 360 and PS3 platforms. It will just add D:\Lua\ or /app_home/Lua/ in front of the filename if it does not exist yet. Here we can see another pain point of C code, it takes so freaking long to do very simple tasks (in .NET this would be a 2-liner).

static void CheckForValidFilenameWithPath(char* fullFilename,
  const char* filename, const char* platformPath)
{
  int i = 0;
  int pathLen = (int)strlen(platformPath);
  // Already have a filename with the correct path?
  if ((int)strlen(filename) > pathLen)
    for (i = 0; i < pathLen; i++)
      if (filename[i] != platformPath[i])
        break;

  // Found path?
  if (i == pathLen)
  {
    // Then just use the existing file, happens when loading libraries from Lua
    strcpy(fullFilename, filename);
  }
  else
  {
    // Copy path
    for (i = 0; i < pathLen; i++)
      fullFilename[i] = platformPath[i];
    // Add the relative filename
    for (i = 0; i < (int)strlen(filename); i++)
      fullFilename[i+pathLen] = filename[i];
    // And finally close the string
    fullFilename[(int)strlen(filename)+pathLen] = 0;
  } // else
} // CheckForValidFilenameWithPath(fullFilename, filename, platformPath)

And that are the most important code changes to make everything run on the Xbox 360, PS3 and PC. Here is the main method for all the TestLua projects (always pretty much the same, only the Xbox 360 uses a slightly different way to print to the console):

int _tmain(int argc, _TCHAR* argv[])
{
    printf("Initialize Lua\n");

    Lua = lua_open();
    luaL_openlibs(Lua);
    lua_register(Lua, "print", lua_print);

    // Simple print test from Lua:
    luaL_dostring(Lua, "print(\"Hi from Lua\")");

    // Now load a Lua file, which also loads another Lua file (LuaUnit.lua)
    if (luaL_dofile(Lua, "SomeClass.lua") != 0)
    {
        printf("Error: Unable to execute Lua file %s: %s\n", "SomeClass.lua",
            lua_tostring(Lua, 1));
    } // if (luaL_dofile)

    // Stop the Lua engine and cleanup
    printf("Closing Lua\n");
    lua_close(Lua);
    Lua = NULL;

    // Keep command window for debug mode (ctrl+f5 does not have this issue)
    char input[100];
    gets_s(input);

    return 0;
} // _tmain

Summary:

• Using Lua on other platforms is easier than Python or other dynamic languages I have checked because Lua is written completely in ANSI
• Making code run on multiple platforms is still hard in C++ because you not only end up with ugly #define code, but you also have to test a lot
• Building native applications is freaking time-intensive, it took me almost a full day to write this app and test it on every platform from serval PCs. In .NET it would take me less than an hour, but then again there is no .NET on the PS3 and our current game engine is using native code anyway.
• On the Xbox 360 you always have to use full paths like D:\Lua\SomeClass.lua, relative paths won't work. The SDK says you should use Game:\, but since our game engine uses D:\ too, I left it that way in this sample too.
• The PS3 is even more complicated with lots more possible mounted paths, but it seems /app_home did always work in both my samples and our game, so I'm sticking with it :)

If you are interested in Lua from .NET, check out the LuaInterface project. Someone also started NUA (Lua on the DLR) 2 years ago, but it seems this project (and many other attempts) have died a long time ago. There was also a port by the Lua guys (especially the great Fabio Mascarenhas, who has also started the LuaInterface project and is still involved) called LuaCLR, but it is still in alpha stage and pretty useless, LuaInterface on the other hand is a fully working library! The difference here is that LuaCLR emits CLR code like any other .NET language, while LuaInterface still relys on the Lua runtime. LuaInterface uses only 2 very simple C++ files, the rest is in C#, but because of that you currently cannot run it with XNA on the Xbox 360. IronPython, while fully written in C#, does not work on the Xbox 360 either because reflection is not supported by the Xbox 360 .NET framework.

And finally here is a link to Lua running on the iPhone: Lua v5.1.3 for iPhone/iPod, kinda cool, but since we can run C# code on the iPhone with help of the great Unity3D engine, I'm happy with that at the moment. The main problem for other languages on the iPhone is the missing libraries anyway (can't use Cocoa or any own UI) ..

Just in case you have not read about it yet: John Carmack has ported the good old Wolfenstein 3D game from 1992 to the iPhone as well.

Last week I finally bought a Intel's X25-M 80GB SSD and I have been using it for about a week now at work. A few weeks back I also posted my impressions about the cheap OCZ 64 GB SSD I used before and how bad it worked out with Visual Studio compiling for me (also some helpful links in there if you want to optimize your SSD).

My first impressions of the Intel X25-M 80GB are as follows:

• Initially very fast read, easily archived over 210 MB/s read performance with HDTune and other benchmarks.
• Blocks above 4kb were faster, smaller blocks were slightly slower. 4096 bytes also seems to be a good NFTS block size for them.
• Writing on it is slower with 70-80 MB/s, which is less than the OCZ SSD I had before in benchmarks. But real world usage tells another story as I was usually limited by the network, reading data from other disks and other factors. Other than the OCZ SSD the Intel one stayed very fast even with many files and programs open and doing crazy compiling in the background.
• Using Windows Vista is fine, but I remembered how much better Windows 7 was for SSD when I was using it a month ago. You can tweak Vista quite a bit with the helpful tips from the OCZ Foraum, but it is not the same thing as having the operating system handle all the dirty stuff for you in a much better way.
• So I'm back at Windows 7 again. Yes some apps and games do not work and I even have problems now doing some work related stuff (building content) for which I have to use another PC or use my secondary disk and boot Vista for that. But most tools I use everyday at work are just fine in Windows 7. Boot time is reduced, memory usage is way better, SSDs work great and the overall desktop experience is just so much better (I love the jiggle windows trick to show the desktop).
• Compile times for our game have gone way down (more on that in a second).
• When I'm going home and do some work there I clearly notice the difference. But even if I just open up FireFox or some other complex program, it seems like it takes forever on a normal hard disk. I was pretty happy with my system at home before, but I have not upgraded it for 2 years. It has never really bothered me before since I can still play all the games in Full-HD and everything is quick in Windows. I will probably wait until some cool new DirectX 11 cards will come out and of course some cool 8-core CPUs for some serious multicore action.
• I can sense a slight slowdown to last week, probably due the fact that I use the SSD each day heavily and it is pretty full. Hopefully it will not get worse over time, but even then I could quickly just copy everything over to my secondary disk, reformat and copy everything back.
• Do not believe most SSD reviews, it is really stupid that in the last 6 months almost no SSD review checked the usability of those things, even Linus Torvalds agrees with that (he is a Intel SSD user too ^^).
  
• Also check out this very good new article on AnandTech: The SSD Anthology: Understanding SSDs

I could probably go on and on about this and there are certainly advantages (yeah, reading is fast) and disadvantages (wtf, this is costly, and you will run out of disk space sooner than you can say "copy this over there"). Lets take a look at some random numbers of compile-time from our current game project (which I'm still not allowed to talk about, so just pretend it is a huge C++ project with many many files). Please note that all those times are not in any way accurate, both because I just checked a few times and estimated others and it is also very different from PC to PC anyway plus there are many other factors like having intermediate files already present or not, etc. I'm also not a hardware tester like Tom's Hardware Guide, go there if you want more and real benchmarks, this is just what a noticed in the last weeks. Keep in mind that I could only overclock my current Intel i7 setup a few days ago since the default CPU cooler by Intel for i7 is really crap, loud and can't cool very good. So I bought a Zalman 9900 instead and it allowed me to go to 3.75 Ghz without any problems (up from 2.66 Ghz default).

Older PCs we have at work are obviously slower anyway, but the point here is that the SSD gives an additional bonus. I also only included the PCs I have worked on, some interns have slower PCs, but they usually do never have to do a full recompile anyway. Keep in mind that I do not compare .NET compile times here (they are all below 2 seconds on all those systems even in the larger .NET projects I have). This is all unmanaged C++ code, but since we also do a lot of our code in scripts we do not have to compile C++ most of the time - and even when - we usually only change certain parts of the code and then a incremental build is used.



PS: I was watching the current South Park S13E02 Episode while writing this and I think it was pretty crappy (superheroes, no real story). Last weeks episode with Kenny's girlfriend was way better and a really good season starter.

One of my interns, Stefan, just send me this, this is hilarious:


The Samsung 256GB SSD MLC still seems to be expensive (and is not even available in online shops here in Germany), costs probably around 800 euro anyway. It does not seem to use the crappy JMicron controller, but I could not find any good benchmarks for it.

Today I bought a Intel X25-M 80G SSD today, which I will use instead of my current one from OCZ, which pretty much sucks for Visual Studio compiling as reported before.

I just copied this whole blog (wow, it was >500 MB already, but that is mostly log files and some downloads) to a better server, which can handle all those requests much better. The last few days the old server was completely overwhelmed (it was just a server with a crappy 1mbit dsl line staying in the office).

This is actually the first time this blog is being hosted on a real server, in the past it was always hosted by my own server staying at home or at the office with some crappy 1-2mbit dsl and a dynamic IP, which changed quite often and confused Google a lot. Hopefully with the new static IP things will be better now :)

Yep, it is time for the MVP Summit again (starts this Sunday) and for that reason I will fly to Seattle once again in a few hours. Seems like I go there every year now ^^

This time I will have some extra time to run around and enjoy the cold weather (its not better here in Germany anyway). I will make photos as usual and have hopefully something interesting to say about whats going on.

Since I have a Zune, an iPod and a Creative Zen player and I will take all of them with me:

• Zune for playing around with XNA
• iPod for many apps and since I'm doing a bit of iPhones development right now
• And the Zen for actually hearing music and watching some movies. I just uploaded a crap-load of stuff onto that thing. Really cool that each movie or tv show is just a few 100 MB.

All 3 devices have pretty cool hardware. The Zune is not used much and kinda expensive, but at least interesting for its XNA capabilities.

The iPod is just way to expensive, but interesting for iPhone development. The iPhone is kinda cheap, but you will need a very expensive phone provider contract, which is over 1000 euros over 2 years here in Germany.

And then there is the Zen (or other devices), which do not have many extra features, but its cheap and it just works. It even got a SD slot so I can extend the flash memory. But the main reason why I use the Zen all the time is the amount of hours I can hear music or watch videos on (Zune or iPods with hard disks just suck in that regard, they will be empty in half a plane ride) and because I can put whatever I like on that thing. There is no stupid Zune player or iTunes to install. It does not tell me that I cannot use any files or that I have connected an iPod on too many PCs. Just drag and drop files over, which I often use for safes or copying files from home to work, or just drag in music, add some podcasts and for the plane ride I now also got some videos on there :)

BTW: Everything with the Zen mp3 player is great, even the software you can optionally install is non-intrusive for the most part. The only little annoyance is that it takes a long time to convert videos to .wmv files with 320x240 pixel resolution (required for playback) using the Creative Video Converter. But there are some other programs that can help:

• I tried AutoMKV, which always crashed for me when trying to do .wmvs, but other formats did work.
• There are tons of iPhone Video converter apps out there, most cost money and almost all of them are pretty crappy (both for performance and stability).
• And then there is the Free Zune Video Converter 1.1, which works pretty nice and is incredible fast.
• After that I stopped searching, there is probably more out there (even converters on the web, where you upload your movies, wtf?). Creative Video Converter is quick and dirty minus the quick, Free Zune Video Converter is fast and easy and VirtualDub is for the more complex jobs ^^

Ok, time to go to bed now, plane is lifting off in a few hours ..

My new PC at work has a 64 GB SSD (Solid State Drive) hard disk and installing and using it went pretty ok. Installing Windows 7 and Windows Vista was pretty quick. Most benchmarks, tools, games and programs run also fine and reading a lot of small or big files is also fast. Even extracting big archives goes quickly. But the one thing I wanted to use the SSD for does not perform very well: Compiling in Visual Studio. Our project at exDream is pretty huge (17 GB, about 30k files) and recompiling everything takes quite a while. Reading small files is working well and it seems like opening and working with files in Visual Studio is also good enough. But once the compilation starts (also happens sometimes when installing programs with lots of little files), the performance goes way down. Once more than 50 files per second are being opened and written to, it takes up to 1 whole second (usually you have access times of <0.1 ms with SSD) to open new files. This is obviously very bad.

For now I'm just using the SSD for Windows and the Program files, but store all the code and projects I want to compile on a normal hard disk, plus I also moved the temp folder to D (the normal hard disk). Now things are pretty good again, but I did not gain the performance improvement I was looking for with this :(

Since I tweaked my Vista quite a lot already (see post below), I already fine-tuned a lot of settings that are also good for SSD, but OCZ has a nice guide with some extra tricks on what to do to get the best performance possible with those cheap SSDs:
General discussion and XP tricks: http://www.ocztechnologyforum.com/forum/showthread.php?t=43460
Vista 32/64 tweaks: http://www.ocztechnologyforum.com/forum/showthread.php?t=47212
Speeding up Vista: http://www.ocztechnologyforum.com/forum/showthread.php?t=43525

The SSD is the OCZ 60 GB SSD V2, which still has a JMicron controller, which just plain sucks for many IO operations. Its specifications say that it can do 170 MB/s reads and around 100 MB/s writes, the truth is more like 130 MB/s reads and around 100 MB/s writes for sequential data. The problem is other when copying files I never have those cases.

My brother has a Patriot Warp v2 SSD, which has similar issues because it also uses a JMicron controller. It is a little slower for sequential reads/writes, but a little faster for writing smaller files.

BTW: I tried using 2 Raptor Hard Disks with a Raid-0 configuration 2 years ago and it was a similar issue, the stupid software raid controller from the mainboard was slower than just using one single hard drive without any raid. Reading data was way quicker, but writing stuff to the raid was just way slower!

It seems only expensive SDDs like Intel's X25-M 80GB SSD (costs around 400 bucks) have a really good controller and can do many reads/writes with small files without any problems.

Ferdi has posted a German Tutorial on how to use Visual Studio for Unity over at the MobileBits blog. This is kinda useful because Unity3d currently only works on the Mac and I'm pretty much unable to use the Mac for anything because I cannot use my own keyboard layout over there. I have been working a little via VNC, but having a familar IDE like Visual Studio is just so much better ;)

An English tutorial can be found at the UnifyCommunity site:
http://unifycommunity.com/wiki/index.php?title=Setting_up_Visual_Studio_for_Unity

Basically you create a console application in Visual Studio where you write all the c# scripts for Unity. Then a little FTP upload tool is used to upload those c# files to the FTP server on the Mac, where you can then compile the scripts in the Unity IDE. More on that next week. I have not worked much with Unity yet because we have been busy at exDream with our current project and we got many new interns this week plus marketing guys coming by and stuff like that.

I recently build a new work pc and used a small SSD hard drive (64 GB) for it. I was using Windows 7 last week, but since so many programs (daemon tools), drivers (printer) and especially games (everything new?) did not work I just switched back to Windows Vista x64 SP1 yesterday.

One of the problems with this small SSD drive is that it gets full really quick. Especially with Vista, which takes almost 20 GB right out of the box. Then you have Visual Studio, all the tools and you are left with 20-30 GB. Our current project is huge and can take up to 37 GB (thanks to subversions great feature to double any project size, arg) and I still want it on my SSD drive because it is so much faster to work with many little source code files (Visual Studio is just not the same anymore, everything is soooooo much faster ^^).

Well, the point of the story is, with a really small drive it is kinda important to delete everything that I do not need. So I decreased the page file to 256 MB (from 6GB) and allowed the system to use the secondary drive to use a page file (there I have 500 GB free space). I also deleted everything I could from windows, program files or users (don't do this at home, I backed up everything on my Server where I have like 12 TB of space, hehe). A few more GB free space, yay. I also messed with Subversion and killed all the .svn directories for artist stuff I do not update very often, so I still got around 15 GB free right now :)

But I still wanted to delete the hiberfil.sys file, which takes up several GB too, but unlike Windows XP there is no option to do that. So you have to type in the following command (or create a simple .cmd file for that):

powercfg.exe -h off

And to enable it again type:

powercfg.exe -h on

which is useful if you really need hybernation. I use it all the time on my laptop, and Riley is using it in Alien too:

I have updated my CR_Commenter plugin quite a lot last weekend and tested it all week. CR_Commenter v1.9 new features include full hotkey and feature customization, much better support for C++, Lua, Python and way better light-commenting features and more powerful and stable region features. Thanks to the many people writing me emails using the addin or even improving it themselfs and sending me updated code back (full thanks are listed below).

Download links for CR_Commenter v1.9 (2008-11-27):

• Click here to download the installer: CR_Commenter.exe v1.9 (98 kB)
• And here is the sourcecode for: CR_Commenter_src.zip v1.9 (90 kB)

If you do not know about the CR_Commenter plugin for CodeRush yet (I originally wrote it back in 2004 and still use it every single day), please read the article for version 1.8 from earlier this year where I also added a lot of features and this article of version 1.7 from 2006 or check out the original Commenter - A CodeRush plugin which helps you to comment your code article on CodeProject.net from 2004, which I have finally updated too (its about time).

To use CR_Commenter you need to install CodeRush 2.0 or later, which is a commercial product you can get from DevExpress here. You can also use the free DXCore component with a reduced feature set, but all the basic CR_Commenter features should work fine there too. Please note that there is also a recently released free CodeRush Xpress Edition, which should work too, but I have not tested it.

If you have used CR_Commenter before the first thing you will probably notice is the completely new options screen:


Let me go through all the new and improved features. Over the years so many things were added to CR_Commenter and since the options screen, hotkeys and feature set has changed so much all the time, it is hard to keep track what features are available in the latest version. To make it easier for new users to understand what you can do with CR_Commenter v1.9 I decided to explain every feature, which might also be useful for existing users as they might have missed some of the cool features before. All these features can now be enabled or disabled and you can set your own hotkey for most of them (if it makes sense). I will go through the same list as seen in the Options screen, which is also represented in the FeatureHotkey.cs class that has a lot of comments and explanations for each feature. As mentioned before all Hotkeys can be changed to whatever you like. Please note that the pictures might look a little different than your Visual Studio code, I just use a different theme and bigger font sizes, it is still Visual Studio 2008.

CR_Commenter v1.9 Features:

• Automatically add comments after closing blocks with }
  Hotkey: }
  This is the most basic feature of CR_Commenter. In v1.9 you can now enable or disable it as you like. Please also note that you can now specify to only generate comments for blocks greater than a certain size (by default 5), which reduces the amount of comments generated greatly and is also the new default setting! Note: Since this is only parsed after pressing } it only works for C like languages like C, C++, C#, fx, UnrealScript, etc. For some of the newly supported languages like Lua (or VB, which is still untested and more or less unsupported sadly) or even Python that use identation for blocks you have to use Ctrl+1
  
  
  
  
• Force Comment Block
  Hotkey: Ctrl+1
  Same as }, but allows us to comment whole classes and methods easier by positioning the text cursor on what you want to update. Use Ctrl+6 for always updating the whole file no matter where you are located. Can also be used for languages not using } for commenting or if you have disabled } commenting.
  
  
  
  
• Add namespace for the current word. (Disabled by default)
  Hotkey: Ctrl+2
  Used to support automatically adding a namespace in C# 3.0/VS 2008. Not longer needed since VS supports it on its own now! This feature could be used back in 2006 to add namespaces in VS 2005.
  
  
  
  
• Autogenerate region around the current block
  Hotkey: Ctrl+3
  Autogenerate region around the current block (method, property, constructor, etc.), for selections use Ctrl+R (default CodeRush hotkey). The most important thing about this feature is that it will automatically give your new region the method name and it will make sure all comments belonging to the method are put inside the region. Note: In the picture the #region SetNewValue and #endregion parts are hidden by CodeRush with lines to make the source code look cleaner.
  
  
  
  
• Autogenerate regions for the whole file (all methods)!
  Hotkey: Shift+Ctrl+3
  Same as Ctrl+3, but will go through the whole file and generate region blocks for every method that does not seem to have one yet! You can also just go to the class level and press Ctrl+3 to generate all regions for all class methods. Since v1.9 also works with C++ and will generate #prama regions for you. BTW: Parsing the whole file with Ctrl+1 will also generate a includes region automatically for you.
  
  
  
  
• Toggle collapsing and expanding current selection
  Hotkey: Ctrl+4
  Very useful to avoid having to use the mouse for these things. Please note that Visual Studio also supports collapsing sections with Ctrl+M+M, but that will only work on sections, not on regions, which is a pretty useless feature in C#, Ctrl+4 is much better :) Since v1.9 also supports regions if CodeRush does not parse them (e.g. #pragma region in C++), so you can now even collapse C++ regions.
  
  
  
  
• Toggle collapsing and expanding everything at top level
  Hotkey: Ctrl+5
  Will expand or collapse all regions in the current file at once. First time pressed it expands everything, second time everything is collapsed, good for getting a quick file overview. Note that again Visual Studio supports collapsing all sections with Ctrl+M+L, but this really sucks because all methods, xml comments and everything else is collapsed too and expanding all of that again is a pain. Again, Ctrl+5 is much better.
  
  
  
  
• Update all comments and xml blocks for the whole file.
  Hotkey: Ctrl+6
  Similar to Ctrl+1, but will not only process the current section, but the whole file. This generates and updates all comments in the whole file, updates the header comment and makes sure all Xml comment blocks are generated if missing (see below).
  
  
  
  
• Update just the header comment.
  Hotkey: Ctrl+8
  Force updating the header comment, can be useful to test new headers and to remove any old header data. Also updates the created and modified timestamps. Since v1.9 changing or having different header comments now works, but it will also remove any existing header comments to replace with the new header comment if the format is different. Also check out the new cool options in the Commenter options screen, which allows you to specify seperate header comments for each solution or project if you like.
  
  
  
  
• Generate xUnit or NUnit unit tests automatically for us
  Hotkey: Ctrl+9
  Generate xUnit unit tests automatically for us for all methods! All unit tests will have to be written and tested until everything is implemented correctly. Existing test methods will not be overwritten! You can either enable xUnit Tests (like in the screenshot below) or NUnit Tests.
  
  
  
  
• Remove All Comments, use this to cleanup a file
  Hotkey: Ctrl+0
  Remove All Comments, use this to cleanup a file. It will only remove generated comments (after } blocks). Also nice for testing and cleaning up other peoples code.
  
  
  
  
• Obfuscate whole file (removes ALL comments, xml comments, empty lines, headers, etc.)
  Hotkey: Ctrl+\
  Obfuscate whole file. All code will stay the same, but we will remove ALL comments, Xml Comments, empty lines, headers, etc. to reduce file size and make code harder to read. Not really a Commenter feature, but I implemented this for another app and it was easy to implement here too.
  
  
  
  
• Autogenerate XML Comments
  This feature will automatically add XML Comments to methods, properties, constructors, classes, etc. when the normal comments or generated. You can specify if you only want to generate XML comments for public methods or if you want to generate XML tags for all parameters (which was default on, but is default off now to produce much less empty xml parameter comments, which are usually not updated by the programmer anyway).
  
  
• Other Improvements in v1.9
  
  • Added cool new per-project and per-solution header comments
    
    
  • New shortcut Ctrl+6 to just update the header
    
    
  • Added lower limit for block size to reduce comments for small blocks (also works recursive)
    
    
  • Added new features and hotkey logic, everything now easier to customize
    
    
  • Easier visibility if plugin is installed and working (more visual events, better options screen)
    
    
  • Fix generating comments for new files (messes up the using directives region), seems to be related to generating header comments
    
    
  • Improve line length count for removing comments (good new standard: 5 lines)
    
    
  • Fixed several problems in Format-Strings or SQL code.
    
    
  • Added whole method identifier from the accessibility keyword to the closing parenthesis for region directives.
    
    
  • Improved the CodeRush-autogenerated regions (Ctrl+3), make sure comments are not messed up
    
    
  • New default: Do not generate Xml parameter comments (mostly not used for better documentation anyway, adding them yourself for more details is more comfortable)
    
    
  • Added C++ "pragma region - #pragma endregion" support (thanks for testing to Enrico)
    
    
  • C++: include includes into using directives region
    
    
  • C++: Fixed header comment (project name could not be found)
    
    
  • C++: Do not use xml comments, just leave normal comments!
    
    
  • Fixed wronly generated regions outside of our plugin
    
    
  • C++: Fixed #includes region, do not modify include lines
    
    
  • Fixed region generation when CodeRush generates region before us!
    
    
  • Fixed header in case some other header comment is used
    
    
  • Added multiline header comment support for // /* */ -- and # to support all kinds of languages and comment formats
    
    
  • Also force generate comments on properties (methods work, also test constructors)
    
    
  • C++ allow Ctrl+4 and Ctrl+5 for C++ region collapsing (using visual studios Edit.ToggleOutliningExpansion and Edit.ToggleAllOutlining as fallbacks)
    
    
  
  
• Older and Improved Features (from v1.8 and before):
  
  • Fixed '}' inside string or comment, even if CodeRush has not processed the whole file yet (double checking now). E.g. the following string will not longer produce any comments:
    SQL = String.Format("UPDATE GC SET Status = {0} WHERE ID = {1}", Msg, ID);
    
    
  • Supporting to generate regions again, but split up function names, also suppress CodeRush region generation with Ctrl+3, Ctrl+R (also suppressing VS hotkeys now) Better support for auto generated region blocks for big methods, just press Ctrl+3 to generate region (let CR_Commenter choose the name automatically, allow editing after pressing Ctrl+3).
    
    
  • Fixed #if DEBUG in Using directives (caused trouble, was removed)
    
    
  • Also added code to always remove #if DEBUG and #endif, especially at the beginning and end of the Using directives region
    
    
  • Fixed double line #using Using directives issue when using directives got mixed up.
    
    
  • Fixed long line commenter generation, will now still work, but not longer cut of any of your code. If you have long code lines (>100 letters), you can still generate comments for them and the Commenter will not longer cut anything off (you are responsible for formating the code).
    
    
  • Fixed auto-collapsing of regions, especially when closing blocks in methods, the method should not auto-collapse!
    
    
  • Checked support with other languages (which are at least similar to C#) like Lua, Python, etc.
    
    
  • In Python/Lua, etc. we had to disable } comment generation and also use other comments (# for python, -- for lua) when using Ctrl+1 for header generation.
    
    
  • Test all other hotkeys and make sure they do not affect the code badly in CodeRush unsupported languages (note: VB is still not supported since it is very different from c# code).
    
    
  • Supports now any unit test framework, actually does not care anymore what you use, only the unit test generation code will add Xunit, but you can easily change that code if you want to.
    
    
  • Options are now available directly into Editor tree of the CodeRush options!
    
    
  • Refactored all classes, added about 4 new classes and removed 1 unused class. The source code is now much easier to read, to follow and to change.
    
    
  • Remove commented out code, cleaned up the source code a lot.
    
    
  • And finally applied the commenter to the commenter source code :)
    
    

I would also like to thank all the people that wrote me emails, use the plugin, all my colleagues that use this tool everyday too (shoutout to Enrico Cieslik and Henning Thöle, and also Fabienne Khöle) and the following people who have contributed with ideas, code improvements and helpful bug reports (they are also mentioned in the TODO.txt file for their specific bug report or improvement idea): Andy (mangledmonster), Andrew Jones, Gil Yoder, Alexey Romanov, Enrico (Judge), Ed Blankenship, Sven Heitmann, Jan Diederich, Rob Heyes, Dan Avni, Andy Jacobs and everyone else :)

I watched quite a lot of PDC 2008 Talks this week and I really like that they put all that stuff online for everyone to watch. To learn more about the PDC, check out event photos or see all the sessions, directly go to http://www.microsoftpdc.com/. Another good list of all sessions can be found here: http://coolthingoftheday.blogspot.com/2008/10/pdc2008-quick-video-link-list.html

Here are my personal favorite session recordings (from what I have seen, I have not seen all talks):

The Future of C# (by Anders Hejlsberg)
http://channel9.msdn.com/pdc2008/TL16/
As always Anders is fun to watch and you learn a lot about C#, the new dynamic keywords and whats to come. Not really stuff you will probably use right now, but good to know what we can do and probably will do in a few years. In case you want to watch IronPython or IronRuby, this session is a good prerequisite too.

Deep Dive: Dynamic Languages in Microsoft .NET
http://channel9.msdn.com/pdc2008/TL10/
Jim Hugunin (the creator of IronPython and DLR) talks about IronPython and the DLR (Dynamic Language Runtime) in general. Fun to watch, interesting insights and generally always interesting to anyone interesting in dynamic languages.

Panel: The Future of Programming Languages
http://channel9.msdn.com/pdc2008/TL57/
I like Panels like this and they had really great guests: Gilad Bracha, Douglas Crockford, Anders Hejlsberg, Erik Meijer, Wolfram Schulte, Jeremy Siek. They get a little bit off-topic at the end, but overall an interesting and very fun to watch discussion with a lot of interesting arguments on all sides (lot of different languages are discussed).

IronRuby: The Right Language for the Right Job
http://channel9.msdn.com/pdc2008/TL44/
I have learned Ruby few months ago and used it for a few days, but I never really liked it and continued to use Python instead, but John Lam (the IronRuby inventor) is a cool dude and give a lot of tips and shows off a lot of cool demos, which are equally interesting even if you would use some other dynamic language than Ruby.

Microsoft Visual C# IDE: Tips and Tricks
http://channel9.msdn.com/pdc2008/TL46/
Interesting and useful Tricks, Dustin shows off a lot of shortcuts and the whole presentation is in Visual Studio, which is cool.

Visual Studio Debugger Tips & Tricks
http://channel9.msdn.com/pdc2008/TL59/
I like Tips & Tricks talks and this one is good too. Informative and fun to watch as well. I will probably watch it again some time soon, like I will with some of the other talks too (because I usually code or do something else while the videos are running on a secondary monitor ^^).

Mono and .NET (by Miguel de Icaza)
http://channel9.msdn.com/pdc2008/PC54/
One of the last talks I discovered after watching many other talks. At first I thought this might be not really interesting for me, since this will probably focus only on Linux and Mono development, but I could not be more wrong. This was a really great talk, Miguel is such a great presenter, even when things go wrong, he makes fun remarks and is always on the topic. He gives great insight of game programming on Mono for Linux, the Mac and even the iPhone, which sounded really cool to me. Other topics were also interesting and fascinating.

Managed Extensibility Framework: Overview
http://channel9.msdn.com/pdc2008/TL33/
If you were every interesting in extending Visual Studio or want to hear about good ideas on how to extend frameworks or applications, this talk is helpful and insightful.

Visual C++: 10 is the New 6
http://channel9.msdn.com/pdc2008/TL13/
I'm not doing much C++ coding anymore, but many years ago this was my main programming language and it is still the most used language for games. This talk is highly informative and Boris Jabes talks in detail about new C++ Features in VS2010 and what is possible, what will get better and what even features will come in the future after VS2010. Also a lot of interesting questions and answers at the end.

The two talks Parallel Programming for C++ and Parallel Programming for Managed Developers were also highly informative and gave us programmers some good ideas on how to make better use of Multi-core systems. I especially like the Talk Parallel Library (TPL), which is insanely easy to use:

Parallel.For(0, 100, i => a[i] = a[i] * a[i]);

Microsoft XNA Game Studio: An Overview
http://channel9.msdn.com/pdc2008/TL43/
Frank Savage from the XNA Team talks about XNA Game Studio 3.0, which is highly informative if you don't know much about XNA yet. For me there was not much new to learn in this talk, but Frank is such a cool guy and it is fun to watch how he presents his stuff, which is one more reason to watch this talk. He also shows off some XNA games and the recently XNA Dream Build Play contest winner games.

The keynotes from Day 1 (Azure, Infos) and Day 2 (Windows 7, Visual Studio 2010 and .NET 4) were also interesting to watch. Other sessions sounded also interested and from what I have seen Oslo, Azure, Silverlight, Unit Testing in VS2010 and other new stuff is cool too, but I have no plans to use that stuff anytime soon (maybe Oslo for parsing text, which is easy and cool).

I hope you enjoy these videos as much as I do, definitely a great idea to put them all online :)

Since I'm not posting many XNA articles lately, lets announce some stuff from Ziggyware :)

Ziggyware Fall 2008 XNA Article Contest


Start date: October 12'th 2008
Ending Date: November 30'th. (Midnight CST)

Contest Subjects

• Write an XNA related tutorial for XNA developers (see examples)

Important: Read How to create an article for Ziggyware - Entries are subject to approval and may be disqualified per disgression.

Prizes

• First Place
  • XBox 360 Elite (USA NTSC)
  • One Year subscription to the Microsoft XNA Creators Club
• Second Place
  • One Year subscription to the Microsoft XNA Creators Club

More information can found at Ziggyware.com:
http://www.ziggyware.com/news.php?readmore=905

As you can read on Scott Guthrie's Blog the Visual Studio 2008 and .NET Framework 3.5 Service Pack 1 Beta can be downloaded now. It has a amazing set of features and improvements and I highly recommend it, especially if you live on the bleeding edge :) I like the VS IDE and ASP.NET Routing Engine (previously part of the new ASP.NET MVC framework) the most.

David Hayden has summarized the list of changes (Scott's Blog posts are usually very long ^^):

• ASP.NET Dynamic Data -  The wonderful ASP.NET RAD Data Scaffolding Support similar to what you get from Ruby on Rails.
• ASP.NET Routing Engine - The routing engine we have been enjoying in the ASP.NET MVC Framework and ASP.NET Dynamic Data.
• ASP.NET AJAX Back/Forward Button History Support - What you know and love from the ASP.NET 3.5 Extensions
• ASP.NET AJAX Script Combining Support - Haven't seen this one before. It allows you to declaratively define multiple script references within a new custom tag.
• Visual Studio 2008 Performance Improvements HTML Designer and HTML Source Editor - Always good to see :)
• Visual Studio 2008 JavaScript Script Formatting and Code Preferences
• Better Visual Studio Javascript Intellisense for Multiple Javascript/AJAX Frameworks - This will help with jQuery, cool!
• Visual Studio Refactoring Support for WCF Services in ASP.NET Projects
• Visual Studio Support for Classic ASP Intellisense and Debugging - Need this for ASP.NET MVC Framework Views.
• Visual Web Developer Express Edition support for Class Library and Web Application Projects
• SQL Server 2008 Support
• ADO.NET Entity Framework and LINQ to Entities - We expected this. Awesome!
• ADO.NET Data Services (formerly code-named "Astoria") - This is awesome, too!
• C# - The C# code editor now identifies and displays red squiggle errors for many semantic code issues that previously required an explicit compilation to identify. Every little bit helps.

Read all this in more detail on Scott's blog or just download VS2008 SP1 Beta.

As you can read on the XNA Team Blog the first CTP (kinda early beta/alpha) for XNA Game Studio 3.0 is available for download now.
The major new features beside full VS 2008 (Express, Standard, Pro, etc., but not longer VS 2005) support you can now write and deploy games for the Zune. Luckily for me I own one (not many non-US people do since the Zune is ONLY available in the US). Please note that while the Zune is a powerful device it obviously will NOT support any advanced rendering techniques at all, but all the other parts of the XNA Framework are supported very well. This means 3D games or shader based games will be possible on the Zune and the display is just 320x240 pixel and there are other things you have to think about (controls and playing music), but other than that it should be pretty easy to start developing Zune games right away :)

While the new features are really cool and I definitely will check out the Zune development options (XNA MVPs already saw some cool samples and stuff at the MVP Summit about this ^^), the show stopper for me is the "no 64 bit support" issue. All my PCs (work, home, laptop and even the servers) run on 64 bit vista/win2003/win2008, it will not be easy to play around with XNA 3.0 ("Hey, go away from that PC, I wanna test something").

Anyway, check out XNA Game Studio 3.0, especially if you own a Zune and find this Zune-targeted game development cool. Other features worth checking out are better sound APIs in the new Media namespace, namely the SoundEffect and MediaPlayer APIs. They allow you to play sounds and music much easier and in more formats (yeah, mp3s).

I worked on the CR_Commenter v1.8 Update all weekend and tested it a little bit today at work. If you do not know about the CR_Commenter plugin for CodeRush yet (I originally wrote it back in 2004 and still use it every single day), please read this article of version 1.7 from 2006 or check out the original Commenter - A CodeRush plugin which helps you to comment your code article on CodeProject.net from 2004

Here are the coolest new features in v1.8:

• New and updated hotkeys, also made sure all of them work as advertised! You can also now switch on and off much more features and hotkeys if you like (e.g. disable Ctrl+1, Ctrl+3 or Ctrl+.). Normally the commenter will do his work every time you press '}' while coding and automatically add comments, xml blocks and the header (no need for any hotkeys during normal coding). But you can also press the following hotkeys to perform special actions:
  
  
  • Ctrl+1: Does perform the same action as closing a block with '}', but can be used anywhere inside a method or class.
    
    
  • Ctrl+2 or Ctrl+.: Adds a namespace for the current word at the cursor. Used to support automatically adding a namespace in C# 3.0 (back in v1.7), but now works the same way as VS 2008 plus saving you from pressing Enter to accept a namespace suggestion. If you like the VS2008 way better (allowing different namespaces), just disable Ctrl+. and VS will work as before.
    
    
  • Ctrl+3 or Ctrl+R: Auto generate region around the current block (method, property, constructor, etc.), for selections use Ctrl+R. You can also press Shift+Ctrl+3 to process the whole file! This feature was originally added in CodeRush v1.5, but abandoned in v1.6 and v1.7 after CodeRush supported it internally. Now the feature is back with lots of improvements, auto generating regions all over the place and producing region names for you. Can of course be disabled if you like the CodeRush way better (having to type the region name yourself).
    
    
  • Ctrl+4: Toggle collapsing and expanding current selection. This feature often caused trouble in v1.7 because it collapsed sometimes regions you were working in (e.g. method blocks with sub regions), now it will not longer disrupt your work and never close any region you are working on (except you want to with this hotkey).
    
    
  • Ctrl+5: Toggle collapsing and expanding everything at top level. Very cool to quickly check out what this file is about, use it in conjunction with first generating all region blocks (Shift+Ctrl+3).
    
    
  • Ctrl+6: Update all comments and xml blocks for the whole file. Was Ctrl+9 before and Ctrl+F9 even before that, this method is not required very often because Ctrl+1 handles most of this already. Please also note that Ctrl+7 was abondoned for now (resort language blocks and regions) because this feature is very complex and not implemented yet (see above for v2.0).
    
    
  • Ctrl+9 or Ctrl+F9: Generate all unit tests automatically for us (I use F9 to test, Ctrl+F9 makes sense to generate tests)! This will add a sub class called Tests, which contains all public methods as tests, which will all fail by default. Then all unit tests will have to be written and tested until everything is implemented correctly. Existing test methods will not be overwritten! This features is the major improvement in v1.8 for me, it will add unit tests automatically for us at the bottom of file to provide 100% code coverage (all unit tests should fail initially, add and change code to make them work).
    
    
• Thought about auto sorting and removing namespaces for all files, but that works just fine with the new VS2008 Powertools.
  
  
• Fixed '}' inside string or comment, even if CodeRush has not processed the whole file yet (double checking now). E.g. the following string will not longer produce any comments:
  SQL = String.Format("UPDATE GC SET Status = {0} WHERE ID = {1}", Msg, ID);
  
  
• Added option to not update comments on } (or Ctrl+1)
  
  
• Supporting to generate regions again, but split up function names, also suppress CodeRush region generation with Ctrl+3, Ctrl+R (also suppressing VS hotkeys now) Better support for auto generated region blocks for big methods, just press Ctrl+3 to generate region (let CR_Commenter choose the name automatically, allow editing after pressing Ctrl+3).
  
  
• Press Shift+Ctrl+3 to generate all regions at once for the whole file! The commenter will also not longer produce regions were you already have some (even if they are different), especially if you press Ctrl+3 multiple times you will see what I mean ^^.
  
  
• Fixed #if DEBUG in Using directives (caused trouble, was removed)
  
  
• Also added code to always remove #if DEBUG and #endif, especially at the beginning and end of the Using directives region
  
  
• Fixed double line #using Using directives issue when using directives got mixed up.
  
  
• Fixed long line commenter generation, will now still work, but not longer cut of any of your code. If you have long code lines (>100 letters), you can still generate comments for them and the Commenter will not longer cut anything off (you are responsible for formating the code).
  
  
• Fixed auto-collapsing of regions, especially when closing blocks in methods, the method should not auto-collapse!
  
  
• Checked support with other languages (which are at least similar to C#) like Lua, Python, etc.
  
  
• In Python/Lua, etc. we had to disable } comment generation and also use other comments (# for python, -- for lua) when using Ctrl+1 for header generation.
  
  
• Test all other hotkeys and make sure they do not affect the code badly in CodeRush unsupported languages (note: VB is still not supported since it is very different from c# code).
  
  
• Supports now any unit test framework, actually does not care anymore what you use, only the unit test generation code will add Xunit, but you can easily change that code if you want to.
  
  
• Options are now available directly into Editor tree of the CodeRush options!
  
  
• Refactored all classes, added about 4 new classes and removed 1 unused class. The source code is now much easier to read, to follow and to change.
  
  
• Remove commented out code, cleaned up the source code a lot.
  
  
• And finally applied the commenter to the commenter source code :)

Some cool and highly complex ideas (like the auto refactoring and resorting of all code elements in a file) did not make it into v1.8 and will probably lay on ice for a while until I start developing CR_Commenter v2.0. But other than that I'm pretty happy with the new version and both the new region generation code and the new unit test generation code is really helpful, plus the many fixes and improvements of existing features make it more enjoyable to use CR_Commenter.

Download CR_Commenter v1.8 here (free as usual, you will need DXCore, which you can download here freely):

• Note: Check out the CR_Commenter link on the right side for the most recent version!
• Click here to download the installer: CR_Commenter_v18.exe v1.8 (95 kB)
  
  
• And here is the sourcecode for: CR_Commenter_v18_src.zip v1.8 (90 kB)
  
  

After updating so much on my blog the last 2 days and posting all the unfinished posts I had prepared a few weeks ago I finally reach the last important one I wanted to make. But due the fact that I will fly to the MVP summit tomorrow I better not stay away too long. Hopefully this post still contains all the vital information. Let's get started.

First of all you need the latest version of the free NSIS (Nullsoft Installer System):

• http://nsis.sourceforge.net/Main_Page

There are TONS of information on that website and if you are new to the NSIS scripts be sure to read at least the quick start guide (called simple tutorials) and use the NSIS Forums search feature for more specific problems and questions.

Good, now just let us go quickly through the .nsi file I currently use for all my XNA Game Setups. Please note that this file has evolved over many years (I guess I started using NSIS in 2002) and parts might not be simple to read or even be how it is done today, but the script was always a very helpful and quick way to create Setups for me. While detecting all the requirements to make XNA work is not a piece of cake many NSIS tutorials helped me to figure all the parts out.

The following .nsi script is from the previous blog post "Loading Collada Models in XNA 2.0 and doing Skinning and Animation" and was used to build the XnaSkinningColladaModelsSetup.exe (2.2 MB) file. If you want to use it for your own XNA Game project just use a text-editor and replace "XnaSkinningColladaModels" with your own game name (it occurs many times in the file). Also create a .bmp file with the name of your game for the setup header (check MUI_HEADERIMAGE_BITMAP) or comment the 2 Header Image lines out with a ; or # if you do not want any header image. All files are grabbed from the /Release directory and will be installed to the target (make sure that you only have files in the /Release directory that you want to have deployed). This should be already enough information to make your setup work, the rest of this article will just go into the details of the script.

The XnaSkinningColladaModelsSetup.nsi file is organized in 6 parts (Initialization, onInit, File section, Optional sections, Helper functions and Uninstall):

• Initialization: Just sets all the names, the required modules, and how the setup is displayed to the user (all the pages, see MUI_PAGE). Version information and file information are also a vital part here and you should enter some meaningful values here. Also put any language strings for different languages (this installer is just using english text) at text part of the initialization code.
  

  !include "MUI.nsh"
  SetCompressor /SOLID lzma
  
  ;Name and file
  Name "XnaSkinningColladaModels"
  OutFile "XnaSkinningColladaModelsSetup.exe"
  
  ;Default installation folder
  InstallDir "$PROGRAMFILES\XnaSkinningColladaModels"
  
  ;Get installation folder from registry if available
  InstallDirRegKey HKLM "Software\XnaSkinningColladaModels" ""
  
  !define MUI_COMPONENTSPAGE_NODESC
  
  ;Interface Settings
  !define MUI_ICON "${NSISDIR}\Contrib\Graphics\Icons\modern-install-blue.ico"
  !define MUI_UNICON "${NSISDIR}\Contrib\Graphics\Icons\modern-uninstall-blue.ico"
  !define MUI_HEADERIMAGE
  !define MUI_HEADERIMAGE_BITMAP "XnaSkinningColladaModels.bmp"
  !define MUI_HEADERIMAGE_UNBITMAP "XnaSkinningColladaModels.bmp"
  !define MUI_ABORTWARNING
  
  ;Pages
  !insertmacro MUI_PAGE_WELCOME
  !insertmacro MUI_PAGE_COMPONENTS
  !insertmacro MUI_PAGE_DIRECTORY
  !insertmacro MUI_PAGE_INSTFILES
  
  !insertmacro MUI_UNPAGE_CONFIRM
  !insertmacro MUI_UNPAGE_INSTFILES
  
  ;Languages
  !insertmacro MUI_LANGUAGE "English"
  
  ;Version Information
  VIProductVersion "1.0.0.0"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "ProductName" "XnaSkinningColladaModels"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "Comments" "XnaSkinningColladaModels project, see http://BenjaminNitschke.com"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "CompanyName" "BenjaminNitschke.com"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "LegalCopyright" "© BenjaminNitschke.com 2007"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "FileDescription" "XnaSkinningColladaModels"
  VIAddVersionKey /LANG=${LANG_ENGLISH} "FileVersion" "1.0.0.0"
  
  ; The text to prompt the user to enter a directory
  ComponentText "This will install XnaSkinningColladaModels for XNA 2.0."
  
  UninstallText "This will remove XnaSkinningColladaModels for XNA 2.0 from your computer."
  
  ; New .NET Check function, we want at least v2.0
  !define DOT_MAJOR "2"
  !define DOT_MINOR "0"
  
  ; The text to prompt the user to enter a directory
  DirText "Please select a directory for the installation:"
  
  AutoCloseWindow true
  

  
  
• .onInit: This method is executed before the installer is run and can perform any checks you like. I just try to find out if any c:\games directory exists in several languages, if it does, then we can use it, else just use the default c:\program files location.
  

  
  Function .onInit
      ;First try to find "Games" (eng), "Spiele" (ger), "jeux" (fr),
      ; "giochi" (ita), "jogos" (port) or "spelen" (nl) directory!
      StrCpy $R1 "C:\Games"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\Spiele"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\jeux"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\giochi"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\jogos"
      IfFileExists $R1 UseIt
      StrCpy $R1 "C:\spelen"
      IfFileExists $R1 UseIt
      Goto DontUseIt
  UseIt:
      StrCpy $INSTDIR "$R1\XnaSkinningColladaModels"
  DontUseIt:
  FunctionEnd
  

  
  
• File section: This is the most important section and always required because it tells the setup where the files come from (in our case from the Release sub directory) and where they should be installed (in #INSTDIR).
  

  
  ; The stuff to install
  Section "XnaSkinningColladaModels"
    SectionIn RO
    ; Set output path to the installation directory.
    SetOutPath $INSTDIR
    ; Put file there
    File /r "Release\*.*"
   
    ; Write the installation path into the registry
    WriteRegStr HKLM "SOFTWARE\XnaSkinningColladaModels" "Install_Dir" "$INSTDIR"
    WriteRegStr HKLM "SOFTWARE\XnaSkinningColladaModels" "Version" "v1.0"
    ; Write the uninstall keys for Windows
    WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\XnaSkinningColladaModels" "DisplayName" "XnaSkinningColladaModels"
    WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\XnaSkinningColladaModels" "UninstallString" "$INSTDIR\uninstall.exe $INSTDIR"
    WriteUninstaller "uninstall.exe"
  SectionEnd
  

  
  
• Optional sections: These sections are optional and the user can decide not to create desktop shortcuts, not to install XNA 2.0 if it is not found, etc. I usually leave them all on by default because they make sense. Most important here are the DirectX and XNA prequisites sections, which make a lot of use of the helper functions.
  

  
  ; Optional sections
  Section "Create Start-Menu shortcuts"
    StrCpy $R1 "$SMPROGRAMS\XnaSkinningColladaModels"
    CreateDirectory "$R1"
    CreateShortCut "$R1\XnaSkinningColladaModels.lnk" "$INSTDIR\XnaSkinningColladaModels.exe" "" "$INSTDIR\XnaSkinningColladaModels.exe" 0 "" "" "XnaSkinningColladaModels"
    CreateShortCut "$R1\XnaSkinningColladaModels Uninstall.lnk" "$INSTDIR\uninstall.exe" "" "$INSTDIR\uninstall.exe" 0 "" "" "Uninstall XnaSkinningColladaModels."
    CreateShortCut "$R1\BenjaminNitschke.com.lnk" "http://BenjaminNitschke.com" "" "" 0 "" "" "BenjaminNitschke.com website from the creator of this game"
  SectionEnd
  
  Section "Create Desktop shortcuts"
    CreateShortCut "$DESKTOP\XnaSkinningColladaModels.lnk" "$INSTDIR\XnaSkinningColladaModels.exe" "" "$INSTDIR\XnaSkinningColladaModels.exe" 0
  SectionEnd
  
  Section "Check for .NET 2.0 Framework (required)"
    Call IsDotNETInstalled
    Pop $0
    StrCmp $0 1 found.NETFramework no.NETFramework
  no.NETFramework:
  ;MessageBox MB_OK ".NET was not found .."
    ; First of all check for correct IE version!
    Call CheckForCorrectIEVersion
    ; First check if DotNetFx.exe exists
    IfFileExists "$EXEDIR\DotNetFx.exe" 0 DotNetFileNotFound
    ReadRegStr $R0 HKLM "SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" "AutoAdminLogon"
    ExecWait '"$EXEDIR\DotNetFx.exe" /q'
    StrCmp $R0 1 RestoreAutoLogon End
  RestoreAutoLogon:
    WriteRegStr HKLM "SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" "AutoAdminLogon" "1"
    goto End
  
    ; If DotNetFx.exe was not found, try downloading online
  DotNetFileNotFound:
    MessageBox MB_YESNOCANCEL|MB_ICONQUESTION ".NET Framework 2.0 was not found. XnaSkinningColladaModels will not run without it! $\r$\nWould you like to automatically download and install it? $\r$\n $\r$\nChoose Yes for an automatic download and installation. $\r$\nChoose No, you want to select the download yourself (e.g. locally if already downloaded). $\r$\nMicrosoft's .NET Framework Download page will open up! $\r$\nChoose cancel to abort any action here, you will have to install the .NET Framework (2.0) $\r$\nyourself. That can be done for example automatically by the Windows Update $\r$\n(just go to http://windowsupdate.microsoft.com )" IDYES DL IDNO IndirectDL
    goto End
    IndirectDL:
      ExecShell open "http://www.microsoft.com/downloads/details.aspx?FamilyID=0856eacb-4362-4b0d-8edd-aab15c5e04f5" "" SW_SHOWNORMAL
      goto End
    DL:
      Call ConnectInternet ;Make an internet connection (if no connection available)
  
      StrCpy $2 "$TEMP\DotNetFX_Setup.exe"
      NSISdl::download http://