The last few days I was working a little bit with the DLR and got to a point today where I was wondering about the performance of a very simple while loop in the DLR, specifically in the ToyScript DLR sample project and of course IronPython. I am totally aware that the following comparison DOES NOT really provide accurate information about the actual performance of the languages and techniques used. Please keep this in mind when reading this article, it only covers a very certain aspect of all used languages. More specially how they perform when adding numbers in a while loop a lot of times! Some of it reflects that compiled languages are faster than dynamic languages, but you should not compare the absolute values, Python or Lua is not really 100 times slower, in many cases you can reach almost similar performance as long as you implement or call performance-critical code in a clever way (e.g. calling C++ code, using frameworks, etc.).

I was only interested about very ruff comparisons to figure out which ways are fast and which languages or techniques are kinda slow (click image on the right for a quick overview). Some code written in Assembler or direct IL code execution was really fast (duh, no wonder), but all execution times of C++, C#, Assembler or IL are way faster than all other dynamic languages or DLR approaches. The important thing for me was to figure out why dynamic languages like IronPython or ToyScript on top of the DLR were like 100 times slower for this specific code. At the end of writing this article I was able to provide pretty good performance with the DLR, which is just 2-3 times slower than the fastest execution time. That's pretty good and could probably be even improved more.

Basically I used the following code sample, which adds 1.0 to counter 10 million times while decreasing the loops variable until it reaches 0. The following code block is not valid syntax in any of the presented languages, but it is very similar to Python. In languages that use compile time types, counter would be a declared as a double (64bit floating point number), most dynamic languages use double as the run-time type for floating point numbers also.

loops = 10000000
counter = 0.0
while loops > 0
  loops = loops - 1
  counter = counter + 1.0

print counter

Languages and techniques used and tested (all with detailed code examples and explanations):

• C# 3.0
• Native C++
• Assembler
• IronPython 2.6
• ToyScript
• Python 2.6
• Python 3.1
• Lua 5.1.4
• Script.NET
• IL Emit
• Irony + DLR (dummy)
• DLR AST with objects
• DLR AST with doubles

Before boring you with all the different implementation details, here is the fancy Code Generation Performance comparison. This article became very long; I just wanted to post a few sentences about the DLR and this is what happened. The following graphs were created with Excel with some help from my good old friend ViperDK. The second graph (the first one was already shown above) shows just the execution time of the while-loop code block in each of the used language or technique.


And this graph also shows the other times like compile & build time where it makes sense or script or DLR start-up times to see how much this could affect the overall time from coding to seeing the result on the screen. This is important for my approach to my own language since I want a extremely low time for the coding to see result loop. Start-up time can be ignored in those cases, but it is still important since it is still kinda slow with the DLR (but will hopefully get better in the future).



Here are the values used to generate those graphs. Tested on my almost 3 years old 2.4 Ghz 6600 CPU; this is probably way faster on my i7@3.6 Ghz at work:

Language or Technique	Code lines	Compile&Build	Script Startup	Parsing&AST	Execution
C# 3.0	10	~500ms	-	-	~37ms
Native C++	8	1500ms	-	-	~31ms
Assembler	15+	~1100ms	-	-	~15ms
IronPython 2.6	6	-	~300ms DLR	160ms	~9000ms (~1250ms for files)
ToyScript	8	-	~430ms DLR	103ms	~8400ms
Python 2.6	6	-	~50ms	~100ms	~3500ms
Python 3.1	6	-	~50ms	~100ms	~4500ms
Lua 5.1.4	7	-	~25ms	~150ms	~1400ms
Script.NET	8	-	~100ms	-	~39752ms
IL Emit	30+	-	~100ms	~1ms	~16ms
Irony + DLR (dummy)	30 + 6	-	265ms Irony	87ms	~138ms (dummy)
DLR AST with objects	68+	-	433ms DLR	4ms	~459ms
DLR AST with doubles	60+	-	434ms DLR	5ms	~98ms

Implementing and testing the following code was not too bad for me since I was learning how to generate AST statements in the DLR, generating System.Reflection.Emit IL code, doing stuff with the ToyScript DLR sample, etc. anyway. I was of course interested in how to get the best performance for my code generation, but it is useful to understand why each approach is slower or faster. Sorry if the comparison presents your project or technology you might use in a bad light, but for example Script.NET is slow in comparison (this is explained below in more detail), but still fast enough for probably 99% of its uses. It still took much longer than I expected, writing this article did take some time too, but most of the time was spend writing and testing code in each language and technique and then gathering the execution time results. But performance evaluation is always fun :) Again: This is not a representative performance comparison of the languages and techniques involved, it only applies to my very short and stupid test code and should only give you a general idea.

C# 3.0

First of all the very simple implementation of the while loop:

public static double DoLoop(int loops)
{
    double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0;
    }
    return counter;
} // DoLoop(loops)

And now some code to measure the execution time and to print out the result to see if the loop was executed as many times as we expected:

    long startTime = WindowsHelper.GetPerformanceCounter();
    
    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    // Note: We did printing in other tests too (does not matter anyway)!
    Console.WriteLine("DoLoop result=" + result);
    
    long endTime = WindowsHelper.GetPerformanceCounter();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + WindowsHelper.ConvertToTime(endTime - startTime));

Note that the WindowsHelper class is just providing a little bit more accurate information than just using a StopWatch (which is used in most other samples here), which is fine too. With System.Diagnostics.StopWatch this code looks like the following (actually results in the same results on Windows 7 and Vista):

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Do simple performance test in C#
    const int NumberOfLoops = 10000000;
    double result = DoLoop(NumberOfLoops);
    //Note: We did printing in other tests too (does not matter much anyway)!
    Console.WriteLine("DoLoop result=" + result);

    timer.Stop();
    Console.WriteLine("Executing the while loop " + NumberOfLoops +
        " times took: " + timer.ElapsedMilliseconds + "ms");

The following output is generated:

DoLoop result=10000000
Executing the while loop 10000000 times took: 37ms

Native C++

Here is the whole program used for testing. This has nothing to do with the DLR or .NET anymore, I just wanted to know how my little while loop would perform in native C++ and Assembler. Sometimes when using for loops the C++ compiler will optimize the code away and just replace it with something static that results in the same value as if the for loop would have been executed. Please also note that it did not make any difference to use a 64bit platform and doubles or just 32bit floats, but since most script languages and especially .NET compile to x64 too, we should not use 64bit numbers when compiling a 32bit application here in C++. Switching to x64 is also not too hard, you just need the correct 64bit libraries and set the linker target platform to X64.

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    // C++ version
    int loops = 10000000;
    float counter = 0.0f;
    //64bit code: double counter = 0.0;
    while (loops > 0)
    {
        loops = loops - 1;
        counter = counter + 1.0f;
    }
    
    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    // Wait for user input (not required for Debug.StartWithoutDebugging)
    //char input[128];
    //scanf(input);

    return 0;
}

And after compiling (usually a lot faster than 1s when just incrementally compiling) and running this code, it generates this output:

counter=10000000.000000
This took 31ms

The result is not very accurate, I either get 31ms or 47ms, but not values in between, the accuracy of GetTickCount pretty much sucks and I should use similar code as the GetPerformanceCounter() stuff above from C#, but I do not really care about more accurate values here, C++ is more than fast enough and we do not need to discuss its runtime performance!

Assembler

Based on the C++ solution I was able to write some Assembler instructions. It has been a very long time since I used assembler code at all, probably in 2002 when optimizing some path finding algorithms for ArenaWars, which were called from C++ code, which was called from C# code. Before that in the C++ days I used it now and then, but never for more than some performance critical inner loops. Please also note that sometimes you make non-optimal assembler code (as I probably did in this example) and just having your C++ compiler generate assembler instructions can be better, so always profile your code and only optimize to Assembler if you have no other option left and if you are too lazy to rethink the problem :D

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();
    
    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
startLoop:
        // counter += 1.0f
        fld counter
        fadd counterInc
        fstp counter

        // loops--
        dec loops

        // loops > 0? 
        cmp loops, 0
        // Then goto startLoop (else we are done)
        jg startLoop
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

I don't have much to say about this ugly assembler code, as you can see it is a lot more lines of code and this was the first idea not even using registers as you should in assembler. After running this code and getting 31-47ms too (like the C++ version), I added some minor optimization by using registers more explicitly and this is the resulting code, it ran about 2-3 faster than the above code:

#include <stdio.h>
#include <windows.h>

int main(int argc, char* argv[])
{
    long int before = GetTickCount();

    int loops = 10000000;
    float counter = 0;
    float counterInc = 1.0f;
    __asm
    {
        // Copy loop counter to eax register (32bit)
        mov eax, loops
        // Load both float values
        fld counterInc
        fld counter
        
startLoop:
        // counter += 1.0f
        fadd st(0), st(1)
        //st(0) has still the counter value! no need to store: fst st(0)

        // loops--
        dec eax

        // loops > 0? 
        cmp eax, 0
        // Then goto startLoop (else we are done)
        jg startLoop

        // We are done, copy counter value back!
        fstp counter
    }

    printf("counter=%f\n", counter);

    long int after = GetTickCount();
    printf("This took %dms\n", (after-before));

    return 0;
}

This program does produce the following output:

counter=10000000.000000
This took 15ms

This is pretty good and I did not want to optimize it further, especially not when the timer is so inaccurate. Running the code 10 times resulted in 140ms. Again, fast enough, no reason to argue the superior runtime performance. But this comes at a high price since the code is in no way easy to understand, its hard to write and read.

IronPython 2.6

Speaking of code, I would say the implementation in Python is the shortest and very beautiful since it is so easy to read and write:

loops = 10000000
counter = 0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

I just used my ScriptManager from my .NET libraries, which can handle IronPython (and other DLR languages) as well as Lua and wrote this short unit test to see how IronPython performs:

    public void TestPythonDoLoop()
    {
        // Create script manager without any output redirection or special directories
        ScriptManager manager = new ScriptManager(null, null);
        
        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Simply execute line by line as we would in a console window
        manager.ExecuteCode("loops = 10000000", ScriptType.Python);
        manager.ExecuteCode("counter = 0", ScriptType.Python);
        manager.ExecuteCode(@"while loops > 0:
loops = loops - 1
counter = counter + 1", ScriptType.Python);
        manager.ExecuteCode("print counter", ScriptType.Python);

        timer.Stop();
        Console.WriteLine(
            "Executing the while loop 10000000 times in python took: " +
            timer.ElapsedMilliseconds + "ms");
    } // TestPythonDoLoop()

And this results in the following output:

Executing the while loop 10000000 times in python took: 9151ms

Update 2009-04-21: As Simon Davy correctly states in the comments of this post, ipy.exe (IronPython) is actually a lot faster when executing a file with the code above (e.g. use import time). This results in an execution time of just 1250ms, so a lot faster! I have tested it too with ipy, but I entered the code line by line (same way as I called my ScriptManager code in the unit test above) and it still takes 9 seconds. But Simon is right, once I execute everything at once (with ipy test.py) it only takes 1.25s (can also be done with my ScriptManager, but I have not thought of that). So it is probably just an issue with REPL (entering statements one by one .. less optimizations that way). While this certainly does make IronPython a much faster dynamic language than all the other ones, it still is slower than the compiled languages or the DLR code at the end of this article.

Thanks for noticing!This is certainly not good. The same thing happend after I tried using the IronPython 2.6 console or IronPython 2.0, it always took about 9 seconds to complete the big while loop on my PC. I wanted to figure out why this is so slow compared to C#, C++ or Assembler runtime, but then again we have to remember that Python is a dynamic language. The most important reason for the long runtime here is probably just all the boxing and unboxing of values, if you can say that for dynamic languages. What I mean is figuring out which type we got, and then performing the add/sub operation, which can be slow if you do it 10 million times. Please note again that this is probably not affecting any real Python program and performance for more complex programs will be way better. More on the type-performance and this issue in general at the end of this article!

ToyScript

To learn more about the DLR I played around with the DLR ToyScript sample quite a bit the last days. Since a lot of its code is based on the same ideas as IronPython and IronRuby, you can't expect much different performance results. This is the ToyScript code for the while loop:

loops = 10000000
counter = 0
while (loops > 0)
{
    loops = loops - 1
    counter = counter + 1
}
print counter

Similar to the ScriptManager above I wrote some helper classes for the ToyScript language allowing me just to execute code, viewing the AST (abstract syntax tree) and write a lot of unit tests. The following code is such a unit test for the ToyScript class to check the performance for these 10 million while loop iterations:

    public void TestBigLoop()
    {
        const int NumberOfLoops = 10000000;

        Stopwatch timer = new Stopwatch();
        timer.Start();

        // Initialize the language first (takes its own time, around 400ms)
        ToyLanguage language = new ToyLanguage();
        // Execute something else (takes maybe 100ms)
        Assert.Equal(3.0, language.Execute("print 1+2"));

        timer.Stop();
        Console.WriteLine("Startup time: " + timer.ElapsedMilliseconds + "ms");
        timer.Reset();
        timer.Start();

        Assert.Equal((double)NumberOfLoops,
            language.Execute(
@"loops = " + NumberOfLoops + @"
counter = 0
while (loops > 0)
{
loops = loops - 1
counter = counter + 1
}
print counter
"));

        timer.Stop();
        // Show total execution time
        Console.WriteLine("Executing the while loop " + NumberOfLoops +
            " times took: " + timer.ElapsedMilliseconds+"ms");
    } // TestBigLoop()

This code runs for about the same time as IronPython, maybe a little faster since ToyScript has a lot less features than IronPython, but in general it uses the same approach. This is the output from the unit test:

Startup time: 650ms
Executing the while loop 10000000 times took: 8914ms

Investigating the Lambda Expression AST for the code block shows that quite a lot of statements were created in order to make this ToyScript code run on the DLR:

 // Expression: Expression`1
 
 .lambda (<toyblock>$2 Microsoft.Func`1[System.Object] #1)
 
 // LambdaExpression: <toyblock>$2(1)
 .lambda System.Object <toyblock>$2 ()(
 ) {
   .label 0x03d11d12:
   .comma  {
     .block  {
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)10000000)
       ,
       .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
       (Object)0)
       ,
       .loop break:.label 0x03fe2d21  {
         .block  {
           .if (
             .site (Boolean) CallSiteBinder(ConvertTo to System.Boolean) (
               .extension CodeContextExpression ( )
               ,
               .site (Object) CallSiteBinder(DoOperation GreaterThan) (
                 .extension CodeContextExpression ( )
                 ,
                 .extension GlobalVariableExpression ( )
                 ,
                 0
               )
             )
           ) {
             /*empty*/
           } .else {
             .block  {
               /*empty*/,
               .break .label 0x03fe2d21
             }
           }
           ,
           .block  {
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             .site (Object) CallSiteBinder(DoOperation Subtract) (
               .extension CodeContextExpression ( )
               ,
               .extension GlobalVariableExpression ( )
               ,
               1
             ))
             ,
             .extension AssignmentExtensionExpression ( .extension GlobalVariableExpression ( )
             'ToyHelpers.Add'(
               (Object).extension GlobalVariableExpression ( )
               ,
               (Object)1
             ))
             ,
             /*empty*/
           },
           /*empty*/,
           /*empty*/
         }
       },
       'ToyHelpers.Print'(
         .extension CodeContextExpression ( )
         ,
         .extension GlobalVariableExpression ( )
       ),
       /*empty*/
     },
     .null
   }
 }

The IL code was even more confusing so I only tried to figure out the simple "print 1+2" expression IL code that can be produced by calling ScriptCode.SaveToAssembly(..), more DLR IL code analysis will happen at the end of this article:

  .method public specialname static object <toyblock>$1$1(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope $scope,
 class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext $language) cil managed
  {
    .maxstack 3
    .locals init (
      [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context)
    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
    L_0007: stloc.0 
    L_0008: ldloc.0 
    L_0009: ldc.r8 1
    L_0012: box float64
    L_0017: ldc.r8 2
    L_0020: box float64
    L_0025: call object ToyScript.ToyHelpers::Add(object, object)
    L_002a: call void ToyScript.ToyHelpers::Print(class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext, object)
    L_002f: ldnull 
    L_0030: ret 
  }

There is a lot of extra stuff in there, but you can clearly see the 1+2 in L_0009 to L_00020 with all its boxing fun. Then those two are added with help of a method Add that is defined in the ToyHelpers class (unboxing the doubles again, adding and boxing them to an object again, which is returned). Then the result is passed along to the ToyHelpers.Print method, which just calls Console.WriteLine in my case (usually it is used by the ConsoleHost class for the normal REPL console interaction).

I am not a DLR expert and there is probably a lot of other things going on, but to me all this boxing, unboxing and extra code just to add some numbers is clearly too much for the big loop. At the end of this article I will try to write some DLR code that does less boxing/unboxing.

Python 2.6

After testing the DLR it was time to check out the native Python implementations to see if this is releated to the dynamic nature of dynamic languages like Python or Lua. Well it obviously is related, but more specifically I wanted to see if certain implementaions are maybe more clever or optimized.

The Python code is the same as above for IronPython:

loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
print counter

Executing this takes about 3.5 seconds, we could cheat a little and use the following code to execute this loop a little faster:

counter = 0.0
for loops in range(0, 10000000):
    counter = counter + 1.0
print counter

This runs about 2 times faster (less than 2 seconds) because Python is clever and optimizes for loops. The same actually applies to IronPython, a for loop is much faster here too (5.5s instead of 9.1s). But this defeats the whole point of this exercise, I wanted to know what is slow about the DLR and implementations like IronPython or ToyScript and why. It is nice that Python executes for loops quicker, but the whole counter = counter + 1 thing is still very slow compared to C#, C++, etc.

Python 3.1

Executing Python 3.1 is pretty much the same as Python 2.6. It is slighly slower, but that can also be because I use a alpha build (3.1 alpha 2 from 2009-04-04). Again, executing the for-loop version with range was twice as fast (2.1s instead of 4.3s). In general is is quite known that Python is not one of the fastest languages, there are many dynamic languages that can be faster (for example Ruby or Lua), but there are also much slower ones like PHP. If you are more interested about comparing IronPython with Python read the performance comparisons at the IronPython project at codeplex: http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Performance

Lua 5.1.4

The lua code is pretty similar to Python, but it executes much faster:

loops = 10000000
counter = 0.0
while (loops > 0) do
    loops = loops-1
    counter = counter+1
end
print(counter)

This executes in less than 1.5 seconds and returns the expected 10000000 (not 10000000.0 as Python however since Lua treats everything as a 64bit double anyway). The for-loop trick works here too and reduces the time in half (0.7s instead of 1.4s):

counter = 0.0
for i=1,10000000 do
    counter = counter+1.0
end
print(counter)

The reason why Lua is much faster than all the other dynamic languages I have tested is probably because Lua has much simpler data types and is optimized a lot for math, for loops and stuff like that. I use Lua more than Python these days and the performance is quite good I have to say (using it for AI, some physics and some game logic). As you can see here in many benchmarks Lua is several times faster than Python (but Python is still a lot more powerful with the huge amount of libraries available for it).

Script.NET

Script.NET is a great project that uses the Irony compiler toolkit to parse and evaluate simple C# code. It does not generate any IL or immediate code, it just evaluates the AST directly. This is a very nice approach and easy to understand because all the code is C#, but there is a slight problem with it: This approach is it VERY slow. It was optimized many times already and runs much faster than a year ago, but I would not recommend it for anything other than some scripting in C# (guess what, that's the whole name of the project). If you do not need high performance Script.NET is just fine for executing a couple lines of code, even for games as long as you do not do any physics or rendering code with it ^^

loops = 10000000;
counter = 0;
while (loops > 0)
{
    loops = loops-1;
    counter = counter+1;
}
Console.WriteLine(counter);

The output appears after almost 40 seconds and displays correctly:

10000000

At least I now know why generating some IL code is very important, with or without the DLR. Next I will investigate IL code generation with Reflection.Emit, but I can still remember the pain from the last time I tried to do something with it.

IL Emit

So let's get right to it, the following code will generate the required IL for us:

    // Create a dynamic method called DoLoop where we put all our IL code
    DynamicMethod DoLoop = new DynamicMethod("DoLoop",
        typeof(double), null, typeof(string).Module);

    // Start a timer once again to measure creation time and execution time!
    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Get an ILGenerator and emit a body for our dynamic method
    ILGenerator il = DoLoop.GetILGenerator();
    
    // Create counter (double)
    LocalBuilder counter = il.DeclareLocal(typeof(double));
    il.Emit(OpCodes.Ldc_R8, 0.0);
    il.Emit(OpCodes.Stloc, counter);
    // Assign loops variable to 10000000 (10 million) for the loop!
    LocalBuilder loops = il.DeclareLocal(typeof(int));
    il.Emit(OpCodes.Ldc_I4, 10000000);
    il.Emit(OpCodes.Stloc, loops);
    
    // Start loop label
    Label startLabel = il.DefineLabel();
    il.MarkLabel(startLabel);
    
    // Do this 10000000 times:    
    // Increase counter by 1.0
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ldc_R8, 1.0);
    il.Emit(OpCodes.Add);
    il.Emit(OpCodes.Stloc, counter);

    // Decrease loops by 1
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 1);
    il.Emit(OpCodes.Sub);
    il.Emit(OpCodes.Stloc, loops);

    // Check if we still have to loop (as long as loops > 0)
    il.Emit(OpCodes.Ldloc, loops);
    il.Emit(OpCodes.Ldc_I4, 0);
    il.Emit(OpCodes.Bgt, startLabel);

    // End loop and return counter!
    il.Emit(OpCodes.Ldloc, counter);
    il.Emit(OpCodes.Ret);

    // Create a delegate that represents the dynamic method. This
    // action completes the method, and any further attempts to
    // change the method will cause an exception.
    DoLoopDelegate loop = (DoLoopDelegate)
        DoLoop.CreateDelegate(typeof(DoLoopDelegate));

    timer.Stop();
    Console.WriteLine("DoLoop IL creation time: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();

This creates the dynamic DoLoop method, which we can call with loop() from here on, which will also return the result (counter) once we call it. This all takes almost no time at all:

DoLoop IL creation time: 0ms

Next, we need to invoke this dynamic method, which is rather easy:

    timer.Start();
    Console.WriteLine("Invoking DoLoop() returned: " + loop());
    timer.Stop();
    Console.WriteLine("DoLoop IL execution time: " + timer.ElapsedMilliseconds + "ms");

And this only takes about 16ms and also returns the correct value:

Invoking DoLoop() returned: 10000000
DoLoop IL execution time: 16ms

So IL code is rather fast (faster than everything else here except the fine-tuned assembler, which is similar to this), but don't ask in how many problems I ran into just writing these few lines of code. Let's just say a lot of times I used wrong opcodes, did not know how to implement stuff and finally when stuff compiled I got syntax errors when executing, not really telling me what I did wrong, so I had to comment out code until it worked again and then slowly added code until I could figure out what had to be changed. Let's just say not a pleasant thing to write a compiler this way.

Irony + DLR (dummy)

As mentioned before Script.NET actually uses the Irony compiler toolkit for parsing and building the AST (abstract syntax tree). Irony itself is rather fast (can parse 15000 lines of code or more per second), only the execution part of Script.NET wasn't so good because no code is generated, everything is evaluated on the fly.

While I have started using Irony just a week ago after playing around with ANTLR first (see earlier blog post here), but I still like it very much and I hope Roman Ivantsov (the guy behind Irony) will soon release a long promised update with some cool new features I can sink my teeth into. As you will see in my sample below I have added some own functionality to Irony already to make it easier to test code and I will probably extend it even more when I write more of my language grammar.

If you don't know how Irony works and you are interested in building compiler grammars directly in C# you should check out the project and the Lang.NET talks Roman did this and last year about it, plus the great Irony article on CodeProject of course. Basically you define some non-terminals (variables, numbers) and terminals (logical blocks of your code) in the constructor of a Irony.Grammar derived class and then define some rules for it (in BNF = Backus-Naur Form). Then you set the root node and you are ready to parse some code. Please note that this grammar is very simple and will only handle this sample code and very similar code blocks. I'm still just playing around with grammars, there is no concrete implementation of my language yet, just a lot of ideas .. Writing some grammars is a good training, I have written one in ANTLR and 4 other more complex ones with Irony yet.

    public TestDoLoopGrammar()
    {
        // 1. Terminals
        var variable = new IdentifierTerminal("variable");
        var number = new NumberLiteral("number");

        // 2. Non-terminals
        var program = new NonTerminal("program");
        //var commandList = new NonTerminal("commandList");
        var command = new NonTerminal("command");
        var assignmentOperator = new NonTerminal("assignmentOperator");
        var assignment = new NonTerminal("assignment");
        var whileLoop = new NonTerminal("whileLoop");
        var compareOperator = new NonTerminal("compareOperator");
        var condition = new NonTerminal("condition");
        var binaryOperator = new NonTerminal("binaryOperator");
        var operation = new NonTerminal("operation");

        // 3. BNF rules
        // Please note that this program grammar is rather limited and not really
        // useful, it is only used for testing and learning Irony ..
        // We can have many commands in case we want to go crazy
        program.Rule = MakePlusRule(program, null, command);
        assignmentOperator.Rule = Symbol("=");
        assignment.Rule = variable + assignmentOperator + number;
        whileLoop.Rule = Symbol("while") + condition + ":" +
            program + "end";
        // We only allow some simple compare conditions
        compareOperator.Rule = Symbol("<") | "==" | ">";
        condition.Rule = variable + compareOperator + number;
        // We can either use + or -
        binaryOperator.Rule = Symbol("+") | "-";
        operation.Rule =
            variable + assignmentOperator + variable + binaryOperator + number;
        // Commands can be assignments, while loops or operations
        command.Rule = assignment | whileLoop | operation;

        // 4. Operators precedence and punctuation
        RegisterPunctuation(":", "end");

        // 5. Global settings
        this.CaseSensitive = false;
        this.ThrowGrammarExceptionsOnError = true;

        // 6. Set grammar root
        this.Root = program;
    } // TestDoLoopGrammar()

Note that ThrowGrammarExceptionsOnError is not a property of Irony, I added it to make sure all Grammar errors immediatly cause an exception to be thrown, which makes it much easier for unit tests and figuring out what when wrong. Not really sure why Irony does not do that, I can understand not throwing exceptions for syntax errors when parsing some source code (because that might be slower than just reporting errors in a list form and more helpful because its easier to continue after the error). But grammar errors mean usually that you cannot use the grammar for anything anyway. Some things also just produce a warning, but I throw Exceptions then too because I don't want hidden fixes of my grammar, which would make it hard to fix real problems later.

Anyway, with the TestDoLoopGrammar class we can now test and parse some code. Irony usually expects you to use its Grammar Explorer program to do this job, which is a pretty nice application, but it is not unit test and we all know I'm all about unit testing (don't we?). I still use the Irony Grammar Explorer from time to time and I recommend it very much if you want to test out grammars and visualize the AST a little better than just throwing some text into a console window. The following simple unit test will parse our while-loop program source code and produce a nice AST for us, which it kindly prints out:

    public void GenerateDoLoopAST()
    {
        AstNode program = IronyHelpers.ParseWithErrorHandling(
            new TestDoLoopGrammar(),
@"loops = 10000000
counter = 0.0
while loops > 0:
    loops = loops - 1
    counter = counter + 1.0
end");

        IronyHelpers.PrintAst(program);
    }

This will quickly print out the AST in the following way, which might seem a little complex and verbose at first, but you have to remember that we do not have to generate code for every single node. Often several nodes together will just be one single instruction. Other nodes like program are just used to hold code blocks together and allow several child nodes.

 program
   assignment
     loops [variable]
     assignmentOperator
       = [Symbol]
     10000000 [number]
   assignment
     counter [variable]
     assignmentOperator
       = [Symbol]
     0 [number]
   whileLoop
     while [variable]
     condition
       loops [variable]
       compareOperator
         > [Symbol]
       0 [number]
     program
       operation
         loops [variable]
         assignmentOperator
           = [Symbol]
         loops [variable]
         binaryOperator
           - [Symbol]
         1 [number]
       operation
         counter [variable]
         assignmentOperator
           = [Symbol]
         counter [variable]
         binaryOperator
           + [Symbol]
         1 [number]
     end [variable]

Now this tree has to be translated to some code. We could use Reflection.Emit for this, but we just saw how much work it was for this very simple sample. Thats the reason I'm using the DLR instead, which also gives my language some cool dynamic capabilities for free (plus the other benefits like interacting with other DLR languages). But since the DLR is not a piece of cake and I'm not done exploring it yet (just started last two week with it, gimme more time), I will not present a full project with this language here yet (totally out of the scope of this article). I used a modified version of ToyScript and some other sample projects (following next) to generate some DLR code here, but it is still hacked up very much. I will probably write another long article when I'm done with that and have some cool results to show.

DLR AST with objects

Okay, so let's dig deeper into the DLR. While I got some parts of the above Irony AST working in the DLR with help of the ToyScript sample, the whole DLR framework, the ToyScript sample and especially IronPython is still very hard to understand and its also kinda annoying that recent code drops do not work at all with ToyScript. They compile, but running ToyScript will just produce funny errors like "node not reducible", which is not very helpful if you just want to learn about the DLR. A lot of stuff in the DLR still constantly changes (just see the huge amount of code drops at dlr.codeplex.com). I totally like that the DLR team releases all this wonderful code and does it so often, but without any working documentation (almost everything you find on the internet is based on DLR versions from a year ago, when stuff worked in a totally different way) and not even the included samples working all the time is kinda painful. So what does a programmer do in such a case? Yes, he writes tests and figures it out on his own. If there were not this many classes that all do almost nothing in functionality (what the hell, in my projects each class has many hundred lines of code, but in the DLR it seems like 50% of all classes are just 10-30 lines long) it would a much easier job. Debugging is also much harder because stack-traces become 2 miles long after just calling one of two actual functions, that do not just call other functions. Okay, enough ranting, let's just hope these things get better in the final 1.0 release since Microsoft is not allowing any contributions from the community to the DLR and I certainly do not want to write my own. It is pretty amazing what the DLR can do and how well it performs, it just has to become easier to use. Someone had to say it :)

So the bare minimum you need for the DLR to compile some AST for you is a LanguageContext and a DefaultBinder, define one class of each and derive it accordingly. Then define a constructor for your language context, set the binder and while you are at it add the default System (mscorlib) assembly to the script domain manager. Note this code uses DLR change set 21259 (from Apr 5 2009), but I could also use the more recent 22459 version (from today), but I reverted to make the ToyScript sample work again.

    public YourLanguageContext(ScriptDomainManager manager,
        IDictionary<string, object> options)
        : base(manager)
    {
        Binder = new YourBinder(manager);
        manager.LoadAssembly(typeof(string).Assembly);
    } // YourLanguageContext(manager, options)

Next you need to provide an implementation of ParseSourceCode to make the compiler happy and to actually allow your program to parse some source code in your language:

    protected override ScriptCode CompileSourceCode(SourceUnit sourceUnit,
        CompilerOptions options, ErrorSink errorSink)
    {
        // Create lambda expression
        LambdaBuilder program = AstUtils.Lambda(typeof(object), "YourLanguage");
        program.Body = Expression.Block(GenerateStatements(program, sourceUnit.GetCode()));
        LambdaExpression ast = program.MakeLambda();
    
        Expression<DlrMainCallTarget> globalAst =
            new GlobalLookupRewriter().RewriteLambda(ast);
    
        return new LegacyScriptCode(globalAst, sourceUnit);
} // ParseSourceCode(context)

The interesting thing here is how we build an DLR AST from statements (Expression.Block) with help of the LambdaBuilder. The statements do not exist yet, that's what the GenerateStatements method will be used for. We pass the source code to this method and let its do its magic (this is a reduced version, the actual unit test uses a more complex version with some extra code paths and statements to make sure we can initialize the DLR first and to generate code for the next technique too):

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(object), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Convert(
                    Expression.Constant(0.0), typeof(object))
            )
        );
        Expression loops = program.Variable(typeof(object), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Convert(
                    Expression.Constant((int)10000000), typeof(object))
            )
        );
        
        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    Expression.Convert(loops, typeof(int)),
                    Expression.Constant(0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Convert(
                            Expression.Add(
                                Expression.Convert(counter, typeof(double)),
                                Expression.Constant(1.0)
                            ), typeof(object))),
                    Expression.Assign(loops,
                        Expression.Convert(
                            Expression.Subtract(
                                Expression.Convert(loops, typeof(int)),
                                AstUtils.Constant(1)
                            ), typeof(object)))
                ),
                AstUtils.Empty()
            )
        );
        
        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(object) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

This is obviously not dynamic code, it just adds a bunch of expression statements to a list and returns it. Please also note that I tried to use object for counter and loops to demonstrate this in a similar way ToyScript or IronPython will generate code (thats how I figured out some of these expressions). But since .NET or the DLR does not know how to add two objects for example and does not even allow to assign a number to an object, we have to do a lot of casting, which is in essense just boxing and unboxing.

This code is then executed by a unit test, which does basically create a ScriptRuntime for us, makes sure it uses this language implementation and then calls engine.CreateScriptSourceFromString with the code to be executed. Please note that GenerateStatements completely ignores the source code, it will always generate the same statements right now.

    Stopwatch timer = new Stopwatch();
    timer.Start();

    // Create runtime
    ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
    Type provider = typeof(YourNamespace.YourLanguageContext);
    setup.LanguageSetups.Add(
        new LanguageSetup(provider.AssemblyQualifiedName, provider.Name));
    ScriptRuntime runtime = new ScriptRuntime(setup);

    // Load Engine
    ScriptEngine engine =
        runtime.GetEngineByTypeName(provider.AssemblyQualifiedName);
    timer.Stop();
    Console.WriteLine("Create DLR runtime: " + timer.ElapsedMilliseconds + "ms");
    timer.Reset();
    timer.Start();

    // Execute command
    ScriptSource code = engine.CreateScriptSourceFromString("Objects");
    code.Execute();

    timer.Stop();
    Console.WriteLine("Execute code: " + timer.ElapsedMilliseconds + "ms");

This test will spend about 430ms in the DLR runtime creation (which is kinda slow and will hopefully be faster in the future) and that is the reason it is checked separately. Then it will spend a few ms for the statement creation (4ms as mentioned in the table above) and finally it will execute the code in about 459ms, which is a lot faster than the other DLR languages. But then again, we cheated quite a bit by providing the exact statements we need. The only thing we can optimize is to get rid of all those boxing and unboxing actions. While it is nice to be under half a second for these 10 million while loop iterations with one add to counter and one subtract from loops statement each, it is still a far cry from the performance we have seen in the compiled languages. Let's check out the IL code generated from the DLR AST to see whats going on here:

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] object obj2,
    [2] object obj3)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: box float64
  L_0016: stloc.1 
  L_0017: ldc.i4 0x989680
  L_001c: box int32
  L_0021: stloc.2 
  L_0022: ldloc.2 
  L_0023: unbox.any int32
  L_0028: ldc.i4.0 
  L_0029: cgt 
  L_002b: brfalse L_0035
  L_0030: br L_003a
  L_0035: br L_0063
  L_003a: ldloc.1 
  L_003b: unbox.any float64
  L_0040: ldc.r8 1
  L_0049: add 
  L_004a: box float64
  L_004f: stloc.1 
  L_0050: ldloc.2 
  L_0051: unbox.any int32
  L_0056: ldc.i4.1 
  L_0057: sub 
  L_0058: box int32
  L_005d: stloc.2 
  L_005e: br L_0022
  L_0063: ldloc.1 
  L_0064: call void [mscorlib]System.Console::WriteLine(object)
  L_0069: ldnull 
  L_006a: ret 
}

It might be a little hard to figure out which line does what, but once you found the constants like 0x989680 (which is just 10000000) and the add and sub operations, you can get an idea how the statements were translated. The code between L_0035 and L_0063 is the inner loop and the performance critical code. There are 2 unboxing and 2 boxing operations in that code and probably some other unnecessary assignments.

DLR AST with doubles

While playing around with the code even more and getting to a version like the one that will be shown in this section, I tried to understand why the DLR did certain things in the way that it does. A lot of dynamic behavior is only possible when you use very generic types, but I still think for my language implementation it should be possible to use a much more strict rule set and only allow dynamic code where necessary or wanted, not in thinks like loops or math, where we deal with numbers anyway. Lua is a great example of that, numbers are just numbers. They are always doubles, so there is no need to box or unbox them all the time, which is probably one of the reasons it can handle math better than other dynamic languages.

I also found this useful explanation of what IL the DLR emits for dynamic code, it could explain why stuff is slower than IL code, but still a lot faster than evaluating everything ever time. The DLR does a great job of caching and optimizing dynamic code, but as you can see the IL generated is quite long and verbose too, which is ok for dynamic code I guess. I'm just interested in ways to optimize this further to at least provide proof that the DLR is in fact just as good as a compiler for more static languages.

After the last section it is quite obvious what we have to do do generate statements that can be executed quicker: Get rid of all that boxing and unboxing, use specific data types and make sure all the assignments use the correct format too (else convert constants or other variables). This should not be too hard to do in an actual language once we figure out that we are only dealing with numbers (similar to Lua we could only allow one kind of number, like doubles). This is the updated code for GenerateStatements:

    private List<Expression> GenerateStatements(LambdaBuilder program, string code)
    {
        List<Expression> statements = new List<Expression>();

        // Try simple while loop for performance testing!
        // counter = 0.0
        Expression counter = program.Variable(typeof(double), "counter");
        statements.Add(
            Expression.Assign(
                counter,
                Expression.Constant(0.0)
            )
        );
        Expression loops = program.Variable(typeof(double), "loops");
        statements.Add(
            Expression.Assign(
                loops,
                Expression.Constant(10000000.0)
            )
        );

        // while (loops > 0) { counter = counter + 1; loops = loops - 1; }
        statements.Add(
            AstUtils.While(
                Expression.GreaterThan(
                    loops,
                    Expression.Constant(0.0)
                ),
                Expression.Block(
                    Expression.Assign(counter,
                        Expression.Add(counter,
                            Expression.Constant(1.0))),
                    Expression.Assign(loops,
                        Expression.Subtract(loops,
                            AstUtils.Constant(1.0)))
                ),
                AstUtils.Empty()
            )
        );

        // Print out results of counter to the console!
        MethodInfo consoleWriteLine = typeof(Console).GetMethod("WriteLine",
            new Type[] { typeof(double) });
        statements.Add(
            Expression.Call(
                consoleWriteLine,
                counter
            )
        );

        return statements;
    } // GenerateStatements(program, code)

BTW: To analyze the IL code I use the ScriptCode.SaveToAssembly method and use Reflector to show the IL code. As you can see the code has become shorter and we can't see any signs of boxing and unboxing anymore. The code is not even optimal because we use doubles for everything (remember that the C#, C++ and Assembler samples all used a simple integer for the while loop), but as the execution time with less than 100ms shows us, the performance is quite good with this approach.

{
  .maxstack 5
  .locals init (
    [0] class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext context,
    [1] float64 num,
    [2] float64 num2)
  L_0000: ldarg.0 
  L_0001: ldarg.1 
  L_0002: call class [Microsoft.Scripting]Microsoft.Scripting.Runtime.CodeContext
[Microsoft.Scripting]Microsoft.Scripting.Runtime.ScriptingRuntimeHelpers::CreateTopLevelCodeContext(
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.Scope,
class [Microsoft.Scripting]Microsoft.Scripting.Runtime.LanguageContext)
  L_0007: stloc.0 
  L_0008: ldc.r8 0
  L_0011: stloc.1 
  L_0012: ldc.r8 10000000
  L_001b: stloc.2 
  L_001c: ldloc.2 
  L_001d: ldc.r8 0
  L_0026: cgt 
  L_0028: brfalse L_0032
  L_002d: br L_0037
  L_0032: br L_0054
  L_0037: ldloc.1 
  L_0038: ldc.r8 1
  L_0041: add 
  L_0042: stloc.1 
  L_0043: ldloc.2 
  L_0044: ldc.r8 1
  L_004d: sub 
  L_004e: stloc.2 
  L_004f: br L_001c
  L_0054: ldloc.1 
  L_0055: call void [mscorlib]System.Console::WriteLine(float64)
  L_005a: ldnull 
  L_005b: ret 
}

This is a more complex version of the above unit test from last section, the 10000000 is printed by the DoLoop DLR code:

runtime creation: 336ms
get engine: 16ms
create script: 4ms
Init code and first script startup time: 232ms
10000000
execute again (DoLoop code): 98ms
and shutdown: 0ms

Okay, I would say this blog post is long enough now and I wrote all day on it and the code samples presented here. Could even be the longest blog post on my blog (html is 150kb). Who knows .. it is probably a good idea to go to bed now.

Good fight, good night, till next time :)

In accordance to the very good blog with many useful links for IronPython at ironpython-urls.blogspot.com I will also try to post some useful links about my recent DLR (Dynamic Language Runtime for .NET) research.
I have been working a bit on the DLR before, mostly together with Silverlight, which was cool, but Silverlight was way too hard to work with and I still think it is not distributed enough. On that note if you are interested in Silverlight, check out the . And even before that quite a bit with the Visual Studio SDK, early IronPython versions and other language implementations in .NET and even with native c code (but usually I just modified existing samples). I have also modified Lua for my own needs recently and made it run on the Xbox 360 and PS3 and modified some behaviour for my projects, since we use it for our upcoming game and I use it quite a lot together with IronPython on the tools I write at work.`

But my overall goal remains: Writing my own language, preferbly running on .NET and the DLR seems to be a good fit. Today after we finished the latest demo of our game for the GDC next week, I had finally some time and motivation to get this thing started again :) I probably should do some more XNA projects since I'm a DirectX/XNA MVP and I have still something I work on from time to time, but it is still a long way to go until this project is completely done.

Okay, to get things started again I searched for some new DLR information and samples on the net and here are some helpful links:

The Tools:

• DLR - Dynamic Language Runtime, which runs on top of the CLR in .NET (even in Silverlight)
  
  
• ANTLR v3 - The best tool ever for constructing your own interpreter/compiler/parser, etc.
  And these are the most useful links to get started (and other links that interested me):
  
  • Five minute introduction to ANTLR 3
    
    
  • ANTLR 3 - FAQ - Getting Started
    
    
  • Test-Driven Development with ANTLR
    
    
  • Quick Starter on Parser Grammars - No Past Experience Required
    
    
  • How do I get case insensitivity?
    
    

Learning the basics and also some good blog posts:

• Create your own language on top of the DLR - Detailed article about creating a language with the DLR
  
  
• DLR - Build Your Own Language (without tears). Kinda fast paced article that skips a lot of steps, but at least it got me motivated.
  Part 2, Part 3, Extra Tips
  
  
• CLR Inside Out - IronPython and the Dynamic Language Runtime - Good article about the DLR, IronPython and DLR ASTs
  
  
• Generate AST for the DLR (Code Project). Nicely written, easy to understand and gives a good overview of writing your own custom AST for the DLR. The sample itself is kinda useless because it always generates the same AST, but you will at least understand how code should be parsed and passed to the DLR as an AST. In this tutorial there is no parser or lexer, the generated AST is just static code, but at least the source code is easy to understand :) The article will also explain all the basics you need to understand ASTs and how the DLR uses them.
  
  
• Building a DLR Language - Answering Some Questions - Also a very helpful blog. Goes into detail about whats going on under the hoods, investigates with Reflector. Be sure to also read the other posts of Martin Maly.
  
  
• How to write a simple DLR host in C# using Hosting API
  
  
• Building a DLR Language - ToyScript (Martin Maly's blog)
  
  
• Jim Hugunin's Thinking Dynamic blog - The One True Object articles are interesting, also links to a lot of good DLR talks
  
  
• DevHawk - Blog by Harry Pierson, one of the main guys behind IronPython
  
  
• Martin Maly - Building a DLR Language (more general, see above for more comments)
  
  
• Sapphire Steel blog: Antlr 3 and the Dynamic Language Runtime
  
  
• DLR Hosting and related stuff...
  
  
• Knowing .NET blog about IronPython and .NET
  
  
• The .Net DLR makes building a new language implementation ridiculously easy
  
  
• Building On The DLR - by Tomas Restrepo - An approach to create a simple language with the DLR
  
  
• I still have a problem with the DLR - by mithrandir (Incanus On Rails) - A critice on how the DLR beta was written. I think most of the issues have been solved or not as bad as portraied there.
  
  
• Darrell Hawley's blog, mostly about IronPython integration. Has some cool "Note to self" posts on there.
  
  
• Another critice about the DLR, this time about the fact that it is not really open source since no contributions are accepted. Little bit too harsch.
  
  
• The Weekly Source Code 11 - LOLCode DLR Edition (from 2007) - by Scott Hanselman's ComputerZen.com - fun read.
  
  
• Another post like this from a year ago with helpful DLR links: Adventures in Compilers - Building on the DLR
  
  
• Embedding the Dynamic Language Runtime - IronPython & .NET Applications - A nice tutorial by Michael J. Foord (at voidspace.org.uk)
  
  

Compiler Books:

• Basics of Compiler Design by Torben Ægidius Mogensen - Free eBook - Easy and enjoyable read about a hard topic!
  
  
• Compiler Construction by Niklaus Wirth - Also a free eBook (about Oberon)
  
  
• Compiler Design and Implementation - Free eBook about some guy writing his own compiler - step by step
  
  
• Compilers: Principles, Techniques, and Tools (2nd Edition) - A heavy 1000+ pages and kinda expensive book about a heavy topic, was written in 1986, but is still very useful and recommended
  
  
• The Definitive ANTLR Reference: Building Domain-Specific Languages by Terence Parr - This is the book you should read when using ANTLR seriously. Very nice book; I plan to read it when I use ANTLR more.
  
  
• Lots more Free Compiler Design / Construction Ebooks
  
  
• And even more Free Compiler Books (newer)
  
  

Other useful DLR links:

• Homepage of IronPython - The main usecase for DLR - quite a lot of releases, performance comparisons and a small, but active community
  
  
• DLR Pad, a very simple app that lets you write and test code with a simple XAML WPF app using IronPython and IronRuby
  
  
• IronPython and the Dynamic Language Runtime
  
  
• Make Your Application Extendable Using the DLR - simple example on how to use IronRuby or IronPython in a little app.
  
  
• Boo also uses ANTLR and is a very interesting project overall (mixture of C# and Python). I used it a bit 2 years ago, but I stopped because of the lack of a decent IDE (yes, I know about SharpDevelop, but that is not decent enough for me).
  
  
• IronPython (and DLR) nightly builds. In case you want to be on the bleeding edge.
  
  
• Silverlight Dynamic Languages SDK
  
  

Okay, back to improving my lexer and parser. I think I got the research aspect covered for now.

I have been working quite a bit in Python and Lua last year and I also tried to use those languages in our current game project at exDream (will be announced shortly, so I'm finally allowed to talk about it ^^). While both Python and Lua are written in C (there are obviously other ports like the great IronPython project for .NET, but I'm not going to talk about that in this post), it was not easy at all to make Python run on the Xbox 360 or PS3. I was able in early testing to get some parts running on the Xbox 360, but too much was missed and commented out and the PS3 port looked like way too much work. So I decided to just use Lua, which is probably a better choice for our game anyway since I only use it to do some AI and physics code with it along with some settings. I'm not going to benefit from the Python libraries either way, they are not needed for our game and they won't run on those consoles without serious effort anyway.

Ok, so I was writing some Lua code and used mostly LuaInterface to make the Lua code run from .NET and of course to work with all my .NET code and even XNA. The second step was to make this run in our new game engine, which is written in native code (C++) since the whole engine is based on the Unreal 3 Engine. For that I used the normal Lua 5.1.4 release, which compiles just fine and works great. Since Lua is written completely in ANSI C I was able to quickly test some code on the Xbox 360 and PS3 too. This is important because our game will be a multi-platform title (PC, Xbox 360 and PS3), which is thanks to the Unreal 3 Engine a much simpler task than writing our own engine for all those platforms (it has obviously it own disadvantages, not only do we have to use native code, but we also have to work with a lot of code in the engine that we don't really need for our game).

So far so good. While it sounds like a simple task to make Lua run on all platforms, there is a lot of pain involved to make it run perfectly and behave the same way on all platforms. Since the Unreal 3 Engine is written in C++ and uses its own scripting Language UnrealScript, it does not help out a lot when you want to integrate your own code or libraries in there, so this was half of the work (just see my previous post about compile times and you see why this is a much lengthier process than writing some .NET code). Secondly there are many special rules on both the Xbox 360 platform and the PlayStation 3, some ANSI C methods are just missing or just don't do anything because of the platform design. For example on the Xbox 360 you can't use relative paths, you always have to specify a full path when opening a file, which is something that the Lua engine does not do out of the box. After figuring out where to store the Lua scripts on the consoles and how to load them, everything else just falls into place and works finally as expected.

To make things simpler for other people trying to run Lua code on the Xbox 360 or PS3 I wrote the following little solution:

• LuaXbox360PS3PC.zip (241.92 KB)
  

As you can see on the right the solution consists of 4 projects:

• LuaSupport, which is based on the current Lua 5.1.4 release from 22 Aug 2008 with my own additions to support the XBOX and PS3 platforms (uses XBOX and PS3 defines).
  
  
• TestLuaPC is a simple test project for the PC to make sure everything still runs as expected on the PC. Please note that on the PC Lua assumes that all files can be found in the current directory. I just change to the Lua script directory and then execute the Lua files there normally.
  
  
• TestLuaPS3 will probably only compile if you have the PS3 SDK installed, just unload this project if you don't have it installed yet. Also please test other SDK samples first to make sure everything is correctly setup (it uses a lot of extra include paths). In case you are not able to open the files, check for yourself if the /app_home/Lua/ directory contains the required files after building the project (use the ProDG Target Manager for that). Debugging can also be complicated, I guess it does not work on Windows 7 at all, but I was able to debug and run the app on a normal Windows Vista PC. In any case you can also just run the .elf file from the ProDG Target Manager and then see the output in the console.
  
  
• TestLuaXbox360 is the project to test Lua on the Xbox 360. You will also need the XDK (Xbox Development Kit SDK) installed and have a Xbox 360 DevKit in your network for testing. A TestKit also works, but you won't be able to debug, you can just run the app and check the console output yourself.
  
  

Some of the Lua functionality is not available on the Xbox 360 or PS3 like lua_popen or lua_pclose (among others), and there is also some code that will not do anything like os_execute, os_getenv and some other os methods because they won't make sense on a console anyway or could not be implemented. I did not use these methods anyway, but if you need those and want to write your own code for them, go ahead.

Lets focus on the 2 most important code changes. First of all the Lua default directories had to be adjusted in luaconf.h:83 to support Xbox 360, PS3, PC and Linux platforms:

#if defined(XBOX)
/* On the Xbox we just use the D:\ drive for everything, all lua files should be
   located there */
#define LUA_LDIR    "D:\\Lua\\"
#define LUA_CDIR    "D:\\"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(PS3)
/* On the PS3 we just use the /app_home/Lua/ sub directory for everything,
   all Lua files should be located there */
#define LUA_LDIR    "/app_home/Lua/"
#define LUA_CDIR    "/app_home/"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(_WIN32)
/*
** In Windows, any exclamation mark ('!') in the path is replaced by the
** path of the directory of the executable file of the current process.
** Note: We use the ..\Lua\ directory by default, but that is also the current
** directory, we change it in AIEngineInterface before running scripts!
*/
#define LUA_LDIR    "!\\..\\Lua\\"
#define LUA_CDIR    "!\\"
#define LUA_PATH_DEFAULT  \
        ".\\?.lua;"  LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua;" \
        LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#else
/* Default linux/mac directories */
#define LUA_ROOT    "/usr/local/"
#define LUA_LDIR    LUA_ROOT "share/lua/5.1/"
#define LUA_CDIR    LUA_ROOT "lib/lua/5.1/"
#define LUA_PATH_DEFAULT  \
        "./?.lua;"  LUA_LDIR"?.lua;"  LUA_LDIR"?/init.lua;" \
                    LUA_CDIR"?.lua;"  LUA_CDIR"?/init.lua"
#define LUA_CPATH_DEFAULT \
    "./?.so;"  LUA_CDIR"?.so;" LUA_CDIR"loadall.so"
#endif

And the most important code change is located in luaL_loadfile method, which is called by the lua_dofile macro. Please note that I did not change the code formating and tried to leave as much code as is. I have my own coding conventions, but it does not make sense to apply them when only changing <10% of the code of an existing library.

LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  char fullFilename[128];
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  lf.extraline = 0;
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    // Note: Always open as the requested file, Lua should not care about
    // our crazy directory remapping.
    lua_pushfstring(L, "@%s", filename);

#if (PS3)
    // On the PS3, check always the /app_home/Lua/ directory!
    // This works both for dev-testing via remote HDD and on the game disc.
    // These work probably too just for developing, but app_home is fine!
    // /host_root/Lua/ or /dev_bdvd/PS3_GAME/USRDIR/Lua/
    CheckForValidFilenameWithPath(fullFilename, filename, "/app_home/Lua/");
#elif XBOX
    // For the Xbox we have to make sure always to load files from D:\ because
    // fopen ALWAYS expects a full paths, there are no current directories on the
    // Xbox 360 and therefore no relative paths! Check always "D:\Lua\<file>"
    CheckForValidFilenameWithPath(fullFilename, filename, "D:\\Lua\\");
#else
    // On the PC we just use the default search logic (see luaconf.h) and we
    // don't care about directories since we will already be in the correct one!
    // In earlier versions we had a lot of extra checks here.
    strcpy(fullFilename, filename);
#endif

    // Rest of this code is untouched, we just use fullFilename now!
    lf.f = fopen(fullFilename, "r");
    if (lf.f == NULL)
      return errfile(L, "open", fnameindex);
  }
  c = getc(lf.f);
  if (c == '#') {  /* Unix exec. file? */
    lf.extraline = 1;
    while ((c = getc(lf.f)) != EOF && c != '\n') ;  /* skip first line */
    if (c == '\n') c = getc(lf.f);
  }
  if (c == LUA_SIGNATURE[0] && fullFilename) {  /* binary file? */
    lf.f = freopen(fullFilename, "rb", lf.f);  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, "reopen", fnameindex);
    /* skip eventual `#!...' */
   while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;
    lf.extraline = 0;
  }
  ungetc(c, lf.f);
  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (filename) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, "read", fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}

The code in luaL_loadfile will call the new method CheckForValidFilenameWithPath, which helps us to build a full path filename for the Xbox 360 and PS3 platforms. It will just add D:\Lua\ or /app_home/Lua/ in front of the filename if it does not exist yet. Here we can see another pain point of C code, it takes so freaking long to do very simple tasks (in .NET this would be a 2-liner).

static void CheckForValidFilenameWithPath(char* fullFilename,
  const char* filename, const char* platformPath)
{
  int i = 0;
  int pathLen = (int)strlen(platformPath);
  // Already have a filename with the correct path?
  if ((int)strlen(filename) > pathLen)
    for (i = 0; i < pathLen; i++)
      if (filename[i] != platformPath[i])
        break;

  // Found path?
  if (i == pathLen)
  {
    // Then just use the existing file, happens when loading libraries from Lua
    strcpy(fullFilename, filename);
  }
  else
  {
    // Copy path
    for (i = 0; i < pathLen; i++)
      fullFilename[i] = platformPath[i];
    // Add the relative filename
    for (i = 0; i < (int)strlen(filename); i++)
      fullFilename[i+pathLen] = filename[i];
    // And finally close the string
    fullFilename[(int)strlen(filename)+pathLen] = 0;
  } // else
} // CheckForValidFilenameWithPath(fullFilename, filename, platformPath)

And that are the most important code changes to make everything run on the Xbox 360, PS3 and PC. Here is the main method for all the TestLua projects (always pretty much the same, only the Xbox 360 uses a slightly different way to print to the console):

int _tmain(int argc, _TCHAR* argv[])
{
    printf("Initialize Lua\n");

    Lua = lua_open();
    luaL_openlibs(Lua);
    lua_register(Lua, "print", lua_print);

    // Simple print test from Lua:
    luaL_dostring(Lua, "print(\"Hi from Lua\")");

    // Now load a Lua file, which also loads another Lua file (LuaUnit.lua)
    if (luaL_dofile(Lua, "SomeClass.lua") != 0)
    {
        printf("Error: Unable to execute Lua file %s: %s\n", "SomeClass.lua",
            lua_tostring(Lua, 1));
    } // if (luaL_dofile)

    // Stop the Lua engine and cleanup
    printf("Closing Lua\n");
    lua_close(Lua);
    Lua = NULL;

    // Keep command window for debug mode (ctrl+f5 does not have this issue)
    char input[100];
    gets_s(input);

    return 0;
} // _tmain

Summary:

• Using Lua on other platforms is easier than Python or other dynamic languages I have checked because Lua is written completely in ANSI
• Making code run on multiple platforms is still hard in C++ because you not only end up with ugly #define code, but you also have to test a lot
• Building native applications is freaking time-intensive, it took me almost a full day to write this app and test it on every platform from serval PCs. In .NET it would take me less than an hour, but then again there is no .NET on the PS3 and our current game engine is using native code anyway.
• On the Xbox 360 you always have to use full paths like D:\Lua\SomeClass.lua, relative paths won't work. The SDK says you should use Game:\, but since our game engine uses D:\ too, I left it that way in this sample too.
• The PS3 is even more complicated with lots more possible mounted paths, but it seems /app_home did always work in both my samples and our game, so I'm sticking with it :)

If you are interested in Lua from .NET, check out the LuaInterface project. Someone also started NUA (Lua on the DLR) 2 years ago, but it seems this project (and many other attempts) have died a long time ago. There was also a port by the Lua guys (especially the great Fabio Mascarenhas, who has also started the LuaInterface project and is still involved) called LuaCLR, but it is still in alpha stage and pretty useless, LuaInterface on the other hand is a fully working library! The difference here is that LuaCLR emits CLR code like any other .NET language, while LuaInterface still relys on the Lua runtime. LuaInterface uses only 2 very simple C++ files, the rest is in C#, but because of that you currently cannot run it with XNA on the Xbox 360. IronPython, while fully written in C#, does not work on the Xbox 360 either because reflection is not supported by the Xbox 360 .NET framework.

And finally here is a link to Lua running on the iPhone: Lua v5.1.3 for iPhone/iPod, kinda cool, but since we can run C# code on the iPhone with help of the great Unity3D engine, I'm happy with that at the moment. The main problem for other languages on the iPhone is the missing libraries anyway (can't use Cocoa or any own UI) ..

Just in case you have not read about it yet: John Carmack has ported the good old Wolfenstein 3D game from 1992 to the iPhone as well.

I have updated my CR_Commenter plugin quite a lot last weekend and tested it all week. CR_Commenter v1.9 new features include full hotkey and feature customization, much better support for C++, Lua, Python and way better light-commenting features and more powerful and stable region features. Thanks to the many people writing me emails using the addin or even improving it themselfs and sending me updated code back (full thanks are listed below).

Download links for CR_Commenter v1.9 (2008-11-27):

• Click here to download the installer: CR_Commenter.exe v1.9 (98 kB)
• And here is the sourcecode for: CR_Commenter_src.zip v1.9 (90 kB)

If you do not know about the CR_Commenter plugin for CodeRush yet (I originally wrote it back in 2004 and still use it every single day), please read the article for version 1.8 from earlier this year where I also added a lot of features and this article of version 1.7 from 2006 or check out the original Commenter - A CodeRush plugin which helps you to comment your code article on CodeProject.net from 2004, which I have finally updated too (its about time).

To use CR_Commenter you need to install CodeRush 2.0 or later, which is a commercial product you can get from DevExpress here. You can also use the free DXCore component with a reduced feature set, but all the basic CR_Commenter features should work fine there too. Please note that there is also a recently released free CodeRush Xpress Edition, which should work too, but I have not tested it.

If you have used CR_Commenter before the first thing you will probably notice is the completely new options screen:


Let me go through all the new and improved features. Over the years so many things were added to CR_Commenter and since the options screen, hotkeys and feature set has changed so much all the time, it is hard to keep track what features are available in the latest version. To make it easier for new users to understand what you can do with CR_Commenter v1.9 I decided to explain every feature, which might also be useful for existing users as they might have missed some of the cool features before. All these features can now be enabled or disabled and you can set your own hotkey for most of them (if it makes sense). I will go through the same list as seen in the Options screen, which is also represented in the FeatureHotkey.cs class that has a lot of comments and explanations for each feature. As mentioned before all Hotkeys can be changed to whatever you like. Please note that the pictures might look a little different than your Visual Studio code, I just use a different theme and bigger font sizes, it is still Visual Studio 2008.

CR_Commenter v1.9 Features:

• Automatically add comments after closing blocks with }
  Hotkey: }
  This is the most basic feature of CR_Commenter. In v1.9 you can now enable or disable it as you like. Please also note that you can now specify to only generate comments for blocks greater than a certain size (by default 5), which reduces the amount of comments generated greatly and is also the new default setting! Note: Since this is only parsed after pressing } it only works for C like languages like C, C++, C#, fx, UnrealScript, etc. For some of the newly supported languages like Lua (or VB, which is still untested and more or less unsupported sadly) or even Python that use identation for blocks you have to use Ctrl+1
  
  
  
  
• Force Comment Block
  Hotkey: Ctrl+1
  Same as }, but allows us to comment whole classes and methods easier by positioning the text cursor on what you want to update. Use Ctrl+6 for always updating the whole file no matter where you are located. Can also be used for languages not using } for commenting or if you have disabled } commenting.
  
  
  
  
• Add namespace for the current word. (Disabled by default)
  Hotkey: Ctrl+2
  Used to support automatically adding a namespace in C# 3.0/VS 2008. Not longer needed since VS supports it on its own now! This feature could be used back in 2006 to add namespaces in VS 2005.
  
  
  
  
• Autogenerate region around the current block
  Hotkey: Ctrl+3
  Autogenerate region around the current block (method, property, constructor, etc.), for selections use Ctrl+R (default CodeRush hotkey). The most important thing about this feature is that it will automatically give your new region the method name and it will make sure all comments belonging to the method are put inside the region. Note: In the picture the #region SetNewValue and #endregion parts are hidden by CodeRush with lines to make the source code look cleaner.
  
  
  
  
• Autogenerate regions for the whole file (all methods)!
  Hotkey: Shift+Ctrl+3
  Same as Ctrl+3, but will go through the whole file and generate region blocks for every method that does not seem to have one yet! You can also just go to the class level and press Ctrl+3 to generate all regions for all class methods. Since v1.9 also works with C++ and will generate #prama regions for you. BTW: Parsing the whole file with Ctrl+1 will also generate a includes region automatically for you.
  
  
  
  
• Toggle collapsing and expanding current selection
  Hotkey: Ctrl+4
  Very useful to avoid having to use the mouse for these things. Please note that Visual Studio also supports collapsing sections with Ctrl+M+M, but that will only work on sections, not on regions, which is a pretty useless feature in C#, Ctrl+4 is much better :) Since v1.9 also supports regions if CodeRush does not parse them (e.g. #pragma region in C++), so you can now even collapse C++ regions.
  
  
  
  
• Toggle collapsing and expanding everything at top level
  Hotkey: Ctrl+5
  Will expand or collapse all regions in the current file at once. First time pressed it expands everything, second time everything is collapsed, good for getting a quick file overview. Note that again Visual Studio supports collapsing all sections with Ctrl+M+L, but this really sucks because all methods, xml comments and everything else is collapsed too and expanding all of that again is a pain. Again, Ctrl+5 is much better.
  
  
  
  
• Update all comments and xml blocks for the whole file.
  Hotkey: Ctrl+6
  Similar to Ctrl+1, but will not only process the current section, but the whole file. This generates and updates all comments in the whole file, updates the header comment and makes sure all Xml comment blocks are generated if missing (see below).
  
  
  
  
• Update just the header comment.
  Hotkey: Ctrl+8
  Force updating the header comment, can be useful to test new headers and to remove any old header data. Also updates the created and modified timestamps. Since v1.9 changing or having different header comments now works, but it will also remove any existing header comments to replace with the new header comment if the format is different. Also check out the new cool options in the Commenter options screen, which allows you to specify seperate header comments for each solution or project if you like.
  
  
  
  
• Generate xUnit or NUnit unit tests automatically for us
  Hotkey: Ctrl+9
  Generate xUnit unit tests automatically for us for all methods! All unit tests will have to be written and tested until everything is implemented correctly. Existing test methods will not be overwritten! You can either enable xUnit Tests (like in the screenshot below) or NUnit Tests.
  
  
  
  
• Remove All Comments, use this to cleanup a file
  Hotkey: Ctrl+0
  Remove All Comments, use this to cleanup a file. It will only remove generated comments (after } blocks). Also nice for testing and cleaning up other peoples code.
  
  
  
  
• Obfuscate whole file (removes ALL comments, xml comments, empty lines, headers, etc.)
  Hotkey: Ctrl+\
  Obfuscate whole file. All code will stay the same, but we will remove ALL comments, Xml Comments, empty lines, headers, etc. to reduce file size and make code harder to read. Not really a Commenter feature, but I implemented this for another app and it was easy to implement here too.
  
  
  
  
• Autogenerate XML Comments
  This feature will automatically add XML Comments to methods, properties, constructors, classes, etc. when the normal comments or generated. You can specify if you only want to generate XML comments for public methods or if you want to generate XML tags for all parameters (which was default on, but is default off now to produce much less empty xml parameter comments, which are usually not updated by the programmer anyway).
  
  
• Other Improvements in v1.9
  
  • Added cool new per-project and per-solution header comments
    
    
  • New shortcut Ctrl+6 to just update the header
    
    
  • Added lower limit for block size to reduce comments for small blocks (also works recursive)
    
    
  • Added new features and hotkey logic, everything now easier to customize
    
    
  • Easier visibility if plugin is installed and working (more visual events, better options screen)
    
    
  • Fix generating comments for new files (messes up the using directives region), seems to be related to generating header comments
    
    
  • Improve line length count for removing comments (good new standard: 5 lines)
    
    
  • Fixed several problems in Format-Strings or SQL code.
    
    
  • Added whole method identifier from the accessibility keyword to the closing parenthesis for region directives.
    
    
  • Improved the CodeRush-autogenerated regions (Ctrl+3), make sure comments are not messed up
    
    
  • New default: Do not generate Xml parameter comments (mostly not used for better documentation anyway, adding them yourself for more details is more comfortable)
    
    
  • Added C++ "pragma region - #pragma endregion" support (thanks for testing to Enrico)
    
    
  • C++: include includes into using directives region
    
    
  • C++: Fixed header comment (project name could not be found)
    
    
  • C++: Do not use xml comments, just leave normal comments!
    
    
  • Fixed wronly generated regions outside of our plugin
    
    
  • C++: Fixed #includes region, do not modify include lines
    
    
  • Fixed region generation when CodeRush generates region before us!
    
    
  • Fixed header in case some other header comment is used
    
    
  • Added multiline header comment support for // /* */ -- and # to support all kinds of languages and comment formats
    
    
  • Also force generate comments on properties (methods work, also test constructors)
    
    
  • C++ allow Ctrl+4 and Ctrl+5 for C++ region collapsing (using visual studios Edit.ToggleOutliningExpansion and Edit.ToggleAllOutlining as fallbacks)
    
    
  
  
• Older and Improved Features (from v1.8 and before):
  
  • Fixed '}' inside string or comment, even if CodeRush has not processed the whole file yet (double checking now). E.g. the following string will not longer produce any comments:
    SQL = String.Format("UPDATE GC SET Status = {0} WHERE ID = {1}", Msg, ID);
    
    
  • Supporting to generate regions again, but split up function names, also suppress CodeRush region generation with Ctrl+3, Ctrl+R (also suppressing VS hotkeys now) Better support for auto generated region blocks for big methods, just press Ctrl+3 to generate region (let CR_Commenter choose the name automatically, allow editing after pressing Ctrl+3).
    
    
  • Fixed #if DEBUG in Using directives (caused trouble, was removed)
    
    
  • Also added code to always remove #if DEBUG and #endif, especially at the beginning and end of the Using directives region
    
    
  • Fixed double line #using Using directives issue when using directives got mixed up.
    
    
  • Fixed long line commenter generation, will now still work, but not longer cut of any of your code. If you have long code lines (>100 letters), you can still generate comments for them and the Commenter will not longer cut anything off (you are responsible for formating the code).
    
    
  • Fixed auto-collapsing of regions, especially when closing blocks in methods, the method should not auto-collapse!
    
    
  • Checked support with other languages (which are at least similar to C#) like Lua, Python, etc.
    
    
  • In Python/Lua, etc. we had to disable } comment generation and also use other comments (# for python, -- for lua) when using Ctrl+1 for header generation.
    
    
  • Test all other hotkeys and make sure they do not affect the code badly in CodeRush unsupported languages (note: VB is still not supported since it is very different from c# code).
    
    
  • Supports now any unit test framework, actually does not care anymore what you use, only the unit test generation code will add Xunit, but you can easily change that code if you want to.
    
    
  • Options are now available directly into Editor tree of the CodeRush options!
    
    
  • Refactored all classes, added about 4 new classes and removed 1 unused class. The source code is now much easier to read, to follow and to change.
    
    
  • Remove commented out code, cleaned up the source code a lot.
    
    
  • And finally applied the commenter to the commenter source code :)
    
    

I would also like to thank all the people that wrote me emails, use the plugin, all my colleagues that use this tool everyday too (shoutout to Enrico Cieslik and Henning Thöle, and also Fabienne Khöle) and the following people who have contributed with ideas, code improvements and helpful bug reports (they are also mentioned in the TODO.txt file for their specific bug report or improvement idea): Andy (mangledmonster), Andrew Jones, Gil Yoder, Alexey Romanov, Enrico (Judge), Ed Blankenship, Sven Heitmann, Jan Diederich, Rob Heyes, Dan Avni, Andy Jacobs and everyone else :)

I've updated my Commenter plugin for CodeRush today. I did some refactoring and was getting annoyed that some of the .NET 2.0 features did not get commented properly yet (like generics or anoynmous delegates). I also fixed some older bugs and issues. This is the changelist (full history can be found in the original article): First version for VS2005 and using .NET 2.0 now for supporting generics and anoynmous delegates (more useful comments now inside of anoynmous functions and code blocks defined there). Fixed again: No xml generation inside of methods (recent CodeRush version changed its behaviour somehow). Check is now performed inside the recursive method, this fixes all older issues too. When commenter is disabled the globol hotkeys don't work anymore now. Version number in options and a link to check if any new updates are available Hotkey for generating comments and XML: Ctrl+1. Saves you typing '}' and doesn't require you to go to the last line of a block all the time. If using defines in the using statements the surrounding region is now generated properly. Also adds automatically the #if DEBUG and #endif statements (which I use to exclude NUnit from the release build) around using NUnit.Framework, which is often generated by adding the namespace automatically or using some template.

Click here to download CR_Commenter.zip v1.5 And here is the sourcecode for CR_Commenter v1.5

I also wanted to add an embed with region feature, but it is already pretty simple with CodeRush if you know the hotkey: Select a block you want to build a region around (e.g. a method, you can press ctrl+shift+up/down quickly select the method). Then just press Ctrl+R, thats it.

If you are using CodeRush (and you need it for my Commenter) you should also check out these new CodeRush tutorial Videos from Developer Express, they are pretty cool and informative.

I've also checked out the Boo Programming Language today after playing around with IronPython the last couple of days. Boo is really cool and introduces a couple of ideas I agree 100% with. I have to try it out a bit more. One major problem is the fact there is no Visual Studio addin and typing in some editor or the console is no fun at all. A plugin for MonoDevelop and SharpDevelop does exist (and is actually implemented in the latest versions), but while these are very good open source IDEs, I'm a Visual Studio user and have a lot of addins I would miss in any other editor.

I use also use UltraEdit, which is pretty cool and can be used for all kinds of shit (not only text, hex, html, but also millions of source code formats). However, the Boo language is missing syntax highlighting support in UltraEdit (while almost 1 million other languages are supported). I just wrote this little wordfile myself for UltraEdit Boo syntax highlighting support: boo.txt, just add it to your UltraEdit wordfile.txt file.

Again: Boo is a cool language. I heard of it at the PDC, where I saw even the developer in some session, but I never checked it really out (maybe it was too Linux oriented for me). Maybe I will try to convert the SharpDevelop addin to Visual Studio if noone else does it (because I would really want that feature when coding in Boo). I still like Lua and Python too, but they don't have any intellisense support either, which is one of the main reasons I haven't switched over to writing more in those languages instead of c# right now.

I'm working on a little other project right now and that will take a while before I can show anything. Its about a new programming language and testing out MSIL (.NET immediate language, looks like assembler) stuff.

I wanted to make a review of this year, so instead of thinking to much about it, here it goes. I will only talk about games, books or movies I know and saw, obviously this all is just my opinion! Some entries are older than this year, but I saw, played or watched them only this year.

Btw: I wrote this whole article (which is almost 40k in html, which I always use when writing blog entries) in a couple of hours and I have to say my keyboard layout I invented last year is really kicking off. I'm writing faster than ever and stay as relaxed as if I hadn't typed much at all.

Content

• Best Games
• Best Movies
• Best Software
• Best Programming Books

Best Games

The best game for me is still StarCraft, so lets talk about games that did come out this year.

Action games and others (only the best). In just the last 3 months a really big number of shooters did came out, all of them are really good. Strategy games are handled seperately below this section.

Battle Field 2 4/5: Not much to talk about here. This will be propably game of the year if voted by online gamers. It is a good solid online game, but a bit too time consuming and too realistic for me. I'm more on the fun side when playing a shooter. The predecessor Codename: Eagle (that is before Battle Field 1, back in the year 2000) had already a lot of the game play in it and was oriented more on the fun side. I still like to play it on LANs. Battle Field 2 is also a great game for LANs, if you have enough people.

Serious Sam 2 5/5: Serious Sam is just a great fun game. I especially like it because I can play it like Quake for a couple of minutes and then return to working. There are not much games around today to do that. Also the amazing mass of levels and cool ideas in this game are just great. It is also the best coop shooter ever and part 2 is as much fun as the first Serious Sam. The only thing a bit annoying is the video skip error in coop mode and that there are no times in the games were you can relax, it is action - action - action! I already played it from start to finish twice, really good job croteam (once mostly alone and once in coop again).

Quake 4 4/5: Hey now! I'm a big fan of all Doom and Quake games, I've played them all and they invented the whole shooter world today. Doom 3 was a disappointment for many people and Quake 4 looks exactly the same. However the gameplay is really good and except the fact that the first couple of missions are way to easy IMO the game gets really good and hard in the second half. It has a decent amount of levels and is a really good singleplayer shooter overall. The multiplayer part makes like in Doom3 no sense at all, I still can't understand why so much good singleplayer games force themselfs to implement a multiplayer which just can't work with this kind of a game (same think for Fear btw). The scenes in the first half of the game get really brutal, but if you know about Doom 3, you won't get shooked by Quake 4.

Fear 3/5: This game got me really exciting and the demo was a hell of a demo, one of the best shooter demos I've ever played. The first couple of missions are still good, but then it gets boring and more boring. You have seen all the graphical effects and buh ho the girl and that old man that appear in some corner or directly in front of you don't shock you anymore. The more I played the more bored I got, I haven't finished that game yet. And that is a really bad sign because I usually play every single game I like from start to finish. It reminded me a bit of Project: Snowblind, which got boring after a while too and I never finished it. Anyway, if you get shocked or like horror or action movies this game is for you (ok, you might need a powerful and fast computer too).

Call of Duty 2 4/5: Call of Duty 1 was a big success. I liked really don't like any World-War-II games because they are almost all the same and there are way too much games with this setting. However Call of Duty was different, it was fun to play and the missions were designed really good. The only thing of part 1 which annoyed me a little was the incredible short singleplayer (4 hours and it was already over). Call of Duty 2 is very similar to the first part, but includes new settings and exciting new graphical effects. Overall the game is produced really good. I haven't completed it yet, but I plan to play it soon. People who have played it liked it a lot. There are a lot of fan sites and stuff around, but I remember some strike some time ago because the community didn't want to wait any longer for the game SDK and said they want to stop playing CoD. Dunno what happend to that.

Guild Wars 3/5: Guild Wars is a really good MMORPG (massive multiplayer online role playing game) and a lot like World of WarCraft (which is also a nominee for the best game of this year for most RPG fans), but it doesn't cost 15 bucks each month. I tried it out in may or something, but it didn't motivate me enough. I only played to level 10 or so and then stopped playing because I had other things to do. Thats the main problem with RPG games these days, they take too much time and they are just too chaotic and not shooter like enough. I like the older games like Diablo2 or Sacred (btw: the Sacred addon which came out this year was fun too) much more, which are fun to play for a while and then I can throw them away.

Strategy Games (RTS):

Act of War 3/5: At the first look this game is all about graphics and effects, the videos. Then after the second look it is even worse, there is no good unit handling possible and almost no balance at all in the game. However, it was still fun to play the singleplayer (as short as it was) and overall it was an ok game. The multiplayer is due the missing balance and the immense amounts of hitpoints each unit kills not really fun. C&C players will like this game more than StarCraft or WarCraft gamers.

Armies of Exigo 3/5: This game did come out November 2004, but I had no time to play it back then (only the demo once). Like The Lord of the Rings RTS game (see below) it is very similar to WarCraft III, but IMO Armies of Exigo is way better despite the really bad success it had (almost noone knows about this game). Armies of Exigo was planned as progamer game, has some really decent videos and nice graphics (almost as good as LotR and way better than WarCraft III). However, all community sites have closed and there is absolutely noone playing online (and not offline either ^^ the marketing sucked even worse than for Arena Wars, and this is an EA Game). Anyway, it is still one of the best RTS games I played this year.

The Lord of the Rings: The Battle for Middle-earth 2/5: Basically this game is exactly the same as Armies of Exigo except the big name and the amazing marketing and success. I personally don't like this game at all. The singleplayer is boring as hell, every mission is the same and only the first couple of missions were made interessting. The multiplayer is even worse, there is no good balance and the gameplay itself is boring as hell with all the heroes and bunkering in bases. When I compare LotR online players (e.g. in the ESL or Giga leagues) with other RTS gamers (like WarCraft III or StarCraft) it is really sick: The LotR are sitting there playing with one hand only and clicking the mouse a couple of times per minute. On the other hand WarCraft III or StarCraft players use both hands and click like 50 times per second and almost destroy their keyboards while playing (which is also more fun to watch ^^).

Earth 2160 1/5: Another game where almost every screenshot looks like it is a shooter. Last year a lot of games like that did come out, all sucked. But the bad controls or the strange screenshots are not the problem of this game. The intensions and ideas for Earth are really good and the graphics look fine too, but the game is just no fun to play. The singleplayer is just boring as hell. You get the feeling like no of the designers plays any other RTS game. It is more fun to play Warcraft I or Dune 2 (which are almost 14 years old). The multiplayer makes absolutely no sense at all, it is very hard even to log on and when you finally made it there is noone around. I played online once or twice and it is really just crazy how imbalanced this game is. PC Game magazine claim this game has a great balance and when you actually trying to play it seriously it is like running against a wall. Tip: Choose the alien race and just build few brain bugs, you can't lose a game now. Better tip: Don't play this game at all. And don't ask me why this game has so many awards it doesn't deserve.

Age of Empires III 3/5: And finally a couple of month back a new Age of Empires game came out. The graphics are really good and it is one of the first RTS games ever to make good use of shaders and normal maps. It is a solid game and especially something for fans of the Age of Empires games. The Age of Empires games did never really convince me. Maybe I just don't like the setting, Sci-Fi or the present time is much more interessting to me. Other than that I havn't played AoEIII a lot yet, I can only tell you that the heroes idea and the extras you get from your hometown are really bad ideas IMO. While I would say AoE is more of a singleplayer game than a good balanced multiplayer game, the already big online community of AoE does tell me the opposite. There are already a lot of leagues with AoE in it.

As you can see I'm not very friendly to RTS games. I rather play older games like StarCraft, WarCraft III or even C&C Generals than any of the RTS games this year (yesterday I even played WarCraft I (1994) from start to finish, cool game ^^).

I don't know of any other RTS game that did come out this year, maybe I missed one? Btw: I don't count Tactical games like Panzers into strategy games. Neither do The Sims or games like The Settlers or Civilization count on my list ;-) I'm not saying these games are bad, I just don't play them and couldn't tell you if they are bad or really bad, hehe. Also I think Civilization was the best game in its time and The Settlers 1 was really great fun, but why do I have to play the same game over and over again every year? Its just boring and the new sequel games don't add anything exciting to it. Anyways, the sell like shit and people keep producing them ...

Ok, enough overview. I played few more games this year, but most of the games I just don't like and I uninstall them after a couple of minutes playing them. Other games are only small games or shareware and they are not worth standing next to this big games ^^

So whats the best game for me this year? On the shooter department it is Serious Sam 2, which has the greatest replay value (I didn't even finish Fear and I will not play Quake 4 again for a while). For the strategy games above I would say Act of War and Armies of Exigo were the best RTS games released this year. I don't play AoEIII, but compared to Act of War or Armies of Exigo it is much better produced and has better graphics, I guess most people will vote AoEIII as the best RTS game this year. But please don't mention Earth 2160 ever again :D Lord of the Rings will stay with us for a while, no matter of what I think or how good or bad the game actually is.

So why not make StarCraft the game of the year 2005? I played it more than any other game this year :D It is amazing how much comunity sites and leagues were created this year for this 8 years old game (GosuGamers.net, PGTour, ESL-Pro Bw, GGL AmeriCup, etc.).

Some links about top games this year:
Best of E3 2005
IGN PC Best of E3 2005
Top 10 2004 (more like Top 10 of all time)

Best Movies

Well, I watch some movies from time to time too, but I can't remember them a week later ^^ For this reason this section is really short. For a great movie site and I bet they will do some 2005 review too, take a look at www.rottentomatoes.com.

• King Kong 4/5: I just saw it a week ago, so I can still remember. It is amazing that this movie cost 207 million dollars. It is a really stupid movie and somehow I expected too much. Once you relax and just "watch" the movie (if you don't fall asleep in the first half) it is actually a good action movie. The dinosauriers were fun and King Kong running around in New York was great too. Just don't talk about making sense ^^
• Serenity 5/5: I'm a fan of the Firefly series, so it is no wonder that I like this movie, which is based on the Firefly series, which was canceled after 1 season 2 years ago. Basically the idea was to mix the wild wild west with a sci-fi theme in space. The characters are really interessting and it is just fun to watch, they don't take themself too serious like some of the Star-Trek series. Anyway, this movie is really good, also for non-fans, it is produced ok and the storyline is interessting and exciting too.
• Harry Potter and the whatever this year 3/5: Well, I watched the first one and then skipped the other two and suddently Harry Potter became really old and some of the scenes (like the bubble bath scene) were really inappropriate IMO. For Harry Potter fans this is a good movies, for the rest of us, it is just ok.
• The 40 Year Old Virgin 5/5: This is a really funny movie and I liked it.
• Mr. and Mrs. Smith 4/5: I just like Angelina Jolie, so every movie with her in it has to be good. Well, don't remind me of Tomb Raider. Anyhow, this movie is actually fun to watch and has a funny plot.
• Madagascar 4/5: A nice family comedy (I went with my little sister), but the amount of CGI movies the last couple of years is rather sickening. But if you only watch every 10th of them, they are actually good ^^
• War of the Worlds 4/5: A lot of people really hated this movie. I can't understand why, it was a solid sci-fi movie and I really liked the idea to tell the story from a completly other perspective (not the usually 1 man kills all boring action shit). Maybe it was Tom Cruise, who is really stupid (ever heard him talk in an interview?), but he is a hell of an actor and I still think this movie is one of the better ones from Spielberg these years.

Ok, I can't remember more movies right now, who cares anyway :D

Best Software

Well, the award goes to Visual Studio 2005, there is no competion, next section please :D

No, seriously: For programmers Visual Studio 2005 is the greatest thing happend this year. Even the beta, which I start using at the beginning of this year was very useful and made me much more productive. Ok, lets try to think of anything else:

• TestDriven.NET: This rather simple NUnit addin for Visual Studio is a great way to use unit testing in Visual Studio, you can also use the implemented Visual Studio Test System (which is good if you plan to use Visual Studio Team System in the future), but if you just have Visual Studio Express NUnit is your only good choice. Download it from www.testdriven.net. The TestDriven.NET Version 2.0 for Visual Studio 2005 is available since last month too, good stuff.
• CodeRush: Not really a new tool this year, but it is still updated and the most useful Visual Studio addin for me. They just made a couple of cool tutorial videos for it a couple of days back. I've also written an plugin for it: CR_Commenter.
• FX Composer v1.8: The best shader edit tool around and NVidia still keeps updating it. Even our graphic artists use it now, after hiding the code and just showing them the buttons and scene preview, they are happy and can use it just fine to finetune shaders.
• IronPython: IronPython is the implementation of Python in .NET and it outperforms the original script language by the factor 2 or even better. I played a little bit with it after the PDC this year and it made my think of a couple of new ideas. I'm still a fan of Lua and think it is the better and easier choice for small programs or just saving small scripts. However, the work of the IronPython guys - who work at Microsoft now - is really amazing.

Can't remember more plugins, but I'm sure I used a lot more and tryed out a lot of stuff this year. Check out this list of Ten Must-Have Tools Every Developer Should Download Now (by MSDN Magazine). I will not go into detail about other useful programs and tools like Photoshop CS2, 3DS Max 8, Miranda IM, Skype, etc. there are better sites than mine for that :D

Best Programming Books

Graphics and Shader technologies:

• Game Programming Gems 5 5/5: Like any other Game Programming Gems Book this one is no exception, it is just great. You can read it from front to back or use it as a reference if you look for solution to common game programming problems and even programming problems in general. It covers 7 sections from general programming and mathematics over AI and physics to finally Graphics, Network and Audio. This one is the most recommended game programming book series.

• ShaderX4 4/5: Another book series, I read the first 3 and use them as a reference whenever I look for cool shader effects. The 4th volume isn't released yet (but is comming this or next week, I preordered it a while back). I expect the content to be as good as the predecessors. Wolfgang Engel is a good editor and I heard him speaking at the Quo Vadis developer conference, he knows a lot about shaders ^^.

• GPU Gems 2 4/5: And another book series (does it ever stop?). Good stuff for NVidia developers (I like NVidia), but since most shaders work fine on ATI hardware too (except you tried to write PS3.0 when ATI still hadn't any cards for that), it is a good shader book. It covers some nice tricks and can be compared to the ShaderX books. Unlike the Game Programming Gems or the ShaderX books every page is printed in color and this makes it also a good colorful picture book where I can show effects to my artist guys, who won't even bother to look if it isn't a nice picture.

• Shaders for Game Programmers and Artists 3/5: This is one of the first shader books I've read, but even for a first book I think it is too shallow. It jumps right into post screen shaders, which are very advanced IMO and then continues with much easier shaders. The book is all about ATIs RenderMonkey tool, which I don't like because the output is unusable (I need freaking fx files). It is still a good shader book and may be helpful for artists, beginners or people who just want an overview of shader technologies.

• Programming Vertex & Pixel Shaders 4/5: This book is the best and most complete shader programming book for anyone seriously wanting to go into shader technologies. It is a bit harder to read than the previous one and sometimes way to mathematical instead of just having fun with shaders, but it pays of because you learn all basic shader technologies you will ever need. Again a book by Wolfgang Engel (like the ShaderX books, which are more advanced).

AI Programming:

• AI Game Engine Programming 4/5: I just got this book together with Programming AI by Example few weeks ago, so I haven't read it all yet. But from the looks of it, it goes into a lot of shooter AI problems and discusses useful techniques, not only about AI, but also how to use scripts (Lua), how to write all kind of state machines and neural nets. The CD does not only contain all sourcecode and figures, but also a bunch of useful web bookmarks sorted by category, I like that :)

• Programming Game AI by Example 4/5: This book is a bit more for beginners and immediate programmers than the rest of the books here. It is still a great book, but it "only" explains how to get into AI programming, as soon as you are ready to go, the book ends. The book starts with math and physics and does a good job explaining them. Then it goes to state driven design and continues with game agents. It does also cover questions about path finding, fuzzy logic and scripting. It is written by the founder of www.ai-junkie.com and written really good. My only critic is the fact that it is a good beginner book (and how likes to be considered as a beginner ^^). Anyway, I got it together with the previous book at the same price and thats really ok. Maybe it is good for a reference or to tell interns or wannabe AI programmers "Go read that".

• AI Game Programming Wisdom I and II 5/5: Similar to Game Programming Gems this series is all about finding skilled professionals writing capters and articles, which do really help you out. Steve Rabin is the editor and did also edit the Game Programming Gems AI sections, he also is the creator of www.AIWisdom.com. Like the other two AI books I havn't read all of it (I'm so busy you know), but from what I've seem and read this is a really helpful resource when doing anything releated to AI programming.

General Development and Programming:

• Refactoring: Improving the Design of Existing Code 4/5: Dirk Primbs said to me when I read this book on the flight to the PDC that you could probably dry up a cellar with this book. Maybe he is right, this book is all about crazy design pattern and rules. The first couple of capters are good to read and the rest is more of a reference. Anyway, the idea counts and refactoring is one of the most useful processes today in writing big programs. Good book, was written in 1999 and still applies 100%.

• Rapid Development 4/5: This is a good book about software design and keeping schedules, but it is a bit too long and sometimes hard to read (you know I have this problem of reading only half the book and then never finding time to finish it ^^). It was written back in 1996 and the techniques described still apply, but some of the ideas are not as flexible as they could be with all this new technology around (e.g. when using agile methologies you have to plan differently). Still it is a very good book and it contains a couple of interessting stories from big products like MS Word and how they never kept their schedule ^^

• Code Complete 2 5/5: One of the best books ever for any kind of programmer. It is about describing the process of developing software and helps you to find out the most efficient ways to manage your projects. It does go into great detail by explaining which data structures, which routines, which loops or which strategies are the most useful. It is one of the books I would recommend both to new programmers and to experienced programmers the same way. This book just helps anyone and should be on every programmers desk, not the book shelf.

• Managing Agile Projects 4/5: What does Agile Projects mean? It is about scaling and customizing your project depending on the customer feedback . That sounds still to vage? Ok, it does mean that you don't plan every single bit before writing code, but to cut your project into smaller pieces and only plan the overview and then directly start developing. Now you can present a very early version really fast and with the help of feedback (customers or yourself) you can adjust your project instead of wasting time and resources developing something noone wants. You can also shorten specific parts of your project if you see there is no time left or other parts are more important, which again is not possible if you had planed every bit of your project in advance. It is a technique that goes hand in hand with Unit Testing and Refactoring (see above). This book does give a very broad overview and doesn't really talk about coding, its just methologies. Anyway, its a good book (not too long, this way I can actually finish it even with my book reading problem, see above, hehe).

• Maximizing .NET Performance 5/5: I'm a performance freak. In the past I often tried to reimplement existing and working code into assembler just to see how much faster it could get. Later I found out that it is often much more important to work at a much higher level and rethink a problem until the solution is good enough to run very fast even if not optimized to every bit. This is still true and I still think that most performance problems come from bad coding or suboptimal algorithms and not because of the language. However to even think of good solutions you need some knowledge of what is possible, what is fast and how do certain things affect your performance. This book gives you a very useful inside of .NET and covers a lot of tricks and tips about .NET performance. It is also a good reference book.

• Code Generation in Microsoft .NET 4/5: Last year I heard this DotNetRocks radio show and Kathleen Dollard was the guest. She talked about Code Generation using templates and other tricks. I immediatly bought this book. It is mostly written in Visual Basic, but a c# convertion does exist too. It does a good job explaining how to use Code Generation with the CodeDom, but does not go into detail about MSIL (which I was more interessted in). Anyway, the book is written nicely and I could learn a lot from it. The books presents an entire framework for building SQL bindings, stored proceduces and building WinForms from XML templates. It is also one of the first books on this subject (and maybe still the only one going into the CodeDom instead of MSIL).

Other:

• Programming in Lua 5/5: This is THE book for Lua programmers. It is also available for free to download from lua.org. I first read the online version and then bought the book because it is so good and it helps to have a reference. Lua is a very simple script language. But sometimes it is so simple that you just don't know which keyword to use or what to type. Having a few useful code examples every other page is the biggest help ever. With help of this book I learned Lua in 1-2 days and could really do useful stuff with it (instead of writing just hello like most languages you learn in a short time).

• Chris Crawford on Game Design 5/5: I recommend this book because it is easy and fun to read. Chris Crawford sound sometimes a little crazy and the fact that he hasn't released any games in the last couple of years might speak against him, but his experiences from the past (he talks about the 70th, 80th and 90th and games he has done and many other classic and inovative games. He goes also into detail what is missing in most games today and how to fix that (very theoretical, but his thoughs are interessting). Even if you are not smarter after reading this book, you will feel smarter.

Ok, that's it. I have a couple of more books lying around here (like Extreme Programming Adventures in c#, Physics for Game Developers or The Pragmatic Programmers), but I have not read them yet and can't give a good review about them, but they are all good books I guess. I also recommend The Odd Todd Handbook: Hard Times, Soft Couch, which has nothing to do with programming, but helps you sometime to get up and not end like him.

I hope you like my review of 2005. If not or you have anything to add and don't encounter the Internet Explorer bug not able to post comments, you can post a comment below ^^

Day 19: Some effects in action, the shoot line helps you to see where you shooting at (will be improved with a special texture, its just a line now). Some 3D to 2D converting is also not correct and needs some fixing. In the next days I will try to bring together all the completed parts and start with the mission design in the beginning of the next week.

Ratta ratta, peng peng, boom!

Shoot'em'up

Yeah, hey now. This time I had way to much fun when generating this screenshot, now it is actually possible do aim with the tank and shoot at stuff (with all the new effects and sounds). It is a lot of fun.

Warning, this is a long post (I've written it in the last 2 days), but I have a nice idea: Content links, just click on them to jump to a section:

• Shoot'em'up
• Troubles with IIS
• RenderState errors
• Light effects in DirectX
• OutOfVideoMemory in Unit Tests
• VS2005 needs a lot of RAM
• Links for today
• New StarCraft site: www.GosuGamers.net
• Tropical-Islands resort

Troubles with IIS

I had some trouble the last few days with my IIS (Internet Information Server, the thing producing ASP.NET web sites), it keept crashing and hanging up with out of memory errors (this page was down a couple of times, sorry!) Really strange, that thing is running for over an year now and I never had much troubles with it. The memory usage was also pretty high (total ~900MB, IIS worker process alone had ~400 MB), I guess this has something to do with dasBlog using .NET 2.0, dunno (there are no errors logged anywhere). I had a lot of page hits this week after all the posts on other blogs about me.
The memory consumption of IIS goes constantly up and I had over 200MB mem usage again after a couple of hours, hmm? Don't wounder if the site is down or you get some errors, may happen because of these memory errors ...

I also optimized the page from 50 entries max to 10 and cut down the number of entries you will see on the start page. I hope that helps a bit, my server is really crappy and old and needs a lot more memory (had only 256MB, but I upgraded to 512MB recently, but its slower memory. I guess I need a gig or 2, lots of other services are also running on that server). I also changed the app pool to a special one and set some timeouts for requests and processing, maybe this will help (had no problems since that change).

RenderState errors

Another annoying thing is the RenderState property of the DirectX device, it will tell you you can "get and set" some state like FogEnabled, well you can set it, but if you try to get the state it will just throw an exception. A lot of RenderState gets will throw an exception actually, I'm really impressed by that. Great work not to mention any of these things in the docs (the docs are pretty useless anyway, most descriptions are trival anyway and IntelliSense helps you just as much).

Light-Effects in DirectX

The light effects for the effect engine were pretty easy in DirectX, I remember this was very hard to do in OpenGL (had to invert the alpha color, and with different color settings everything got messed up, so a lot of special cases had to catched and handled).

Basically this code is used (the light effect is a white texture with a spotlight effect in the middle as the alpha channel):

Device.RenderState.AlphaBlendEnable = true;
Device.RenderState.SourceBlend = Blend.DestinationColor;
Device.RenderState.DestinationBlend = Blend.SourceAlpha;

Now make sure both the color channel and the alpha channel will modulate from the texture color and the diffuse color. Switching back to normal SourceAlpha - InvSourceAlpha causes also no troubles with that. I like that :)

DirectX OutOfVideoMemory

Another problem I did have the last days when unit testing was running out of video memory all the when creating the DirectX device (getting D3DERR_OUTOFVIDEOMEMORY exceptions) after ~10 tests. When restarting VS, everything worked again for some time.

I worked with that problem all week and found no solution, I changed a lot of debug settings and changed some parts of the code, but nothing helped and there wasn't any solution for out of video memory exceptions posted anywhere (my card has 128MB, how can I run out of vid mem after a couple of tests?). Then I saw that some part of the screenshot generation code wasn't finished and I saw the presentParams.SwapMode was set to Flip instead of Discard (Discard is somewhat faster, but Flip allowed me to easily catch the back buffer for the screenshot generation). I changed that, reduced the back buffer count to double-buffering and reduced the resolution for tests from 1024x768 to 800x600 and never had any Out of Video Memory exceptions since then. Maybe they happen after over 50 or 100 unit test runs, but then I can just restart VS and it is not as annoying as before.

VS2005 needs a lot of RAM

But VS2005 consumes still a lot of RAM (>800MB, I got only 512MB ram on my dev machine). I guess I have to buy some more RAM soon, everything gets very slow and crashes more often when all the ram is used (and I can't work without VS2005 anymore). Restarting VS helps, but with that much complex unit tests the memory comsumption goes up very fast again. For example todays screenshot is actually just a unit test with around 50 lines of code, making very simple unit tests helps a lot to force you to refactor and simplyfy your code constantly.

Todays Links

The links for today are The Z Buffer and Managed world, both great Managed-DirectX sites.

• The Z Buffer Managed DirectX resources by
• Managed World great site with tutorials and articles by Jason Olson, check out his blog.

GosuGamers.net

If any StarCraft fan reads this, the guys from www.StarCraftGamers.com did have a lot of troubles with their hosting partner and left the partner. StarCraftGamers.net was one of the greatest StarCraft community sites ever IMO, the new site is called www.GosuGamers.net. Looks like the crew will continue their great work there, so check it out (the site has still some bugs because it is new): Good luck www.GosuGamers.net

Tropical-Islands

I'm now going to visit the Tropical-Islands resort (new big thing here in germany) with my brothers and some friends :) Development will continue tomorrow.

Omg. I have become a total slacker the last couple of days, no updates here :( I didn't had much time over christmas to code, but I read some more of my Rapid Development book (see last post) and it is still interessting and insightful. Yesterday I still was in vacation land and had to force myself to get back to coding ^^

I also reconsidered some choices I made and came to the conclusion that this project was originally started to make a quick little game with DirectX9 in c# and also to write a new base engine for upcoming DirectX9 projects. I did obviously spend too much time researching shaders (DirectX9 fx high shader language) and saw last week there is no way to implement that nicely in 2 weeks (when the game is running with all the other parts finished I will reconsider implementing shaders). For the engine part of the deal this is still great work and I have a lot of base classes for any shader effect development ready, but for the "quick little game" Lost Squadron part it is not looking so good. It is a conflict of interests because finishing up something quickly does not work hand in hand with researching and implementing new techniques. For this project I think it is better to focus on developing a solid base engine, next project will be planed much straighter.

Yesterday I collected a lot of effect graphics and sounds and made some new ones. I got now a nice collection of effects with graphics and some funny sounds for the effect engine and made a couple of effect types (parameters, testing, etc.), but when adding more and more effects it became very obvious that this would be a great think for scripts. Instead of hating myself later for not using scripts right away, I just went crazy and implemented the whole effect system in LUA scripts and can add new effects there without changing any code of the engine. I did the base code for that a month ago (check out LUA.NET for doing that in c#) and have read the great book Programming in Lua by Roberto Ierusalimschy.

LUA links in case you are interessted:

• The Programming Language LUA The online version of the book by Roberto I. is also available there for free.
• LUA.NET for implementing Lua in .NET Sadly the LUA IL project isn't public yet, but LUA lib is ok too.
• LUnit Unit Testing with LUA.
• QDE THE IDE for LUA development!
• LuaLite Syntax highlighting support for Visual Studio, but does only work for VS2003.
• Lua-users.org Everything else you need for your LUA development

The effect engine needs still some testing and I haven't worked with lunit yet (will do that tomorrow), everything should be up and running tomorrow and I will post a new screenshot with some nice effects then. The rest of the base classes except wayfinding and enemy ai is also pretty much complete, so I hope I can finish up a small demo till New Years Eve (driving around, shooting at some stupid enemies).

Sorry for no screenshot today and sorry for the long pause. Fabian is ill (my intern helping me out a bit with Lost Squadron), hope you get better till next week :) Tomorrow the LUA effect engine should work and I will post a screenshot of the actual gameplay finally (I want to kill some stuff).