System.AddIn

I wish I had more time to write more often, but I've been quite busy lately. I have however something to share, and it's not much since it's publicly available, but it's still something worth having a look for those who, like me, didn't know about it until yesterday.

I'm talking about the System.AddIn namespace (the 3.5 version, it's an assembly you must reference).

While researching for plug-in frameworks, I've found that our beloved framework allows you to create extensible applications with very little effort. You don't have to code your own assembly attributes, your own interfaces, your own versioning gimmicks, and more importantly your own external assembly loading routines (which are most likely to revolve around the black magic-y world of AppDomains). This doesn't mean, however, that you don't have to design the way these addins interface with your host application - that's another story.

Two CLR Inside Out columns (this, and this one) introduce the System.AddIn pipeline. I wish I'd found this stuff sooner.

Two things I do not like: The first is the way the pipeline is layed out in terms of directory hierarchy; it's surely clean enough, but I don't like it when someone else decides where I should put my assemblies (.NET has a local assembly copy paradigm if you aren't going the GAC route), so, in this regard, it could be improved and allow for a little bit of customization. The second, you need to create different projects for the pipeline segments and I for one already have to deal with enough projects already. However, the shining beauty of this umpteenth incredibly useful feature of .NET quickly dissipates any shadow of doubt. Man, I can't believe I just wrote that.

Of Shame

I'd like to crystallize in a blog post the wisdom of an australian "developer" named "openglaussie", as seen over at Gamedev.net.

I guess nothing good will come from spaghetti eaters, mafia bosses and short , tan people, who bullshit others as a way of living, italians are the worst catergory in everything.

[...]

I don't think i have a wrong attitude, is just my way to put things on, i don't sugar coat, no way, italians aren't good at making games, i am not talking about knowledge, since we all know that italians universities are among the best ( only in litterature ), [...] technologically italy is many steps behind, no entertainment culture finalized at videogames [...], they don't use their cars anymore to spare oil

Please note how he also edited his posts later on. I'm not sure I feel like commenting on this right now.

Edit: well, while I'm waiting for the report I'm working on to finish crunching numbers, I guess I can spend a few minutes writing down how I feel about this.
That people of other countries have a low consideration for italians is nothing new. It's a sad reality, not shared across everyone of course, but it does happen. However, we're talking about game development here and I'll keep it at that.
I think Italy never ever had the possibility to grow in the gaming industry. We simply don't have a business for that. Many bright minds fly away from this country towards more fertile fields, where they can actually work and make a good living out of their talent. The idea of a game dev convention is both a good and an irrelevant news. A good news, seeing as there's always the need to spark a start, but irrelevant, since I really doubt it'll help investors create business. It's all about earning a living eventually (for big companies even more so). We had a few good productions in a distant past (Milestone with Screamer and a few other racing games, or Lightshock with Pray for Death which was a very nice fighting game at the time), but then it all stopped.

I really think it's hard to start competing with multi-millionaire companies ex nihilo. I also think we wouldn't be able to create a Cave Story clone (which is an amazing game, a one man effort worth gold) even if we really wanted. Why is it so? I don't know, all I know is that they can convention all they want, but that's not gonna help unless someone starts changing the business. And this is something not happening overnight.

I for one have always been interested and secretly in love with game programming. I never found anyone to share this passion of mine with where I live. The italian game dev site reminds me of the desert, with a few, lone explorers along its path.

But then again, I can't complain. I for one am not doing much to change this. I cannot.

Designing a Magic System

I just found out this amazing series thanks to gamedev.net dailies.

Designing a Magic System by Andrew Doull

(Plus a fine article from Craig Perko - found in the series above)

Seeing how magic and its systems have gotten a formal definition over time is both entertaining and enlightening. I used to design a pen and paper RPG rules for fun when I was in my teens. The act of balancing, adding to or removing from a specific rule set is not as straightforward as it might seem. Actually, while I'm all for coming up with cool ideas about magic and so on, I must admit players (including me) are used to defined schemes. So, I guess, we're not just balancing fireballs versus shields, but also marketability versus innovation.

And makes me wonder if we will eventually get used to "common" magic...

A change of... hosting

About to move to my newly bought domain... redirect your bookmarks (as if you bookmarked this site, ha!) ;P

Edit: as I mentioned in the comments, the new site is not ready and I'm not sure how long it'll take... :]

Edit 2: I guess I shouldn't have mentioned this "change" seeing as I'm finding blogspot so much more comfortable. It has everything I need at the moment. I'll post again should I ever actually change the hosting of my blog. Hurrah for pointless posts! :D

A wicker soldier

Orel keeps on amazing us with new modifications. I can't wait to see this guy hacking and slashing. I wish I could model and texture like this...!

Parallel Profiling

Did some profiling during my lunch break, since I was curious to see how a one-day implementation of a parallel library would perform against a super rival like Microsoft's TPL. This simple console app has been run in release mode, so no debug stuff lying around. My work laptop's a Fujitsu-Siemens Amilo Pro, Intel Core Duo 1.86 GHz, 1.75 GB RAM.

Rainweaver.Parallel Tester

100000 iterations Bessel function.
Comparison mode.
Parallel (RNW): 21076 msecs elapsed.
Parallel (TPL): 15877 msecs elapsed.
Sequential: 44302 msecs elapsed.
Ratio: Parallel RNW is 2,1020117669387 times faster.
Ratio: Parallel TPL is 2,79032562826731 times faster.

The results are pretty self explanatory, considering noise and everything. While this makes me a bit sad, I'm still happy to see that the difference is not that humiliating - at least, that's how I see it, and it's granted it could be oh so much better. All in all, my implementation is not optimized nor has been given that much forethought, so I can consider this a small step towards Nirvana and a great first-hand experience in parallel coding, which is invaluable.

I'd like to congratulate the TPL team on an awesome job. Can't wait for the release of this great library.

Coming next...

I've been dealing with a few interesting issues and finalized a good part of my base libraries. The time has come for me to start working on Darkmana. I've been messing around with my codebase to find out I was misusing a bunch of objects, resulting in a few annoying exceptions while using TV3D. Hopefully the problem has been understood; the lessons is, when dealing with multithreading, don't assume to follow the same sequential logic you'd use normally. This might be a given, but it's usually not. One enters in this focused state of mind in which everything clicks perfectly, just to find out an important thing has been left out at the very base of the structure.

Enough intro. In the next days, I'll commit some time to work on these subjects:

• Parallel library: why, yes, even if there's one available for download over at Microsoft, I needed to roll my own. I can't rely on their release date (which is unknown at the moment) and on code in beta stage (adding to my own). Luckily a superb Channel 9 video with a throrugh explanation helped me disentangle a few doubts (link for the interested).
• Using the above library to create a proper loading screen. Loading screens are cool, but only if I'm the one making them, otherwise they're annoying as hell. I'd love to let the player move the main character around until the world fades in around him upon load completion. Wouldn't it be cool? I think so.
• Programmatic animations and AI. This is going to become a big matter, there are so many things I need to finalize first.
• User interface: that will be one of my last concerns. I'd rather have a simple guy running around and jumping than a working user interface at the moment. I have learned a lot from my past experiments with user interfaces, and this time I'm sure it won't be as painful as the first time (I'm not even considering the actual first time, since it was a mess :P).

The people who subscribed in our forums for the beta version will receive the link to download the various tests I'll be uploading (when there's something juicy for them to test of course).

Of Cyclical Rewriting

Sometimes I wonder why I even bother. It all started because there was a bug in the NTL compiler apparently not recognizing handler tokens. I could have sat down and probably fixed it in a bunch of hours. But no. I had to put the whole NTL language idea aside (as I said to Zak, it was mainly a "programming masturbation", but a cool one ;( ) and fall back to XML.

Well, I think it's eventually a good choice since XML is widespread, can be read by almost anything, espresso machines included, and is an accepted standard.

Luckily I didn't have to rewrite all the code that actually creates CLR assemblies out of the entity system definitions you spam in, or it would have been serious pain. Much pain.

Anyway. While I was at it, I also removed a feature which I didn't like. When the entities were processed and turned to IL, they were added a field named Prototype, which contained an EntityPrototype object basically describing the objects referenced by the entity; this means components, handlers and handled messages. All of this just to speed up entity creation at runtime.

However, I had forgotten once again about Attributes, and why they're in the CLR. So, armed with loads of patience but handicapped by lack of proper sleep, I set to remove this prototype stuff and go on with attributes. Now I'm in the middle of realizing what's the next step.

The most annoying part is that I am storing Types via their TypeHandles (actually their RuntimeTypeHandle.Values ToInt32()ed); this basically means creating a type storage via indices; this so reminds me of the database stuff I have going on at work - though, when I started to ramble about entity systems, I kind of expected it.

I had a nice demo going on but it's no more. Raaargh. Here's hoping it won't be long until I have something actually doing its job again.

I'm out.

07 Sept. Edit: Well, it wasn't as bad as I initially thought. I managed to get things up and running again, now including some things like proper component installation and uninstallation. Since my programming life is full of happy gummybears again, I can commit some time to the actual entity system editor. I'll keep you posted.

Entity System and Lua

Lately the Entity System code has been getting more and more complex, to the point that I had to read what I was writing very carefully (and sometimes reading aloud too... go figure).
I have some news about it... about time!

The good news is that it is almost complete! It's a joyful day for me. Object pooling, message dispatching... stuff that works! I still have to finish adding and removing components dynamically (right now we just get the components we defined per entity in an NTL file).

The bad news is that Lua hates me, and it's giving me all kind of exceptions. I think I know what's going on and the relative fix, but if said fix did not work, I'd be in serious troubles. Basically, Lua needs to know what types you're working with, otherwise it would just sit and cry pitifully, making you feel guilty for such carelessness. And it makes sense, of course. LuaInterface is a mysterious world, I still have to figure out thoroughly how the .NET bridging works - luckily the source code is freely available.

At some point of yesterday evening, while bashing my head on the desk for the umpteenth exception, I even felt like writing a Lua compiler / interpreter from scratch in order to get rid of problems when interfacing with the CLR. Sadly, it's a task beyond my available time and most likely experience as well. It would be a great challenge, though.

I still haven't performed any performance tests, but I know in advance I will have to make sure a) the object pool isn't raping the heap b) entity creation is quick and painless, and so is its disposal.

I know it's all a bit vague, but I'll be sure to release the NTL environment as soon as I have something worth attention.

Have a good and productive .NET day! (...rolololflol.)

Shameless plug here

Aloha to everyone reading this blog (hi granma!), I'm now back from holidays and ready for a quick recap about my latest endeavours (whoa).

As if the Entity System weren't enough to keep me entertained and make my fellow team mates doubt about my commitment, I've restarted the long-forgotten UI project. This time it's been redesigned the good way. The good thing of the stuff I'm writing lately is that is it 3d engine-independant, I'm just providing managed Truevision3D implementations since this is what we're using at the moment.

The UI includes / will include the following features:

• Full scripting support
• Multiple views (UI on meshes and on screen, for instance)
• Markup compiler (a la WoW interface)
• Event driven design (in combo with the Input Manager contained in the Rainweaver.Core library)
• Layout Customization, allowing your artists to do their job and snap pieces together

I meant to add Add-on support by default but it's very implementation specific and I'm not sure everyone would use it. It'll be very easy to plug stuff into the interface, though.

I decided to restart the interface project since it's the best environment where to test my libraries (engine, input, scripting, console, data structures etc...).

Stay tuned-zorz.

Away Notice

To everybody who sent me an email in these last days: I'm on holiday, I'll be sure to reply as soon as I come back.

Wien (Vienna) and Budapest are awesome :)

PS: If you plan on going abroad with your trusty laptop, be sure to bring a power adapter.

PPS: I'm away for another week and I won't have an internet connection... take good care everyone and see you soon!

Getting back on track...

Well, these have been very busy days. I managed to rewrite a good part of the Entity System, which now supports object pooling.

I have to brush up my Lua knowledge before implementing message handlers properly.

Here's hoping it won't be too long before I post again with more substantial news.

Of Disappointment

I came to the conclusion that I really wanted to implement object pooling for the Entity System, but I think I've hit two or three (or many more) snags along the way.

The more I try to add this feature to the ES, the more I realize I have to rewrite a good part of the compiler too (make it 80%). I have based the design of the ES upon different assumptions, and now stitching new stuff is proving to be very difficult.

Moreover, I'm not sure all of this will be eventually worth the effort.

So, as much as I'd love to release an ES with an object pool, I don't seem able to pull it off with the current design. It'll have to wait for a new ES version, when perhaps I'll have the time to rewrite it from scratch. This will also mean I'll break the compatibility with assemblies produced with the old compiler.

Bah.

Edit: hell, since I love to rewrite code, I think I'm going to redo the compiler from scratch, this time taking in account everything. I haven't been analysing the problem at hand enough, and that's the result! Don't follow my example! ç_ç/

Entity System (Part 4)

As I said in part 2, the core of this entity system is how messages are dispatched to entities (technically, to their message handlers). Since handlers have a state and a priority flag, it'd make sense to create an ordered list of handlers, and shoot the message down this list. Of course, upon component installation, the order must be kept and it depends on the above flags.

At this point, I don't care if the "message pipe" must be a list, a tree, an array, whatever; I'm worried about the performance of the system, to be precise about allocations.

In a game, especially such as ours, it is common to have many similar enemies which share the same type of components (to keep the es terminology) - we know that all of them, at least, have a Health component. Since I mentioned allocations, the attentive reader might have already understood that I'm about to talk about object pooling.

Why pooling? I find it logical to reuse some objects over and over, since I'm not sure how long they'll live. They might be short-termed (entity insta-kill), or long-lived, or anything in between. We know that GCs kick in post allocations, and that short-lived objects are fast to collect. We also know that caches need a policy, or they're just disguised memory leaks (and if Rico Mariani says so...).
However, in our case scenario, I cannot predict the life of the objects composing an entity (plus the entity itself). Should I just allocate whatever I want and focus my attention on, say, a funny youtube video? Or should I consider creating a pool of reusable objects, instead of allocating without a specific order? Still, allocations are very fast in the CLR.

And the story isn't over, yet. A message pipe contains many references to handlers. And there will be x message pipes equal to the number of entities times the number of system messages, each of which will contain y references to y different handlers. It's a lot of references to be tracked by the garbage collection, and I'm scared to just think what'd happen if these references kept some objects alive instead of letting the GC collect them.

My solution (and it's far from being the right one, I suppose) would be to pool stuff, even if I have just advised against it in a recent gamedev.net post. With pooling, I have collections of objects ready to be used or already used, and I can access them via an integer "handle" (namely an index). All my message pipes would contain is a list of these integers - thus no actual pointers but indices. Would my garbage collector be happy? Am I wasting my time? I tend to think the latter, since I haven't managed to make up my mind about this matter yet.

I gotta say I find many .net perf advices a bit obscure.

There's also something else which has kept my mind a bit busy between a general ledger entry table and a C/AL codeunit this morning (brrrrr!). Should I suggest the GC to perform a collection when, say, the user reloads the level or changes map? I'd say so, but it isn't necessarily right. I must remember to test this as well at some point...

Premature optimization, anyone?

.NETAsm

You HAVE to check this out. Sounds bloody amazing to me.

Entity System (Part 3)

Well, the compiler looks almost done; a ntl file in, a .net assembly out. I'm sure there will be a few bugs to iron out, but it conforts me to just think I'm not too far from its release.

The next step will be releasing an editor for the language; for now, I think I'm going to use Roger Alsing's Syntaxbox and some other stuff found in the net.

Another aspect worth discussing is that the Entity System will work on its own thread, and it will communicate with the main thread via the so-called "action gateway". Every action sent to the ES is queued and executed serially; the inverse is alike, however the main thread has to take care of serializing the actions the ES has sent and execute them after whatever comes first (usually rendering and physics update).

Now, in my mind this looks like a good plan, but I know it's bound to change somehow. I can't wait to have some working results to show.

Good weekend, Raine out (nerdish goodbye).

Part 4 is here.

Tree awesomeness

When you have an artist doing stuff like this...


Awesomeness rating: 100%

Well, check out our website if you're interested in how the tools are coming along. :)

Async Resource Loader

A few months ago I had the map loading stuff done, and wanted to try some landscape paging. I thought a worker thread would save my day but, as it turned out, loading stuff from HD was still blocking the rendering. I took a look around and found out that synchronous calls (even in a different thread) wouldn't act any better than if they were in the main (rendering) thread.

Real life then set in, and I couldn't get much done. Some weeks ago I rewrote the async loader, and this time it'd include asynchronous calls. So, a worker thread + asynchronous calls (Begin... / End... methods). I soon realized a bad design mistake and I wiped the folder (lesson learned: don't code anything until you've filled at least a page of a good old paper notebook).

I'm going to write it again before implementing anything else since I want all loads to be performed without rendering hiccups (or at least, as few as possible, since the OS has the final say on everything). So, since it's to be rewritten, I'll take more time to design it carefully.

The ARL (async resource loader, blah blah blah) allows you to use custom loaders (since all it does, on its own, is to load a stream of bytes asynchronously from somewhere). Once you have added at least a loader, you start the "service" and call the Load(ResourceInfo) method wherever you like (provided it's in the main thread). Once it's ready, you get a Resource object back (from a callback, I guess). That's about it.

There are details to consider, however; load requests have to be queued, can be aborted, the object which does the final loading (that is, converting the stream of bytes into an useable resource) must be locked (both inside the custom loader and outside), and last but not least, it really has to be threadsafe and performing, otherwise it's useless.

That's all for the moment, ran out of inspiration....

Entity System (Part 2)

It's Friday, I guess I'm going to keep rambling a bit.

I came up with a simple declarative language, which happens to be very similar to IL (I like the leading full stops) to ease the editing of entities, components and system messages (simply messages from now on):

// build options
.assembly entities.dll;
.namespace EntitySystem;
.debug true;

// declarations
.component HealthComponent
{
// .flags
.field System.Int32 Health;

.handler DamageReceived
{
.priority normal;
.state impl;

// .extern scripts/ondamagereceived.lua;
.body
{
--lua code here
}
}

}

.message DamageReceived
{
.field System.Int32 Damage;
}

.entity Player
{
.components
{
HealthComponent
}
}

The good news is that the parser works; the bad news is that the first version of the compiler refused to work properly (for some unknown reasons CreateType complained some methods didn't have any implementation).

How does it work? Basically, you can define some predefined entities by listing their components. Components have fields and handlers (message listeners) and messages have fields in order to contain the data they carry (what else? :) ).
Handlers have two special members: the .priority and .state declarators. These things specify in which order the handler is installed into the Entity. You cannot specify an index, rather the Entity System inserts these handlers in a stack fashion in the right place.

.priority allows values such as "SystemHighPriority", "SuperHighPriority", "HighPriority", "NormalPriority" etc, whereas .state allows only "pre", "impl", "post". I have to thank the good Mr John Brewer for this idea, which is basically the core of how the message dispatching works.
You can send messages from one entity to another, or from one entity to a family of entities (which is defined by the flags field, which is made to containthe numerical value behind OR'ed enum values).

I'll be releasing the compiler (and probably the source code) as soon as it's completed. This will be plugged into newborn's StoryED in order to allow further customization. You compile the NTL (eNTity definition Language), you reference the produced assembly and the ES assembly into your application and you're done. End.

I like this stuff. What do you think?

Part three is here.

Entity System (Part 1)

Long blog post incoming. And it's, like, my third one.

Entity
1596, from M.L. entitatem (nom. entitas), from L. ens (gen. entis), proposed by Caesar as prp. of esse "be" (see is), to render Gk. philosophical term to on "that which is."

System
1619, "the whole creation, the universe," from L.L. systema "an arrangement, system," from Gk. systema "organized whole, body," from syn- "together" + root of histanai "cause to stand" from PIE base *sta- "to stand" (see stet). Meaning "set of correlated principles, facts, ideas, etc." first recorded 1638. Meaning "animal body as an organized whole, sum of the vital processes in an organism" is recorded from 1683; hence fig. phrase to get (something) out of one's system (1900). Computer sense of "group of related programs" is recorded from 1963. All systems go (1962) is from U.S. space program.

http://www.etymonline.com/

There are posts around the net about entity systems. Not sure there's a particular implementation to follow. I've been reading some related threads over at gamedev.net, but none of them particularly stirred my interest either.

Entity systems are made to favour aggregation over inheritance. There are reasons to use the former, and reasons to use the latter. I'm not going to say one is better than the other. The right tool for the right blah blah blah.

I'll go the Entity System route with Darkmana, mostly because - provided my implementation will work as intended - it'll allow me to define custom behaviours with little effort.

Imagine for a moment your main character is standing near a bonfire, ready with his composite bow. The wannabe game designer in me thinks: wouldn't it be cool to put the arrow on fire by standing close to the bonfire? This means the next shot will also do some fire damage. Cool stuff, yes.

Now, what am I going to do to implement is? What's the way? Many a way. Logically speaking, my arrow is a simple arrow at first (it has some default properties), and then I add some other attributes; that is, it is on fire. From a code standpoint, I have my Arrow object, probably derived from a Missile : GameObject class. Now, it's a FieryArrow object, derived from the Arrow class. So basically, create the new one (probably with a constructor accepting the old Arrow object?) and use it.

I could go on forever (in fact I was about to, I just deleted a good bunch of lines worth of examples), but the problem still remains. Whatever you do, it's hardcoded AND stuck into a specific object hierarchy, which you can't free yourself from.

Scripting comes in rather handy. Well, you still have the fixed hierarchy issue. Why is it an issue? From Cowboy Programming:

"The result of implementing functionality near the root of the hierarchy is an overburdening of the leaf objects with unneeded functionality. However, the opposite method of implementing the functionality in the leaf nodes can also have unfortunate consequence. Functionality now becomes compartmentalized, so that only the objects specifically programmed for that particular functionality can use it. Programmers often duplicate code to mirror functionality already implemented in a different object. Eventually messy re-factoring is required by re-structuring the class hierarchy to move and combine functionality." (http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy/)

Let's get rid of it, shall we? How am I going to define my game objects now? Here's my approach.

The rationale behind removing hierarchies is that entities are made of components, pretty much like a body is made of organs performing different stuff. An entity in itself isn't much; it's the way it reacts to the surrounding system rules that matters; and the way it reacts depend on what composes it.

So, now I'm going to have an object which allows me to add other objects, and these objects will take care of defining the way the owner reacts to some system messages.

I mentioned system rules. They are indipendent from the entities. Entities act into a system with a predefined set of rules, rules which they cannot change (we're talking about virtual environments and not politics) - basically the boundaries set by the game designers.

I'm going to post some more later. I need a cigarette and the lunch break's almost over. Let me know what you think.

Part two is here.

Darkmana

Almost one year ago I sent an email to newborn (Vincent, aka the Captain) asking him if I could join the team as the UI programmer. And here I am, now, trying to figure out the best way to program the whole game. Oooh I like the challenge.

It's not like I have done too much during this year, to be honest. Vincent's been in the spotlight all along, and rightfully so, because of his superdupertastic job with the tools he developed.

But I've been lurking... and researching. I've spent the time mostly designing and trying out many interesting things.

I've recently started to rewrite my own libraries, which ideally should ease the making of the game. I'll be posting about the Entity System and the Async Resource Loader design in the next days, since I think they are interesting topics.

First post

Sooo hi 2 u all. I think I'll use this blog to write and ramble about the development of Darkmana.

See you later, I'm at work now.

PS: Who's "you"? Nobody knows about this blog. oO