Took me a while

In order to keep up the good tradition of this blog, famous for its quality content, I had to meditate for months.

Work, as usual, has not been forgiving with my spare time. Many things also happen outside the event horizon of a computer screen, and it eventually becomes increasingly hard to clear up thoughts and write down stuff.

Blabbering aside, I've managed to finally write a working prototype of the famous Entity System. Not only that, though. The Rainweaver Framework (guess I'll stick to this name for the time being) has been given lots of love as well, and despite my constant refactoring, I can say I'm 80% satisfied with the result. I made a post at Gamedev.net but stirred no interest. Ah well. For fun and not for fame, right?

This Framework of mine is a hobby project. I'm giving myself the chance to learn new technologies such as WPF as well. It has a steep learning curve. It's true. If you want to learn WPF, you gotta have the time to. Time to sit down, open two Visual Studio instances, download Family.Show and get dissecting. Time to wade through thousands of tutorials looking for best practices with a dim light.

With this spirit, I decided to take up a WPF project related to the Entity System. It's called Prototyper. You can still see in the changelogs that the project was named ArchProto, then ProtoArch, it turned WPF from WinForms (just like growing adult), and got its final name.

Project conception aside, I'll explain briefly what it's supposed to do.

Every Entity System has a Schema and a Runtime. The first says how things work, the second makes them happen. Between them, there are Runtime Globals and Scripts. Runtime Globals are implementation-specific methods or properties exposed by the runtime to the schema. Scripts can be modified at runtime (atomically, i.e. no undefined state across modifications) and are generally kept in some kind of storage (files, db).

Prototyper is being written to allow designers to define components, messages and prototypes in a visual fashion; associate a Runtime Globals type for scripts; associate scripts from a typed script storage object to message handlers; compile a schema to CLR Objects.

The Entity System was a challenge and still is, as it's not 100% done. But a bigger challenge lies ahead with this Prototyper thing.

If you feel brave, head over at http://rainweaver.codeplex.com/ and download the latest code drop, play with it, examine the code.

Thanks for reading.

Bye-bye!

Parallelism, Scalability, Persistence - Linkage

Thanks to a bad neck, I finally have some free time. Why not spam the blog, I thought (and why not try Windows 7 on a VM, zomg it's amazing).

While searching for cool programming stuff, I found some interesting links I have to share with you. Just two for the moment! Don't be greedy.
Smoke framework, for starters. The good thing is that what I had in mind is scarily similar to what Intel done, and this makes me a happy gummybear.
Retlang, by Mike Rettig, which seems to be the answer to my concerns about thread-safe message passing.
(And "Stackless C#", by Tim Mcfarlane, which has led me to try and learn how Irony works in order to recreate a Lua CLR compiler that supports coroutines without fugly hacks - but that better be another story for another post).

While those two links lead to resources that can only inspire the mere mortals, they also made me remove my Google Code page until I completely embrace their paradigms. That's to say I wasn't exactly happy with what I'd done and I wanted a fresh start. I think by now you noticed my big problem with finishing my own projects.

Now, you might be wondering, what's this to do with the post subject? Very well. Good question.

Smoke shows an important concept: that is to share the workload among n threads. However, as usual, it is easier said than done. Sharing the workload requires a careful design, so that nothing is left to compromise and everything falls nicely in place when all computations are done.

However, one of the biggest challenges is to actually find good heuristics so that work is spread evenly across threads. You don't want to waste time counting those threaded sheeps jump past the fence. The other is to allow an immediate flow from producers to consumers.

I have a good example of the mental maze you find yourself in once you try to make sense of the above; your rendering thread is told to render something you haven't yet loaded. An animated mesh, for instance, along with its textures and animations and whatnots.
If the rendering waited idly for that asset to be loaded, we'd be back to square one. It becomes difficult to remove dependencies from systems that'd naturally rely on them. What would the pragmatic developer do? Perhaps identify each resource with an ID, and check against that to verify if the asset has been loaded. However, spamming IDs all over the place lacks of elegance, and the initial problem leads to another sub-problem; how to organize data.

There are two kinds of data I can see right now; in-memory data, and assets to be eventually loaded in some memory (they're initially stored on disk, any type of assets, from textures to ai-behaviour-definition files, the latter being just made up).

Both have a stage in which they are not utilizable, as they're to be either created or loaded. And even when you've loaded data, there's another possible step, that is going back to storage. Persistence.

All of this needs a common interface: find necessary data, load it, access it, save it again. Rinse and repeat. Add two tablespoons of sugar. Did I mention data has to be immutable? My head is hurting.

Oh, damn. Lunch time. I was about to make a nice diagram. I'll continue later. >:)

Edit: nice diagram (click to enlarge):

A Rain Song: now with nice pictures!

Why two servers with the same processing units? Our ultimate goal is to plug one more server and see performance double instantly. More or less. The same goes for three servers, and so on.

More on this later, car's getting brake pads changed and I need to take it back.

Parallelism, Scalability, Persistence

In order to not let the blog wither, I think I'll post some notes I took during my stay in Milan. I won't bother you with the details of these past months, as I've been beyond busy... perhaps super busy. I've been working on a data relay system for an international company (first big project, yay!), and I don't feel like enumerating the crapload of snags I hit along the way. I've been using WCF, for the curious. Nice technology, if confusing at first like most MS frameworks.
Anyway, there we go. I'd love to see some feedback, if any.

--

Parallelism means performing many operations with none waiting on another.

Scalability means opening up parallelism over separated but cooperating processing units.

Persistence means the capability of a state to be saved and restored in its entirety
at any point in time.

In order to achieve parallelism, data must be immutable, there must be as little data contention as possible, and no locking, as a corollary of the previous statement. In order to achieve scalability, operations must be serializable, and so their results, so that they can be shared across different parallel processing units. The cost of sending data across a communication channel must be less than the sum of the processing costs on a processing unit; this implies fast communications and smart work-stealing heuristics.

In order to achieve persistence, the selected storage must be able to create a perfect copy of the data being sent to it, save it in a lossless way and retrieve it later; this sequence must be able to happen at any point in time to prevent sudden data loss.

--

The data upon which operations are performed and the operation themselves must be local, that is they must be in the same processing unit. Whenever a state* happens to be partially shared across two different processing units, the one with less workload will receive a full copy of the necessary data and carry out the involved operations. This can be seen as an implementation of the concept of "ghosting".

* A state is a collection of operations to be performed towards a specific result, along
with their determinant data.

Data can also be proactively shared with processing units that don't yet need it. Sharing in this case means transfering a clone of the data until the data is either not needed anymore or completely trusted to another processing unit. Proactive sharing might happen when data gets close to the processing unit boundaries, which can be both abstract or physical (different machines linked by a connection). Data should be sent only when it changes.

I hope I'll be able to post more.

Oooh.

My best virtual croatian friend Joe Basic and I have opened a Google Code page.

http://code.google.com/p/rainweaver/

The Rainweaver Framework (it's a codename, whatever) is a set of libraries for game developers. You can read about our evil plans on the first page. There's something you can download, as well. Let us know what you think, of course!

If you know C# and you're the reliable type, consider joining us. We're busy with work, university, and all the rest, so it's a long term project.

Thanks for reading.