In order to not let the blog wither, I think I'll post some notes I took during my stay in Milan. I won't bother you with the details of these past months, as I've been beyond busy... perhaps super busy. I've been working on a data relay system for an international company (first big project, yay!), and I don't feel like enumerating the crapload of snags I hit along the way. I've been using WCF, for the curious. Nice technology, if confusing at first like most MS frameworks.
Anyway, there we go. I'd love to see some feedback, if any.
--
Parallelism means performing many operations with none waiting on another.
Scalability means opening up parallelism over separated but cooperating processing units.
Persistence means the capability of a state to be saved and restored in its entirety
at any point in time.
In order to achieve parallelism, data must be immutable, there must be as little data contention as possible, and no locking, as a corollary of the previous statement. In order to achieve scalability, operations must be serializable, and so their results, so that they can be shared across different parallel processing units. The cost of sending data across a communication channel must be less than the sum of the processing costs on a processing unit; this implies fast communications and smart work-stealing heuristics.
In order to achieve persistence, the selected storage must be able to create a perfect copy of the data being sent to it, save it in a lossless way and retrieve it later; this sequence must be able to happen at any point in time to prevent sudden data loss.
--
The data upon which operations are performed and the operation themselves must be local, that is they must be in the same processing unit. Whenever a state* happens to be partially shared across two different processing units, the one with less workload will receive a full copy of the necessary data and carry out the involved operations. This can be seen as an implementation of the concept of "ghosting".
* A state is a collection of operations to be performed towards a specific result, along
with their determinant data.
Data can also be proactively shared with processing units that don't yet need it. Sharing in this case means transfering a clone of the data until the data is either not needed anymore or completely trusted to another processing unit. Proactive sharing might happen when data gets close to the processing unit boundaries, which can be both abstract or physical (different machines linked by a connection). Data should be sent only when it changes.
I hope I'll be able to post more.
