I have extensively tested the stability of KVS2 and I am almost satisfied. The infrastructure is rather difficult to kill now, which is a nice thing.
All data written now stores when the last update happened to it and when the data is due to expire, to enable short term memory data functionality and in the future sync'ing data among master nodes. Now let's invent a new term, each database has a certain foundation and principles it's based on, mnesia f.e. defines that as the ACID properties, eg. atomicity, consistency, isolation and durability. Given the design choice of KVS2 we define:
The WORM principle: or "write once, read many"
Only one single process writes a record, every other process can read from it. If several processes write to the same record you will have at some point inconsistent data, given the lack of write locks and random order writes.
It would be more appropriate to call it " one writes, many read", but who can remember the acronym OWMR ??
So, f.e. loot data must be implemented on a zone level and hence be managed through the "zone player" process and auction house handling must be implemented as if it was a zone, WORM for the win.
If we apply the WORM principle to our programming style, in theory, we preserve the ACID property that mnesia enforces on application level and we add an "S" for speed to ACID, becoming ACIDS, because mnesia fails to live up to the part where it should be real-time.
Writing a distributed database is a lot of work and fun. I wonder, how long will I be able to do without write locks ?? Hmmm !!
1) Make sure that a newly elected master node is really running --> with a separate process on each node that verifies the masters list, its status and trigger a reelection process when needed
2) During start up, ping all previously know nodes, try to find running instances there, get more node names, ping them and only then start up or else you get several masters elected