Replicate this !!!

Before we start, I installed the SMASH framework on a Linux machine and boy are those sockets fast, the message sync error commented in the post below does not occur, curious, latency speed gain 3:1 over Windows sockets, I am impressed.

But that is no the main topic of today, I have gone over the edge and I am currently programming a 2nd Key Value Server (KVS2). The original KVS server is a Key Value (or Name server), that caches function call results for x amount of time recaching every n seconds, the KVS2 is different. KVS2 is designed to be a database, aka mnesia, replacement.

Mnesia is a very cool design and I like it, I especially love the fact that f.e. RAM nodes can access the DB just by declaring where the master DB is, you can fragment a DB over several nodes, it has transaction capabilities and it can reside on several nodes. If you have a lazy look up application that is awesome, but I found in real life that usually the different nodes cannot keep up with transaction volume of a MMO framework in absolutely no way, if I create one master node then I face the challenge that when a slave node does a transaction it can take up to several seconds to get a result back and if the master goes down, well, so does the whole system, if you define several masters, synchronization takes too long, way to long and if one node goes down the other starts complaining and blows up.

So here comes the salvation, I hope, the birth of KVS2. The code is distributed through the LB to the code and auto started. The first node in the system reads its data from text files and other slave nodes copy the tables over, each table runs as a named process. When you change or delete data that node will issue a broadcast to the other nodes to do the same transaction, hence they sync and each node enjoys having a local cache, speed, speed, speed. If the master goes down, big deal, the other KVS2 servers are linked to the master and detect its death, so they choose a new one, so far they pick the new master simply by name, I could also very easily ask the LB and cache with KVS the value, so that they all get the same answer, but ehm, I don't. This node is then declared the new master and end of story. The master saves the data every 5 min to disk and you can even force a node to be the master, like when the original comes back up. A word of warning here, if several nodes write to the same record, you can have confusing results, as there is no check that all nodes have the same data, if messages for the same record arrive in different order, you can have different records on each node, be careful, this general scheme assumes that only one process manages a certain record, like for example in SMASH, only supervisors write records. Yet when accuracy is critical, it is very easy to send a message to the master node, the rpc interface handles that, so it can send it to the master and the master will then send it around, an easy way to ensure transactions or uniqueness without conflicting records. Simple, short and powerful. Obviously this still needs to be tweaked and debugged, but it is already working. I will later on integrate a time stamp to enable servers to sync data if someone has more recent data, but for the time being I believe this new scheme solves my database requirements nicely, I have a local cache, self replicating data and a system that will work until the last node is shut down, removing all mnesia draw backs.

I will also integrate, somehow, preferred servers to be master nodes, which should make it easier to know where the files get saved to. The keen observer may have noticed, that we lose any kind of indexes and filters, for the time being. Filters are easy to implement actually given the [X || {Y,X} <- List] functionality, normally we know the key "Y" and get "X" as the value, but we can filter like this:

Value=[Pid || {Pid,Node} <- Answers, Node=:=Filter]

grab the PID from a tuple, when we only know the node name passed to the function as "Filter". Here we grab the value as the search value.

And there will be no indexes, it is cheaper to define a small routine that creates a separate table instead. So, a couple more days until I have the KVS2 debugged and then off to replace mnesia completely with calls to KVS2, I have to replace something like 100 calls, quite doable actually.

And then I will implement the authorization scheme, urgh !!!

Cheers,

Sunweaver

P.D. Debugging is almost done, all possible things seem to be accounted for, given the intrinsic structure of KVS and KVS2, they work like a short and long term memory, hmm I might call them both in the future Skynet Database, sounds better than Key Value Server, doesn't it ??

There is always another way

"There's an Italian painter, named Carlotti, and he uh, ahem, defined beauty. He said it was the summation of the parts working together in such a way that nothing needed to be added, taken away or altered [...]" Cris Johnson (Nicholas Cage) in Next

Good thing he did not talk about our code because each time I look at it I find another way of doing things, Erlang is really much faster when it matches patterns rather than using traditional programmatic code, so perfection or beauty is still a long way down the road, but anyway.

RTFM or ... how I should really read the manual more often, I had completely overlooked the function net_kernel:monitor_nodes(true) . When you subscribe to this function, in our case the load balancer (LB) does as of now, you receive a message {nodeup,Node} the instance a new node comes up so now the code gets copied and started instantly on new nodes, it couldn't be faster than this, if you blinked you just missed the copy+start process. A message of {nodedown,Node} tells you when one disconnects.

I also stumbled upon another very useful function called rpc:sbcast(Name,Msg) which sends a message across all connected nodes to a registered named process. Until now the group process caches a list of all groups on the different nodes and new groups can take a while (up to 20 sec. so far) to be seen by others nodes, but given that we now register each group on each node, this function could speed this process up, without relying on cached mnesia data, my only questions is, what happens when 2 nodes do not yet see each other ??

Speed-Ups: Every time I look at pattern matching I learn an easier way to do something, when I first started I was tempted to use traditional loops like the function lists:foreach then I switched to:

lookup(G, [{G,Pid}|_]) -> [Pid];
lookup(G, [_|T]) -> lookup(G, T);
lookup(_,[]) -> [].

I find that structures like the one here are even faster than the other two:

[X || {Y,X} <- L] where X is our return value , L our list to look up a value from and Y our variable to pick the correct tuple from a list. This function is so fast that it is causing me some synchronization problems among my processes for reasons yet unknown. This function will also serve as a command interpreter, when the client sends a string of commands, like "{cmd1:arguments}{cmd2:arguments}", I convert that list with a function I designed into a list like this [{cmd1,"arguments},{cmd2,"arguments"}] and I can then use the above [fun(Y,X) || {Y,X} <- L] to run some function on each element in the list. I might even displace Mnesia altogether, just load data in lists into the KVS, look up / filter values with pattern matching and sync the nodes with rpc:sbcast. I could use the module dict to create a dictionary instead of a table or list, too. Each dictionary could run in its own thread, hmm interesting. This sounds so crazy, it might actually work.

Remember that Mnesia is nice when speed is not critical, but when you need to do thousands of look ups per second across "n" nodes then the generated overhead is just too much, mnesia can't keep up, no matter how I structure it, be it several disk nodes, 1 disk node several ram nodes, the worst thing is the lack of a local cache on a ram node, hence the KVS working as a local cache. From a mnesia centric application framework I am getting more and more towards a KVS centered data structure, the above scheme might eliminate the caching errors and I do not require really the supertight transaction property that Mnesia offers, I'll gladly trade that for speed.

Designing the authorization module (AUT): it will mean a large impact, because I will need to define a whole lot of logic around this, including what commands each user can perform, guild channels, raid & instance channel, guild structures and more, so this will take some time to complete, but the design isn't finished yet, I am still prototyping the logic. Some nice features that will most likely come up are f.e. you will be able to define as many guild channels as you like and define their authorizations to differentiate the general channel from officer, class lead, raid or other group channels below the guild. I will also consider instances and raid channels with ownerships, to prevent problems like an invited outsider stealing raid IDs, a documented problem that happened to other large MMOs. Under the devised scheme an invitee will not have ownership or admin rights and it should be an easy thing to do that certain instances must be started by the owner/admins and hence avoid getting a started raid instance stolen. The user commands wil also be important to avoid that a guild master leaves the guild without reassigning ownership to somebody else, so a GM will lack a command of GQUIT, but have a GOWN, while other members will have GQUIT, but none afiliated will have the GJOIN command sombody unafiliated will have. The authorization module will need to combine speed with functionality so a huge fun part is coming up here.

With the speedups like the ones above, I am now facing sync problems between threads (that should not exist on the first place), the first or first few messages from a client get lost in cyberspace, I know that this is due to me cacheing values from mnesia, so I am thinking that the easiest way to avoid lost messages will be to implement a ping-pong protocol to allow the client to ping the server (once authenticated) until he gets a pong response and only then start asking the server for more stuff or start chatting. The cacheing delay can be anything from unnoticeable to several seconds.

Another curious fact is that the more functionality I design the smaller the modules get.
The module CS which was in the beginning the master of the whole framework is becoming less and less important, with groups registering now, it is not much more than central resource locater/creator. The CG module is now the local node relay station, down from channel admin, the logic has gone to the channel NPC. Both are still very much required, but their role has drastically changed during development, curious.

Cheers,

Sunweaver

Some more remote utilities

Some might know that SMASH has included into its belly an automatic updating system, which allows to compile all modules and reload them on all connected nodes in run time, without ever shutting the system down. On occasions this fails and a certain service shuts down. To get a hang on this, we have created some utility routines, one permits to execute commands remotely (on the other nodes) and the other shows if any module is executing old code, so very soon we will be able to first check what module on which node is executing old code before a new compile is allowed. So far the problem seems only to reside on longer lasting timers, which are not included in the update signal, all others update correctly (it seems). Also with these utility routines we could now easily start all preconfigured services (in the cluster file) remotely, without needing to touch then anymore. Some other fixes are that the CC proxy can now send tells or whispers without subscribing to any channel; KVS can now cache and forget, when you ask kvs:cache(M,F,A,Timer) it now will update every "Timer" ms and forget about the value after a minute, so the dictionary does not pile up with hundreds of unused values, this works like a garbage collection to keep KVS trim and slim. And more tests on Mnesia have been executed and curious enough on remote nodes the dirty function is slower than the normal database access with transactions and as to be expected KVS suffers from the initial Mnesia access, but afterwards blazes thru repeated accesses, so no matter how much we try to optimize the database access, the cached behind KVS approach is still the best when repeated access to data is required. Dedicated MMO literature also comes to the conclusion that processes on a multi server structure should update their data every 5 sec., less seems to be too much. And the cloud infrastucture has given some new ideas, Erlang already has the ability to spawn remote processes, so I guess, we just might implement our own flavour of it, by speed measuring each connected node and on the other hand check how many processes are already running there, this could be something like a routine task that executes per "X" interval and then when asked to spawn a process it checks for the best node to do so, a kind of "cloud_spawn(M,F,A)". This would make the SMASH ready to process any kind of task actually, not just a specialized Simulation Framework. It just might happen =) .

And Apocalyx can indeed use tables to store toon data, I played around with the Urban Tactics demo and it can easily handle some 200 toons running around there, very nice, now we need this networked. ;)

Laters,

Sunweaver

48 hours of endurance

Alright, not much to report for now, except for some minor changes on the Observer side, which did not catch all required objects. And the channel management requires a major overhaul, whisper commands currently trigger being subscribed to that channel, which isn't exactly desirable, meaning that we will require authorization management, ouch. That's a major CG (chat group controller) rewrite, but anyway.

And now to the 48 hours test: yes the server holds up. The caching Key Value Server (KVS) makes the whole thing fly and no messages pile up, things have never been so fast, it does not matter where a client connects to, the SMASH framework relays the messages blazingly fast to other nodes, so we know by now. This concludes stability and speed tests for the time being.

Leo, if you are reading this, I sure hope that Apocalyx permits me to create each 3D object as a table entry and not just as a variable, like your examples showing variables like avatar0 - avatar6 and so on. I will be needing to create a LUA table and put each avatar as one entry and then update that table. I guess once your new Gun Tactyx is out I might try to "socket" it and permit people to connect to a fighting event, we'll see, so now back to learning Apocalyx. And yes you should be able to see some kind of graph here soon, to reflect what we got so far.

Stay tuned,

Sunweaver
EDIT: I shortened the post.

And there shall be speed

It seems that the new caching KVS server does miracles, the message throughput of the whole framework went through the roof, up from a maximum of 7000 messages per second (averaging 2500 at heavy load) to a minimum of 150'000 messages, that's a 2100% increase, not bad. So, now we are indeed ready for the next step, Apocalyx here we come, the next thing will be to visualize our first GUI.

Sunweaver

Need for speed

When doing the first message bombardment on the server and the news are mixed ones. On the one hand the server does not crash, which is nice, but after a while it starts to behave irrationally, like when logging out and logging back in, it does not seem to delete users from the DB anymore, so a kind of measurement is needed. Writing some quick benchmarks it turns out that the Chat Server (CS) can do 600'000 messages per second (mps) and the DB lookup in the chat group can only do about 6'800 mps, with 16 clients connected we get up to 2'600 mps, but I suppose that the constant bombardment becomes a problem at some point, because Erlang starts piling up messages and the missing client deletion from the DB is most likely the same cause, so a first conclusion is that Mnesia can't keep up with all those lookups on the DB and this causes messages to pile up like crazy. Fortunately this was likely to happen and now we have a legal excuse to use the KVS (Key Value Server), which has been completely dormant until now. The KVS will need to grab this kind of data from Mnesia and put it in a local table. The big difference now will be this, the KVS will only load the data from Mnesia every "x" time intervals, like maybe every second or every 5, meaning that we will lose precision in order to gain speed. We will now have to segregate our operations, those that require ultimate precision like close by moveable objects, banks, auction houses, trades and loot operations and others like chat, or even general environment operations that do not require the most precision. And we need to start with the Chat Group structure. This should be really easy to do, we will see if things speed up afterwards. A general throughput of 500'000 mps looks like our boy, but 7'000 mps does not.

And surprise, surprise, the Playstation 3 is not nearly as powerful as hoped for as a server, it's depending on the test 4-10 slower than a Dual core Windows Laptop, averaging so far like 6x slower, how very disappointing. In the CS test it scores at 95'000 mps, the DB lookup only gets 4'400 mps and it takes 24 sec. to create 10'000 CC clients, while the Laptop does that job in 7 sec., OUCH !!!!!

Long live the benchmarks.

Sunweaver