Some more remote utilities

Some might know that SMASH has included into its belly an automatic updating system, which allows to compile all modules and reload them on all connected nodes in run time, without ever shutting the system down. On occasions this fails and a certain service shuts down. To get a hang on this, we have created some utility routines, one permits to execute commands remotely (on the other nodes) and the other shows if any module is executing old code, so very soon we will be able to first check what module on which node is executing old code before a new compile is allowed. So far the problem seems only to reside on longer lasting timers, which are not included in the update signal, all others update correctly (it seems). Also with these utility routines we could now easily start all preconfigured services (in the cluster file) remotely, without needing to touch then anymore. Some other fixes are that the CC proxy can now send tells or whispers without subscribing to any channel; KVS can now cache and forget, when you ask kvs:cache(M,F,A,Timer) it now will update every "Timer" ms and forget about the value after a minute, so the dictionary does not pile up with hundreds of unused values, this works like a garbage collection to keep KVS trim and slim. And more tests on Mnesia have been executed and curious enough on remote nodes the dirty function is slower than the normal database access with transactions and as to be expected KVS suffers from the initial Mnesia access, but afterwards blazes thru repeated accesses, so no matter how much we try to optimize the database access, the cached behind KVS approach is still the best when repeated access to data is required. Dedicated MMO literature also comes to the conclusion that processes on a multi server structure should update their data every 5 sec., less seems to be too much. And the cloud infrastucture has given some new ideas, Erlang already has the ability to spawn remote processes, so I guess, we just might implement our own flavour of it, by speed measuring each connected node and on the other hand check how many processes are already running there, this could be something like a routine task that executes per "X" interval and then when asked to spawn a process it checks for the best node to do so, a kind of "cloud_spawn(M,F,A)". This would make the SMASH ready to process any kind of task actually, not just a specialized Simulation Framework. It just might happen =) .

And Apocalyx can indeed use tables to store toon data, I played around with the Urban Tactics demo and it can easily handle some 200 toons running around there, very nice, now we need this networked. ;)

Laters,

Sunweaver

And there shall be speed

It seems that the new caching KVS server does miracles, the message throughput of the whole framework went through the roof, up from a maximum of 7000 messages per second (averaging 2500 at heavy load) to a minimum of 150'000 messages, that's a 2100% increase, not bad. So, now we are indeed ready for the next step, Apocalyx here we come, the next thing will be to visualize our first GUI.

Sunweaver

Need for speed

When doing the first message bombardment on the server and the news are mixed ones. On the one hand the server does not crash, which is nice, but after a while it starts to behave irrationally, like when logging out and logging back in, it does not seem to delete users from the DB anymore, so a kind of measurement is needed. Writing some quick benchmarks it turns out that the Chat Server (CS) can do 600'000 messages per second (mps) and the DB lookup in the chat group can only do about 6'800 mps, with 16 clients connected we get up to 2'600 mps, but I suppose that the constant bombardment becomes a problem at some point, because Erlang starts piling up messages and the missing client deletion from the DB is most likely the same cause, so a first conclusion is that Mnesia can't keep up with all those lookups on the DB and this causes messages to pile up like crazy. Fortunately this was likely to happen and now we have a legal excuse to use the KVS (Key Value Server), which has been completely dormant until now. The KVS will need to grab this kind of data from Mnesia and put it in a local table. The big difference now will be this, the KVS will only load the data from Mnesia every "x" time intervals, like maybe every second or every 5, meaning that we will lose precision in order to gain speed. We will now have to segregate our operations, those that require ultimate precision like close by moveable objects, banks, auction houses, trades and loot operations and others like chat, or even general environment operations that do not require the most precision. And we need to start with the Chat Group structure. This should be really easy to do, we will see if things speed up afterwards. A general throughput of 500'000 mps looks like our boy, but 7'000 mps does not.

And surprise, surprise, the Playstation 3 is not nearly as powerful as hoped for as a server, it's depending on the test 4-10 slower than a Dual core Windows Laptop, averaging so far like 6x slower, how very disappointing. In the CS test it scores at 95'000 mps, the DB lookup only gets 4'400 mps and it takes 24 sec. to create 10'000 CC clients, while the Laptop does that job in 7 sec., OUCH !!!!!

Long live the benchmarks.

Sunweaver

Tuning to perform

The initial performance is being tweaked, while not bad we feel, that more can be done here regardng message passing and some first strange errors have appeared, no showstoppers, but something to deal with.

Also under current analysis is how to proceed with zones and where players show up, should one player appear in the next zone when he is close to the next one, that would mean like overlapping zones and if a player is in the center he only reports his position into one if he runs towards a corner he would potentially run into up to 4 overlapping zones ... or is it better to make smaller zones and always send the coordinates into all surrounding zones, which would mean always to transmit into 9 zones. Zones could also be done through range lists and handle everything like one large zone. In order to determine this, Apocalyx will be required, so next steps must include a graphical frontend already. More coming soon.

Work in progress (WIP) of the To-Do List:
* Create an action plug in template [easy / pending]
* Internal user registration (to avoid same user on several nodes) [easy / partially done]
* Cluster behavior [medium / pending]
* Zone Simulation Manager [medium / WIP]
* TPC/IP server with authentication [medium / WIP] + protocol converter [medium / WIP]
* TCP/IP test client [medium / WIP]
* Apocalyx integration [medium / pending]
* Master node [hard / WIP]
* NPC scripts [hard / pending]
* Subscription Server [hard / partially done]
* Interrealm connector [hard / pending]

Sunweaver

First benchmarks

We have conducted the first couple of test series on a single-core notebook (sc), a dual-core notebook (dc) and the PS3, where creating 10'000 users takes 1:11 min., 0:22 min and 0:42 min respectiviely. We then proceeded to send chat messages between the 3 systems and the results are similar in ratio. What is very apparent is that the PS3 displays the text relatively slowly, so in my opinion given that the CEL processor should be very superior to the dual-core it seems that the display is what makes it slow. We will conduct a modified test series, without the display part to check on performance. There is also the aspect of the hard disk performance, but the both series, user creation and chat showed more or less the same difference and user creation requires hard disk access, while chatting does not, so the difference of the 5400 rpm on the PS3 and the 7200 rpm on the dc do not seem to make that much of a difference.

Last not least, it may be that Erlang is not really optimized for CEL processors and maybe it does not distribute the processes correctly, but our personal appreciation is that the display is the bottleneck.

We will see what happens next.

Sunweaver

Stability & endurance tests

Alright, let's get to this week's updates. It is still to early to post any code over here, but we shall give some progress reports. In the last post we commented about next tasks, I took on these fellows:

* Internal user registration (to avoid same user on several nodes or processes) [easy] : Every user that creates a client proxy, automatically registers on the mnesia database to avoid errors. Done for the moment, but this needs to be revised when we get to account management.

* Cluster behavior [medium] : The cluster config will require several things, but the very first one to grab was to define a master. For this a special process called observer was created. The observer is a special registered named service "ob" that stands by and waits that every persistent or longer lasting process needs tro register with and like highlander there can be only one, you cannot run a second observer on your cluster. Once it receives a request it writes that into a public ets table and starts an asyncronous link to that task. If that task or node dies, it will be able to start a stored function, which may decide whether or not to restart or do some desaster recovery, but in the very least, it writes a special local table called monitor (the table does not get replicated for performance issues) to record all of the closed-while-monitored tasks, no matter why, this way we will ALWAYS know what happened, even if you close a whole node, actually, this is why I designed a central Observer, to catch crashed nodes, local receovery was not enough for my taste. The observer is currently a signle point of failure, because there is no backupo for it at this time, this requires improvement.

Now this topic will be ongoing, the cluster behavior obviously does not stop here, it got barely started with this, we will have to look at other topics like:
--> Server tasks (DB, Players, Simulation, NPCs, Chat, etc.)
--> Is a cluster dynamically or statically built, does it load-balance ??
--> We need to notify when a player is between two zones, in order to get smooth character display or else they would just suddenly pop into existence in the next zone,
--> Can we resize the managed zone in run-time ?? For densely populated areas f.e., this maybe rather complicated to implement as it requires continuous terrains.


TCP/IP server with authentication
I have started to develop the TCP/IP interface which worked actually easier than I thought. So far you can connect to it, the server sends back a challenge which in the future will be a MD5(random) sort of function and the client will have to answer with an account name, MD5(challenge), which the server will validate against the database, if the password is wrong the servers disconnects, if it's correct the server will check if the user is already online and so forth and finally create the client proxy, which will be the man in the middle between the TCP/IP interface and the chat client (the low level connector). I am undecided if I do an action plugin to the CC or do a proxy that just uses the CC to connect to the low level functions, most probably I will scratch the action plugin and replace that functionality with this client proxy, it sounds like a better idea, but it also means that per client we would have 3 processes running at the bare minimum. Advantage is that you can just write whatever small program and have it call the CC or call the CC directly from the console and monitor a certain chat channel. So many decisions, so little time :) .

Endurance test:
It seems somewhat early to do stress testing, but I find that most applications don't get off the ground because the backbone has been poorly designed, so thoroughful testing is of the essence. I have tried to open massive clients and I designed a special routine for this, so far I can very easily create 10'000 or 100'000 clients, which open without any problem and send messages between them, we are on the right track and if you close one of those large nodes you get a whole lot of notifications in the database, but no errors, everything is stable, even the code updates keep working, right on !!

Next steps:
Next we will go for the action plugin or client proxy and start designing the simulation behavior, this is a major decision, so a clean and logical design will mean the world (quite literally). Also, a backup behavior for the Observer is needed, given that all his information is stored in an ets table a restart should be quite easy to implement and a second master node needs to be coded. Also the TCP/IP server needs still a lot of work.

Next tasks on the To-Do List:
* Create an action plug in template [easy]
* Cluster behavior [medium]
* Zone Simulation Manager [medium]
* New: backup observer and restart design
* New: mnesia fallback behavior ( to have 2 master nodes, one in stand-by mode )
* TPC/IP server with authentication [medium] + protocol converter [medium]
* TCP/IP test client [medium]
* Apocalyx integration [medium]
* Master node [hard]
* NPC scripts [hard]
* Subscription Server [hard]
* Interrealm connector [hard]