No updates around here ??

As it happens from time to time, excess of activities force us to put some of our projects on-hold during those moments, so for the moment there are no updates because of lack of time for development, but the project is not dead. Also, I recently stumbled across another MMO platform that comes conveniently with source code, which I am studying, too; the Mangos-Project. Interesting and inspiring stuff, although I won't implement their ideas into SMASH. I guess some more news will be out like at the end of September, with the multi-player display and some improved chat functions.

Cheers,

Sunweaver

Dynamic Environments

There hasn't been any update in almost 2 weeks, I know, and there won't be much to talk about at the moment. I am trying to find out how to create Dynamic Environments, sort of "SimCity"-like dynamic scenarios, where you create buildings and everything. Simple as it sounds, this is quite a challenge for the noob-Apocalyx'er like me, so you can imagine the fun I am having here. What I have in mind is something like this:

1) Changeable height maps (at some point), at first the world is going to be flat as a flounder.
2) The Wild West: People build houses, villages, mountains, valleys, farming, dynamic trees, etc.
3) Mega Cities: Think of living in a Mega City like in the movie "The Judge" or "Star Wars" cities like Nar Shaddaa (cf. Wikipedia links).

Once we have the ability to just drop building blocks onto each other we can then build in runtime really large scenarios, with very high buildings, several levels, etc. This sounds very appealing to me and will allow for a more dynamic game play. What it implies is to create a collision detection per object, bounding boxes and a physique simulator, all of which are included in Apocalyx, it's just a question of finding out "how". =O

And well, I have yet to finish the routine to add multiple object to the scene, that will enable the above functionality in the first place, much work ahead and so little time, but then it's better to do it right from the beginning, instead of fixing it later on.

Cheers,

Sunweaver

And the changes go on

There hasn't been much to show for the moment and for some reason. In order to progress on the GUI side it is much required to be able to know, who is present on the channel, so any object present in a zone needs to report it's there, so the CG was modified and now responds with the names of everyone present, this will now require an object database, where you can then look up what's what and who's who, so the client knows what to draw, most likely it will inquire each CC object to avoid DB lookup penalties. Once that is done, we can now proceed to create the SIM Object table on Apocalyx.

Also, another change was made to the CI routine, the one that compiles and installs new code on all connected servers, it now recognizes all Smash processes on all nodes and updates those processes that run old code, it does that by swiping the processes on all nodes and sending upgrade signals to the "old ones", just as a fail safe to make sure that all processes are ready for upgrade anytime. Updating the whole system in runtime has never been safer, remember that one of the goals is to update the system without ever shutting it down. And to all Erlang veterans: this is about a 1000x easier than using the OTP way, which requires a doctor degree to make upgrades happen. In this system just open a node for compiles, debug your code changes until no errors occur in the shell (or else running processes bomb out as per Erlang VM specs) and then type "ci()", which will distribute and upgrade all code on all nodes, now THAT is easy.

Cheers,

Sunweaver

Testing a BSP level

As stated before, the next updates will be somewhat slower because of the obvious learning curve to understand LUA & Apocalyx. Some work has been going on to create a login screen, which is still incomplete, but it's underway at least a first glance: the background comes from the gui.lua demo and is using the GLGooey for the dialog (so far). The background will change, the important things is to try that GUI out.

As I tried to adapt a city level BSP, I ran into some noob problems, but fortunately Leo (creator of Apocalyx) helped promptly to solve them [ Thanks again Leo ], so here is what the same soldier looks like in a city:

What you see here is Leo's city from Urban Tactics. The toon's position now gets transmitted automatically every 200 ms whenever the position is marked "dirty", meaning, that there was a change and the 200 ms is the minimum interval permitted by Smash, send more often and your client will be in trouble, because the sockets are blocked for 200 ms (to avoid flooding) on the server side.

Currently underway is making the login work and creating the table structure for all network created objects like other players, NPCs, cars, trees, grass, boxes and whatever comes to mind. I am thinking of make certain objects client persistent, so the server sends those persistent objects once specifying model, position, orientation, size, expiration and an object name and all the following times the client only checks those objects by expiration date and object name only, instead of downloading the full set of data again, once I get the object table in place that is.

Live long and prosper,

Sunweaver

A first peek at SMASH

A picture says more than a thousand words, so here it goes, this is the very first screenshot taken from the Smash Apocalyx Duo (should I call this SAD ??):

What you are seeing is a modified version of one of the basic model demos from Apocalyx with the console overlay displayed, where one can see the network information, in this case a successful logon, some positional data that the client sent and some other test clients who logged onto the server. Now we need to display other items, users and chat in this empty game world and a change on the framework (probably a whisper from each CG) to know who is on each zone channel, but we are getting there =) .

Laters,

Sunweaver

Still working on a first Apocalyx front-end

Well, Apocalyx LUA is tougher than envisoned, we are still trying to produce a first GUI, but just not ... yet. A couple more days I suppose. So far we merged the MD3 demo with the network demo and the mini applet connects to the server, authenticates and the soldier runs around in circles, you can send position data at the push of a button and that's it so far. We are still fighting with the camera, which just won't stick to the avatar, but well. What's clear is that we need somehow a channel manager that logs who is on each server channel, so newcomers can query for other objects in the gaming world or we need to expand the CG threads to report that info back per node, it would be less error prone if something fails. And also required now is a vocabulary check at the socket server level, to filter out what commands a user can utilize, remember that users should be able to do different things according to their profile and/or authorizations. Also maybe a filter should be implemented that allows only one command of a certain category, like movement, chat, whisper, change equipment and such, or users might wanna cheat sending several movement commands at once and move faster than allowed, so that needs to be taken care of.

So the next steps for the week to come are:
* Sending Apocalyx movement, orientation, char and chat data around a chat channel
* Chose a BSP scenario, not just a blank endless terrain, maybe Genoa or city.bsp
* Modify CG behavior to include a report on who is on that channel
* Command permissions per user level and/or specific user
* Filter multiple commands to avoid cheating (+fix a CC send bug)

And that's plenty for now.

Sunweaver

The Load Balancer is ready for prime time

The Apocalyx frontend is not yet ready, I am still messing around with that and well it's not showing yet anything, but soon, very soon. BUT, the Load Balancer (LB) is ready to be integrated into SMASH. The LB will check for speed on each connected node and how many processes are running, then it calculates what that means in terms of availability for the cloud. This game is a zero sum game, but for the final evaluation, every single node will count, negative values are not permitted, then the total percentage is done. For the end user there are 2 procedures to be called, remote rpc (rrpc) and remote spawn (rspawn). rrpc will spawn a procedure remotely and wait max. for one sec for the result or rspawn will spawn a procedure remotely and return the PID, in case that the procedure takes longer or you need to spawn a server. Those 2 procedures should do for the generic cloud service. This might even mean a strong impact on the Simulation itself that will run on the framework, the initial idea was to run a zone one a single node, but with the LB there is really no need for any kind of restriction, you could run the Sim threads all over the place. Let's see what that does for the framework :) .

Cheers,

Sunweaver

Some more remote utilities

Some might know that SMASH has included into its belly an automatic updating system, which allows to compile all modules and reload them on all connected nodes in run time, without ever shutting the system down. On occasions this fails and a certain service shuts down. To get a hang on this, we have created some utility routines, one permits to execute commands remotely (on the other nodes) and the other shows if any module is executing old code, so very soon we will be able to first check what module on which node is executing old code before a new compile is allowed. So far the problem seems only to reside on longer lasting timers, which are not included in the update signal, all others update correctly (it seems). Also with these utility routines we could now easily start all preconfigured services (in the cluster file) remotely, without needing to touch then anymore. Some other fixes are that the CC proxy can now send tells or whispers without subscribing to any channel; KVS can now cache and forget, when you ask kvs:cache(M,F,A,Timer) it now will update every "Timer" ms and forget about the value after a minute, so the dictionary does not pile up with hundreds of unused values, this works like a garbage collection to keep KVS trim and slim. And more tests on Mnesia have been executed and curious enough on remote nodes the dirty function is slower than the normal database access with transactions and as to be expected KVS suffers from the initial Mnesia access, but afterwards blazes thru repeated accesses, so no matter how much we try to optimize the database access, the cached behind KVS approach is still the best when repeated access to data is required. Dedicated MMO literature also comes to the conclusion that processes on a multi server structure should update their data every 5 sec., less seems to be too much. And the cloud infrastucture has given some new ideas, Erlang already has the ability to spawn remote processes, so I guess, we just might implement our own flavour of it, by speed measuring each connected node and on the other hand check how many processes are already running there, this could be something like a routine task that executes per "X" interval and then when asked to spawn a process it checks for the best node to do so, a kind of "cloud_spawn(M,F,A)". This would make the SMASH ready to process any kind of task actually, not just a specialized Simulation Framework. It just might happen =) .

And Apocalyx can indeed use tables to store toon data, I played around with the Urban Tactics demo and it can easily handle some 200 toons running around there, very nice, now we need this networked. ;)

Laters,

Sunweaver

Does SMASH do Cloud Computing ??

I was asked the other day if the SMASH does Cloud Computing or Grid Computing [cf. Cloud Computing from Wikipedia] and the answer is that while there is still a lot of ground to cover, in essence that is what the basic framework will do. The Load Balancer (LB) will check for the best resources to allocate (this is still future music) and distribute the simulation accordingly. The basic idea is the same, if you look at the graphic on Wikipedia, the cluster table (that we will have at some point and which will define how the cluster should behave) corresponds loosely to service catalog, the provisioning tool basically does what the LB will do and the Monitor is in essence what the Observer does. Once the pieces fall in place we will obviously also need a Systems Management component, which might be part of the cluster behavior.

So, given that this project was started like 4 years ago, the word "Cloud Computing" did not exist at that time, it seems to have formed during late 2007, but the idea of covering massive computations by networking them on many servers, instead of a large scale server is the same, so in essence I guess, SMASH is a Cloud Computing Framework and what's more, thanks to the unbelievable Virtual machine that Erlang runs on, we can already run cross-platform, the test cluster SMASH is being programmed on, consists of Windows, Linux and MacOS X systems.

Cheers,

Sunweaver

48 hours of endurance

Alright, not much to report for now, except for some minor changes on the Observer side, which did not catch all required objects. And the channel management requires a major overhaul, whisper commands currently trigger being subscribed to that channel, which isn't exactly desirable, meaning that we will require authorization management, ouch. That's a major CG (chat group controller) rewrite, but anyway.

And now to the 48 hours test: yes the server holds up. The caching Key Value Server (KVS) makes the whole thing fly and no messages pile up, things have never been so fast, it does not matter where a client connects to, the SMASH framework relays the messages blazingly fast to other nodes, so we know by now. This concludes stability and speed tests for the time being.

Leo, if you are reading this, I sure hope that Apocalyx permits me to create each 3D object as a table entry and not just as a variable, like your examples showing variables like avatar0 - avatar6 and so on. I will be needing to create a LUA table and put each avatar as one entry and then update that table. I guess once your new Gun Tactyx is out I might try to "socket" it and permit people to connect to a fighting event, we'll see, so now back to learning Apocalyx. And yes you should be able to see some kind of graph here soon, to reflect what we got so far.

Stay tuned,

Sunweaver
EDIT: I shortened the post.

And there shall be speed

It seems that the new caching KVS server does miracles, the message throughput of the whole framework went through the roof, up from a maximum of 7000 messages per second (averaging 2500 at heavy load) to a minimum of 150'000 messages, that's a 2100% increase, not bad. So, now we are indeed ready for the next step, Apocalyx here we come, the next thing will be to visualize our first GUI.

Sunweaver

Need for speed

When doing the first message bombardment on the server and the news are mixed ones. On the one hand the server does not crash, which is nice, but after a while it starts to behave irrationally, like when logging out and logging back in, it does not seem to delete users from the DB anymore, so a kind of measurement is needed. Writing some quick benchmarks it turns out that the Chat Server (CS) can do 600'000 messages per second (mps) and the DB lookup in the chat group can only do about 6'800 mps, with 16 clients connected we get up to 2'600 mps, but I suppose that the constant bombardment becomes a problem at some point, because Erlang starts piling up messages and the missing client deletion from the DB is most likely the same cause, so a first conclusion is that Mnesia can't keep up with all those lookups on the DB and this causes messages to pile up like crazy. Fortunately this was likely to happen and now we have a legal excuse to use the KVS (Key Value Server), which has been completely dormant until now. The KVS will need to grab this kind of data from Mnesia and put it in a local table. The big difference now will be this, the KVS will only load the data from Mnesia every "x" time intervals, like maybe every second or every 5, meaning that we will lose precision in order to gain speed. We will now have to segregate our operations, those that require ultimate precision like close by moveable objects, banks, auction houses, trades and loot operations and others like chat, or even general environment operations that do not require the most precision. And we need to start with the Chat Group structure. This should be really easy to do, we will see if things speed up afterwards. A general throughput of 500'000 mps looks like our boy, but 7'000 mps does not.

And surprise, surprise, the Playstation 3 is not nearly as powerful as hoped for as a server, it's depending on the test 4-10 slower than a Dual core Windows Laptop, averaging so far like 6x slower, how very disappointing. In the CS test it scores at 95'000 mps, the DB lookup only gets 4'400 mps and it takes 24 sec. to create 10'000 CC clients, while the Laptop does that job in 7 sec., OUCH !!!!!

Long live the benchmarks.

Sunweaver

Notes on the work in progress

Today we will get into some more detail as to what the WIP list means and check what comes next:

Work in progress (WIP) of the To-Do List:

* Create an action plug in template [easy / pending]
Currently a user logs on and a socket + chat client process are created, what is needed now is another process that will sort out what to do with the messages, what this means is that the socket is the non-transient part of the user, if he disconnects this process gets killed immediately, but the CC client stays there for "x" time, so a user cannot just logout when in danger and no harm will happen to him, this CC socket guarantees, that he cannot cheat by quickly logging out. The plugin is required to process his requests and whatever comes to him, this includes for example:

>> Action dictionary: what actions the user may execute, like chatting, moving, joining other channels, changing clothes and other stuff, so with that you could simply delete chat from his action list and mute the user, remove his movement ability and have him stand still or disallow a certain spell for a while.

>> Swap the action plugin: under certain circumstances you may want to put the user under script control, like for cut scenes or "mind control" a user by another user, so being able to switch the action plugin is helpful.

>> Put in the ability to have several plugins, this way we could f.e. log all chat activity, maybe this is not required.

* Internal user registration (to avoid same user on several nodes) [easy / partially done]
This is done for the CC processes, but needs to be done as well for socket processes, this shouldn't be too hard.

* Cluster behavior [medium / pending]
Here we might predefine what server does what task and what to do when that server is unavailable. This way we don't have to configure our cluster each time we start it up.

* Zone Simulation Manager [medium / WIP]
This part will include a chat manager for a certain zone and to extract from the database all the objects that are contained within it, like fixed objects and moving objects.

* TPC/IP server with authentication [medium / WIP] + protocol converter [medium / WIP]
This is basically done, what's missing is the above mentioned database registration for logged on users. The server has already functionality in place to block certain IPs and to avoid flooding by disallowing the port for 100ms, hence only 10 actions per second are possible.

* TCP/IP test client [medium / WIP]
First client is done, which is a simple text client that displays all messages and crashes *sigh* after minutes or hours because it can't handle all the display stuff, next a graphical client is needed, this will be done in Apocalyx.

* Apocalyx integration [medium / pending]
The first client in Apocalyx is right ahead, this is the next step that we need to create, hopefully not too much of a mess to do.

* Master node [hard / WIP]
The ultimate goal of this is to run a process on the master node that "thinks" for the rest of the infrastructure, like how many processes are running on each node, how many SOX users, CC users, Sims and so on and rebalance the equation, this will require a lot of work and will be most likely the last thing to do.

* NPC scripts [hard / pending]
This will represent a mayor effort to finish, given that this will have to handle scripted quest givers, movement from one place to the other and simulate AI to give the NPCs a certain life and methods of learning, exciting but difficult, given the overall broad approach.

* Subscription Server [hard / partially done]
The first part is already implemented, each user has a validity period after which he cannot log on anymore, you can already, per shell, increase the period, but we cannot handle prolonged subscriptions as of yet nor do we have like a web-based secure server in place for this.

* Interrealm connector [hard / pending]
In a word: Think BIG !! Each realm might be a separate instance of the same Simulation/Game or it might be a completely different game, so user data needs to be copied over, and obviously a domain/realm browser needs to be created. Sounds like fun.

* Physique server implementation [medium / pending]
This can be done the easy or the very hard way by duplicating all the Apocalyx logic into Erlang (sounds like disgusting) or to use Apocalyx itself. If we use Apocalyx, then we have to have a client (probably for each zone) that runs it and every user will send their requests like this:

game client --> SOX --> CC --> action plugin (sorting out movements) --> physique server (sort out the ACTUAL possible moves) --> action plugin --> CC --> SOX --> game client.

In order to make this work, the Physique server needs to do a simple rendering, that does not use any textures or coplex figures, so we might represent of all items only the bounding boxes for collisions and represent users as boxes, this should make this process painless and fast and if a collision is detected then only the real movement gets reported back.


So the next steps are the SOX database registration to avoid double logins and to create a simple Apocalyx client. After that the zoning and char room part with authorizations is due.

Warm regads,

Sunweaver

SOX now upgrades correctly

One issue in the past had been, that the socket server could not be upgraded in runtime, but now it works, so we can as of now upgrade the code in runtime the whole system and not lose a heartbeat, no disconnects, nothing, the system keeps running. What this means is that we can have our game running, while you kepp customizing your code and then just compile+load the code on all connected servers while players are online and they won't even notice you changed something. This should greatly facilitate maintenance tasks and avoid any kind of downtimes. And now back to the other pending items.

Sunweaver

Tuning to perform

The initial performance is being tweaked, while not bad we feel, that more can be done here regardng message passing and some first strange errors have appeared, no showstoppers, but something to deal with.

Also under current analysis is how to proceed with zones and where players show up, should one player appear in the next zone when he is close to the next one, that would mean like overlapping zones and if a player is in the center he only reports his position into one if he runs towards a corner he would potentially run into up to 4 overlapping zones ... or is it better to make smaller zones and always send the coordinates into all surrounding zones, which would mean always to transmit into 9 zones. Zones could also be done through range lists and handle everything like one large zone. In order to determine this, Apocalyx will be required, so next steps must include a graphical frontend already. More coming soon.

Work in progress (WIP) of the To-Do List:
* Create an action plug in template [easy / pending]
* Internal user registration (to avoid same user on several nodes) [easy / partially done]
* Cluster behavior [medium / pending]
* Zone Simulation Manager [medium / WIP]
* TPC/IP server with authentication [medium / WIP] + protocol converter [medium / WIP]
* TCP/IP test client [medium / WIP]
* Apocalyx integration [medium / pending]
* Master node [hard / WIP]
* NPC scripts [hard / pending]
* Subscription Server [hard / partially done]
* Interrealm connector [hard / pending]

Sunweaver

SMASH to Planet Earth ... ??

While not finished by a long shot, we can now communicate between TCP/IP clients and the SMASH framework. The integration is not yet 100% complete, but one can now authenticate from any TCP client against the accounts table and create a chat client (CC: the player proxy). Any chat as of yet joins prefixed chat channels, this needs still a lot of work, like a channels DB, with authorizations to join, chat, listen, member list, password and so on, but at the very least you can now chat in 'general' and broadcast to any connected "player". This also works across nodes, so you can connect to different SOX servers that run on SMASH and even if the CC was created on a different node and still persists, as expected, the SOX thread correctly communicates with that CC, phew !!!

So, what's next ?? Now comes first of all a light test, after creating a scripted Delphi client, we will now do some tests over Apocalyx and create a VERY simple game world. After that there will be some very heavy work where we need to create a character selection / creation something and a channel table with all kinds of authorizations, cheers !!! This will be fun. And also upcoming then is a command parser, to enable the server to start validating stuff, because up to this point each client will decide positioning and capabilities, which is obviously not acceptable. During this step we will also have to implement a physique server, current thoughts are to put an Apocalyx client to do that, but to omit texture rendering and represent clients only as cubes (for line of sight), so we could verify very precisely physical conditions in the game without reinventing the wheel, but much lighter than a client side reproduction, let's see how that works out.


Stay tuned, a graphical test client is just around the corner.

Sunweaver

First benchmarks

We have conducted the first couple of test series on a single-core notebook (sc), a dual-core notebook (dc) and the PS3, where creating 10'000 users takes 1:11 min., 0:22 min and 0:42 min respectiviely. We then proceeded to send chat messages between the 3 systems and the results are similar in ratio. What is very apparent is that the PS3 displays the text relatively slowly, so in my opinion given that the CEL processor should be very superior to the dual-core it seems that the display is what makes it slow. We will conduct a modified test series, without the display part to check on performance. There is also the aspect of the hard disk performance, but the both series, user creation and chat showed more or less the same difference and user creation requires hard disk access, while chatting does not, so the difference of the 5400 rpm on the PS3 and the 7200 rpm on the dc do not seem to make that much of a difference.

Last not least, it may be that Erlang is not really optimized for CEL processors and maybe it does not distribute the processes correctly, but our personal appreciation is that the display is the bottleneck.

We will see what happens next.

Sunweaver

PS3 + YDL + Erlang = ready for prime time

Ok, it's been a long Sunday, but I did manage to install Yellow Dog Linux on my Playstation 3 successfully. It took much longer to backup the PS3 than it did to configure YDL. After that I installed VNC Server to remote control the machine and let me tell you, that the Gnome Desktop is a beauty. Installing Erlang was as simple as downloading and compiling it, so no new really on this end, either. In other words, the server platform is ready for deployment, I will now go ahead and run the SMASH framework on it to compare speed ratios among my test systems, let's see what happens next. This is for me a major milestone, given that THIS PS3 will be the mother of all SMASH development nodes. I will have some time next week as well to push the conection between the socket server (Mr. SOX) and the SMASH framework, so I definitely hope to have some empty game world that people can actually connect to by next week. If that holds true, we might do a very first game world logon, maybe, maybe, maybe.

Oh and by the way, some may ask, WHY Linux and PS3 at all ??? Very simple, the thought process behind that is, that I want to be able to provide a server frame work taht is inexpensive and powerful, because regular servers are definitely more $$$ heavy and not everybody will be able to afford buying Windows Server licenses, so nothing beats Linux moneywise and a U$500 PS3 server should make a nice duo. Now, that much said, many people are scared of Linux, because it's not exactly intuitive, well, given that we only need a DVD install and later on a simple download + install of Erlang, that should be fairly simple to do for everyone and ALL the others tasks can be done thru native Erlang, no more touching of the PS3 is required, I am planning on adminstrating everything from a more comfortable platform, without losing the power of the solution.

Stay tuned.

Sunweaver

What's next ??

Alright, it's 2008 and I finally have some time to come back to the SMASH project. One of my goals was to buy a PS3 and install Linux on it, so at last I got me one and I will install Yellow Dog on it this weekend. Why a PS3 ?? The processor of the PS3 has internally 6 cores that are ready for use and it is said to be a very powerful server, so I will set up Erlang on it and see just how fast it goes. Erlang recognises multiple cores, so this should prove to be a powerful combination, a dual-core versus single-core scores about 40% speed increase, so 6 internal cores should make quite a difference. I will post how it went, both installation and benchmark.

I am also working on the TCP/IP server and I should soon have a first first alpha server up and running where people can connect to (probably a pretty empty world).

See you soon.

Sunweaver