There is always another way

"There's an Italian painter, named Carlotti, and he uh, ahem, defined beauty. He said it was the summation of the parts working together in such a way that nothing needed to be added, taken away or altered [...]" Cris Johnson (Nicholas Cage) in Next

Good thing he did not talk about our code because each time I look at it I find another way of doing things, Erlang is really much faster when it matches patterns rather than using traditional programmatic code, so perfection or beauty is still a long way down the road, but anyway.

RTFM or ... how I should really read the manual more often, I had completely overlooked the function net_kernel:monitor_nodes(true) . When you subscribe to this function, in our case the load balancer (LB) does as of now, you receive a message {nodeup,Node} the instance a new node comes up so now the code gets copied and started instantly on new nodes, it couldn't be faster than this, if you blinked you just missed the copy+start process. A message of {nodedown,Node} tells you when one disconnects.

I also stumbled upon another very useful function called rpc:sbcast(Name,Msg) which sends a message across all connected nodes to a registered named process. Until now the group process caches a list of all groups on the different nodes and new groups can take a while (up to 20 sec. so far) to be seen by others nodes, but given that we now register each group on each node, this function could speed this process up, without relying on cached mnesia data, my only questions is, what happens when 2 nodes do not yet see each other ??

Speed-Ups: Every time I look at pattern matching I learn an easier way to do something, when I first started I was tempted to use traditional loops like the function lists:foreach then I switched to:

lookup(G, [{G,Pid}|_]) -> [Pid];
lookup(G, [_|T]) -> lookup(G, T);
lookup(_,[]) -> [].

I find that structures like the one here are even faster than the other two:

[X || {Y,X} <- L] where X is our return value , L our list to look up a value from and Y our variable to pick the correct tuple from a list. This function is so fast that it is causing me some synchronization problems among my processes for reasons yet unknown. This function will also serve as a command interpreter, when the client sends a string of commands, like "{cmd1:arguments}{cmd2:arguments}", I convert that list with a function I designed into a list like this [{cmd1,"arguments},{cmd2,"arguments"}] and I can then use the above [fun(Y,X) || {Y,X} <- L] to run some function on each element in the list. I might even displace Mnesia altogether, just load data in lists into the KVS, look up / filter values with pattern matching and sync the nodes with rpc:sbcast. I could use the module dict to create a dictionary instead of a table or list, too. Each dictionary could run in its own thread, hmm interesting. This sounds so crazy, it might actually work.

Remember that Mnesia is nice when speed is not critical, but when you need to do thousands of look ups per second across "n" nodes then the generated overhead is just too much, mnesia can't keep up, no matter how I structure it, be it several disk nodes, 1 disk node several ram nodes, the worst thing is the lack of a local cache on a ram node, hence the KVS working as a local cache. From a mnesia centric application framework I am getting more and more towards a KVS centered data structure, the above scheme might eliminate the caching errors and I do not require really the supertight transaction property that Mnesia offers, I'll gladly trade that for speed.

Designing the authorization module (AUT): it will mean a large impact, because I will need to define a whole lot of logic around this, including what commands each user can perform, guild channels, raid & instance channel, guild structures and more, so this will take some time to complete, but the design isn't finished yet, I am still prototyping the logic. Some nice features that will most likely come up are f.e. you will be able to define as many guild channels as you like and define their authorizations to differentiate the general channel from officer, class lead, raid or other group channels below the guild. I will also consider instances and raid channels with ownerships, to prevent problems like an invited outsider stealing raid IDs, a documented problem that happened to other large MMOs. Under the devised scheme an invitee will not have ownership or admin rights and it should be an easy thing to do that certain instances must be started by the owner/admins and hence avoid getting a started raid instance stolen. The user commands wil also be important to avoid that a guild master leaves the guild without reassigning ownership to somebody else, so a GM will lack a command of GQUIT, but have a GOWN, while other members will have GQUIT, but none afiliated will have the GJOIN command sombody unafiliated will have. The authorization module will need to combine speed with functionality so a huge fun part is coming up here.

With the speedups like the ones above, I am now facing sync problems between threads (that should not exist on the first place), the first or first few messages from a client get lost in cyberspace, I know that this is due to me cacheing values from mnesia, so I am thinking that the easiest way to avoid lost messages will be to implement a ping-pong protocol to allow the client to ping the server (once authenticated) until he gets a pong response and only then start asking the server for more stuff or start chatting. The cacheing delay can be anything from unnoticeable to several seconds.

Another curious fact is that the more functionality I design the smaller the modules get.
The module CS which was in the beginning the master of the whole framework is becoming less and less important, with groups registering now, it is not much more than central resource locater/creator. The CG module is now the local node relay station, down from channel admin, the logic has gone to the channel NPC. Both are still very much required, but their role has drastically changed during development, curious.

Cheers,

Sunweaver

And: Action !!

It is 2009 already and I have finally some time again to keep programming.

Action handling
After much thought on how to implement the application behavior, currently all chat was sent around without validation and a MMO needs to have an application logic behind each channel, so specific action handling needed to be implemented to process each client info before sending it around in the channel. I have come to the conclusion, that probably the best way to do that is to implement them as a player (CC) module with an action handler (AH), both are stand alone processes, so whenever a group gets started up, it will automatically seek in a table for the AH of that group or else start up the generic entry as the handler. ANY message to be sent in a group gets sent to that single automated player entity, which then decides how to proceed, so for chat channels one can implement a word filter, NPC data can be implemented from here or zone channels will be able to send the movement data first to a physique server (far future) before sending that data to the clients. This single AH per channel will also keep track of connected clients, no matter what node they come from.

Some lessons learned of programming in Erlang:
First lesson: there are so many ways of implementing a certain behavior, it's mind boggling, one can choose from creating vanilla subroutines, a cherry mono- or multi-tasked applet, a table entry, a DB table or whole self managed entities, the choice is really vast and the problem is that you can be quite dynamic, so I have to sometimes go back and reread where all the magic comes from, all or most choices seem to be on par speedwise. Once decided, implementing it is a snap.
Second lesson: right size your multi-tasking; it's so tempting to create many, many processes, but syncronization can sometimes be a pain, so double check when to MT and when no to. Even when sync-ing is no problem, don't forget the possible size of the outcome, f.e. in the very beginning I sent each chat command as a separate task, but when I did the stress test with 50'000 generated users over 5 servers I found that 2.5 Billon concurrent tasks was a tad too much to handle for the framework, I wonder why that would be, how lame !!! =P
Third lesson: And Erlang programmers will hate me for this: OTP sucks big time, I found that it is much easier to write my own functionality, including but not limited to code upgrades in runtime across all connected servers, which requires a Doctor title in OTP to get to work, but only little effort to programm oneself, nowadays I only kick in some very few lines of code and voila, it upgrades !!
Forth lesson: they say love comes quickly and so does complexity !! Programs in Erlang are short and concise, but when implementing something new I find myself rewriting code in 5-6 programs in parallel, which isn't harder, but very different from "normal" programming languages.

Next to come is a channel authorization table that decides at logon (or later) what client should connect to which channel(s). I will also implement channel authorizations for that, which will be interesting =O, some must be system assigned (zones and instances), others with password, by group assignment (like a guild f.e.), open or user managed, so this will be an important feature and also some piece of work as I need to rewrite some code for that. Before or after that I will implement the first NPCs with some limited actions, oh and I still need to get the object table into the Apocalyx client to handle mutiple objects. But after these tasks I have the basic capabilities for an MMO, that is good news.

And Hello World !! Since my last update I seem to be getting quite a lof of hits on this blog I can even google this blog, much to my surprise actually and scary, too. SMASH is not ready for prime time after all and it may still be a while, due to the fact that I only do this on my spare time. So for the newcomer: feel free to read the MISSION STATEMENT. This is still vaporware, but whatever is mentioned herein is actually implemented, so one day (hopefully this year) the framework will be up and running and I will set up a closed alpha server for some testing. But I will not haste anything, quality overrules time requirements.

That's all for now !!

Cheers

Sunweaver

And the changes go on

There hasn't been much to show for the moment and for some reason. In order to progress on the GUI side it is much required to be able to know, who is present on the channel, so any object present in a zone needs to report it's there, so the CG was modified and now responds with the names of everyone present, this will now require an object database, where you can then look up what's what and who's who, so the client knows what to draw, most likely it will inquire each CC object to avoid DB lookup penalties. Once that is done, we can now proceed to create the SIM Object table on Apocalyx.

Also, another change was made to the CI routine, the one that compiles and installs new code on all connected servers, it now recognizes all Smash processes on all nodes and updates those processes that run old code, it does that by swiping the processes on all nodes and sending upgrade signals to the "old ones", just as a fail safe to make sure that all processes are ready for upgrade anytime. Updating the whole system in runtime has never been safer, remember that one of the goals is to update the system without ever shutting it down. And to all Erlang veterans: this is about a 1000x easier than using the OTP way, which requires a doctor degree to make upgrades happen. In this system just open a node for compiles, debug your code changes until no errors occur in the shell (or else running processes bomb out as per Erlang VM specs) and then type "ci()", which will distribute and upgrade all code on all nodes, now THAT is easy.

Cheers,

Sunweaver

Still working on a first Apocalyx front-end

Well, Apocalyx LUA is tougher than envisoned, we are still trying to produce a first GUI, but just not ... yet. A couple more days I suppose. So far we merged the MD3 demo with the network demo and the mini applet connects to the server, authenticates and the soldier runs around in circles, you can send position data at the push of a button and that's it so far. We are still fighting with the camera, which just won't stick to the avatar, but well. What's clear is that we need somehow a channel manager that logs who is on each server channel, so newcomers can query for other objects in the gaming world or we need to expand the CG threads to report that info back per node, it would be less error prone if something fails. And also required now is a vocabulary check at the socket server level, to filter out what commands a user can utilize, remember that users should be able to do different things according to their profile and/or authorizations. Also maybe a filter should be implemented that allows only one command of a certain category, like movement, chat, whisper, change equipment and such, or users might wanna cheat sending several movement commands at once and move faster than allowed, so that needs to be taken care of.

So the next steps for the week to come are:
* Sending Apocalyx movement, orientation, char and chat data around a chat channel
* Chose a BSP scenario, not just a blank endless terrain, maybe Genoa or city.bsp
* Modify CG behavior to include a report on who is on that channel
* Command permissions per user level and/or specific user
* Filter multiple commands to avoid cheating (+fix a CC send bug)

And that's plenty for now.

Sunweaver