SMASH - MMO Engine in Erlang

Project on hold for a while

2010-05-20T12:10:00.006-05:00

I am currently fighting on too many fronts to be able to keep up with this side project, which doesn't mean it's dead, I definitely want to see some progress here, but at present this is impossible.

This fact not withstanding, whenever I play a MMO I think how I could this or that in SMASH, like f.e. I find myself intrigued with Star Trek Online's design. Cryptic implemented STO in such a way that there is only one realm for the whole game but every location is an instance to avoid sending hundreds if player data to each client, you get sent to a certain instance and that's about 20 players or so, nothing more, chat however is sent to all instances. I like this design and I have wondered how this relates to SMASH. My conclusion is that this could be implemented by means of what server you are connected to, so chat messages could get relayed like they do now, but player data f.e. could be limited to be sent only to the users on that same server, voila, we got instancing. It would only be required to allow instance switching, by means of logging a user out of one server and on the desired instance server, I am not yet 100% sure if I am not overlooking anything, but I guess SMASH can easily handle instancing this way.

To handle instances like private raid dungeons this is a bit more complex, as it would require copying zone data into a new zone, but we are still far from this approach anyway, so not much to worry about, yet.

Not many updates for now, but at least a sign of life. My goal is to have a working SAD demo by the end of 2010 with a simple zone, I cannot promise more given all the stuff I need to handle at the moment and my apologies for that.

Cheers,

Sunweaver

ANNE gets more emotional

2009-09-15T12:31:00.020-05:00

It's been a while since my last post and that is due to the fact that I have been testing ANNE and expanding her emotional range. I am trying to set up a neural network for a "Smith and Ellsworth" style model, which reflects the feelings of happiness, sadness, anger, fear, disgust, surprise, boredom, challenge, hope, interest, contempt, frustration, pride, shame and guilt by determined by the external factors of pleasantness, responsibility, certainty, attention, effort and control. Given that I don't have access to the model I am making up the matrix as I go. It takes around 15 min (!!) to train ANNE, but evaluations are instantaneous (Erlang is speedy), oh well if you need more precision, it takes 149 microseconds (µs = millionth of a second) on the average, a mean of 1µs and a maximum time of 15900µs (most likely the first call), so you could perform over 6700 calls per second to it, not even the most emotionally unstable human could evaluate that many ;) . What this shows is that a single process running this ann could easily evaluate emotional feedback for thousands of NPCs in the game world. Believe it or not, this rather simple matrix of (6, 20, 15) takes up 7 KB !! That's quite a mouthful of data, but anyway. Random tests show very logical although on occasion rather unexpected results, remember that anns do pattern matching, not value matching.

Here some random inputs with E being the emotions ann, note that sometimes even when changing 2-3 inputs they still result in in the same emotion. Curious, but consistent and no worries, it really does evaluate to all possible emotions depending on the training matrix and input. -1.5 denotes the lowest possible value, 1.5 the highest possible one, so f.e. -0.3 pleasantness means a little bit unpleasant, 0.5 control , means some control and so on. The output shows how certain ANNE is regarding the highest match, results near 1 mean very certain, while f.e. a "0.40 surprised" could mean a little bit surprised so you really get an emotion and a degree, the higher, the stronger. I am starting to love neural networks, this is so much fun :) . I am thinking, that animals probably don't need all these emotions, nor all the input factors, which might lead to different anns for different entities.

Well, it seems to me that I will still spend some time with the fine tuning, so be patient with me. If there is enough interest, I might publish the source code, too.

Cheers,

Sunweaver

Fleeting thoughts on AI

2009-08-17T09:53:00.000-05:00

No doubt about it, AI is a fascinating topic and there are no clear cut answers, possibilities are simply endless. AI may depend first of all on the game genre, so a platform game will have a different AI compared to an evolutionary game.

Nowadays from what I have seen most seem to have some kind a Finite State Machine (FSM), where certain conditions are analyzed and actions are taken accordingly, this works similar to IF..THEN..ELSE statements. So a NPC just stands around forever on the same spot doing nothing or walking senselessly around day after day without rest, at most the NPC is going through its FSM and that's it. When aggroed, as pointed out, most NPCs keep a hate list, where the NPC sorts who has caused how much aggro, be it through heals or damage dealt and decides who to hit on. Run out of range and the NPC resets, the hate list is wiped, it's got a clean slate, boy do I envy those NPCs !!!

Well, IMHO I find this kind of NPC extremely boring. I envision a bit more and I'll mention an example inspired by Matt Buckland's book "Programming Game AI by Example".

A story from the fictitious mining town "Backwater":

Miner Irwin goes every morning out to the mine to fill his pockets with gold, during this process he will get thirsty, hungry and sleepy, but he also appreciates digging up a certain amount of gold, once satisfied he goes to the bank and deposits the money, if he gets thristy he visits the local saloon, has a drink and talks to his buddies, if he gets hungry he goes home to have his lady who serves him some stew (both will need to communicate as well), if he gets sleepy he will lay down, when rested he gets up and at it again. Let's add to this some more sense, he may be receiving input from his surroundings, so f.e. he one day detects within 50 yds. an orc, he also determines his level and notes that he has weapons equipped, his emotional ann determines fear and he runs back to town and alerts the guards. His goals may temporarily change in favor of self preservation, Irwin goes to the local saloon for more drinks until the danger is over. Guard Erl hears him shout and stands ready to fight the orc. Erl loses the fight to the orc with the rusty spoon.

-HALT- Design decision ahead -

Option 1: Normal MMO AI's would wipe their memories, Irwin engages in the fight before perishing, the new Erl is just as "savvy", as he was before, and both will just respawn and surely be killed again and again and again.

Option 2: Erl preserves his short term memory through death, he remembers that the orc with the rusty spoon was 3 levels higher than him, he modifies his internal state and makes a note to himself: if dead by attack and enemy_level + 3 then update in table fights[self+3] values pleasantness - 0.1 and control - 0.1 . When next confronted with that orc the ANNE with emotional responses will evaluate the situation differently, there will be a moment when anger becomes fear and he may just run for his life and/or call for additional help, this orc William is just too much for him alone. The internal goal seeker of Erl might determine that when he has lost 3 battles in a row, he needs further training and emerges as a level+1, he will now be tougher than before, after some cycles, the orc finally loses the battle and Erl, 4 levels higher and 20 generations later maybe feels invincible again. He keeps standing around for a while and his abilities might slowly degrade to the original skills/levels (or he might stay at his level), this learn and forget setup would avoid players farming certain NPCs but ensures that on the long run the game world is somewhat stable. He will slowly forget that he lost 8 battles against orc William and won 2. However he just might remember in his long term memory how many battles he lost and won sorted by enemy levels and how many attacked him. He may also communicate this to his peers, in case of SMASH maybe on the same city chat channel or a special guards channel. Erl will also change guards with others and not just stand there forever, but swap with Marlo, while Marlo stands guard, Erl goes for lunch or dinner and sleeps a while. Meanwhile our miner Irwin either asks the guards for dangers and/or notices that the threat has gone away, he no longer perceives orc William in the surroundings scanning some 100 yds (just to be sure) and goes back to his original goal of getting his pockets full of gold.

If enough orcs have attacked the city, the council members might stage an all out attack against neighboring orc villages to protect themselves.

While this is still future music, this is what I have in mind for the AI, a simulation rather than just a cheap FSM.

This requires an agent capable of perceiving its surroundings (sight and words), a certain personality to evaluate perceptions, a goal seeking Hierachical FSM (HFSM) capable of sorting goals, a neural network (-> ANNE) to analyze what emotions result from its current situation applied to the personality and a HFSM that decides on memory, learning and thresholds whether to pursue its goals or take alternate actions and then carry them out. Let's call this the perception system (PS), goal seeker (GS), personality profile (PP), ANNE you already know, the memory records (MR), the decision maker (DM) and the action machine (AM). If you wanted to implement a hive mind, you would need to sync the goal seeker, the memory and probably the personality profile or if you do it in Erlang, just call the processes that represent those per hive mind.

I believe this design should enable for different actions according to each NPC's personality: f.e. when insulted, Eliza might feel sad, run home and cry, while guard Erl while take his sword and apply a subtle lobotomy. When attacked, Irwin might run for his life, while Erl fights back. When applying evolutionary rules, their behavior might even change over time. Most literature does not recommend to enable long term learning when you go life, because results will inevitably be unpredictable, so only time will tell if long term evolution is a good choice or not.

I highly recommend downloading and running "AI Planet" from sourceforge.net and also read the author's design notes, it is fascinating to watch how creatures evolve over time. Imagine this in the settings of a MMO and I am sure it will create immersion for the player, but it would also mean the game to be different according to when you joined, tough call.

In the end though, some restrictions may apply, according to how much processing power you have on the server side, if you have 200'000 NPCs in your game world, you'll probably downsize the AI a bit to match CPU power versus NPC numbers, I am eyeing the shared process structure as mentioned above.

Cheers,

Sunweaver

News on ANNE

2009-07-16T11:04:00.019-05:00

The toughest task in a programmer's life is not the actual algorithm but the bug hunting, I noticed that something was still weird in ANNE and I found some more bugs in my code, but well.

I implemented some improved trainers for XOR operation and for the emotion part, but this time not by running a fixed number of epochs, but by checking for the LMS (least mean squares) of the training sets, which actually works nicely so. When setting the trainer to 0.01% LMS: ann structure = input:(hidden layers): output

* XOR works after 11422 epochs with 2 hidden layers and it takes 11855 epochs with 1 hidden layer, 2:(2:2):1 versus 2:(2):1. Precision seems to be on par of both by 1%.

* Emotions are trained after 25201 epochs with 2 hidden layers and 18387 epochs for 1 hidden layer, the first is 6:(6:6):6 and the other is 6:(7):6. Their precision is about on par for learned values, but the first is much more precise towards one or the other emotion when interpreting new values, the second would allow for some more varied emotions (like primary and secondary).

My interpretation here is that while for most practical purposes a second layer is not required, it does add a lot of precision if you care for it and if you don't mind the extra time for the training.

I believe ANNE is now working correctly. Now I will start trying different AI scenarios for NPCs, it seems that ann's are suited to be the brains of a NPC, let's see.

Regards,

Sunweaver

ANNE to ELIZA: Think before you talk !!

2009-07-08T16:13:00.030-05:00

As I said before, there are so many, many components that need to be built for the complete SMASH, one that I have barely mentioned so far is the AI part, no game ever would be really complete without some intelligence. So let's talk smart today.

Eliza will most likely be part of the NPC perception system and and it will be important for stateful conversations with the added context info (which will later form a sort of decision tree) and when I coded Eliza, I had the idea of also including something else, one thing that I find about basic Eliza is that it feels too cold, there are no emotional reactions behind it. We humans associate words with feelings, like it or not, so I started looking into that.

Now, how do we translate input into emotions ??

In order to do that association I dug up a couple of AI books on my shelf and found a nice model about how a NPC (I recommend the book: Programming Believable Characters for Computer Games by Dr. Penny Baillie-de By) can translate situational information into emotions, which turns out to be an artificial neural network (ann), so after some real messy programming sessions ANNE was born: Artificial Neural Network Entity.

ANNE is a fully connected multilayer feedforward network with normal error backpropagation. I will spare you yet another explanation of how ann's (also called sometimes multilayer perceptrons) work, mathwise, I found however this very insightful page: Introduction to Neural Networks which does an awesome job at explaining them.

While it's tempting to implement each neuron in Erlang as a process, I didn't, because I do not like to have processes just laying around doing nothing 99% of the time, rather, I implemented this as a list of lists, which can now very conveniently be saved to KVS*. This way you don't have to retrain it every time, just save a trained version on KVS2 and with KVS* replication, you now have the model ready to use across all SMASH nodes.

I wanted to see just how many neurons ANNE could support, so after some partial testing I can say that I was able create f.e. an ann with 500 inputs, 2 hidden layers with 5000 neurons each and 500 outputs, let me tell you that this a hell of a list of lists with a total of 30 Mio internal weights, so be patient during the training process. Most literature also does not speak about how many iterations are required to train an ann, but it would seem to me that it requires about 50'000+ epoch (epoch = 1 run with all training sets) to perceive a convergence of the weights. Erlang impressed me yet again, I was unsure if it could hold THAT much data in a single list .... but it did.

Now, let's get a feel for it: My first test was as suggested by Mrs. Penny a simple XOR operation (with 2 inputs, 2 hidden layers neuron in a single layer and 1 output neuron) which after sufficient training works just fine. Once it passed the initial runs I grabbed that emotional model described in the book looks like this (chapter 7.5.2):

What you see here on the left is the lower part of the trained ann with 6 inputs, 1 hidden layer with 7 neurons and 6 output neurons. The ann is then trained with 6 x 6 samples for about 50'000 epochs, returning the new trained ann, the output values should be loosely binary as intended, meaning 5 ceros and a single 1 value.

The call to anne:ann([INPUT], ANN) accesses the anns knowledge, so I fed the values from the book into it, knowing that those values will need to be interpreted and hoping that the 4th value is clearly the highest one of them, .... indeed the 4th parameter is the highest with 0.88 ... which is correct !!! In this example it represents: "fear", also note that the 3rd value is 0.24 which stands for "anger", the beauty of an ann is that it states the most prevalent emotions, but you could also take into account the 2nd strongest one, so maybe this input to the NPC will cause it to run away in fear cursing angrily.

I now need to find some example with 2 hidden layers to test ANNE on, but so far everything works just fine.

I have not decided yet how a fully grown NPC will have to work, it might be a sort of hierarchical finite state machine (HFSM) with modules of Eliza and ANNE integrated into it or just ANNE + Eliza, but I believe this is good progress already on the NPC AI.

One more nice thing is, if you embed an ann into a looped process you would only need a single instance on all of the SMASH servers for all NPCs to interpret that input. Maybe several anns will feed its data into a HFSM and make it trigger its actions and responses.

ANNE is still an uncooked meal, but the ingredients look fine. And that's all for now folks. Until next time.

Cheers,

Sunweaver

Eliza expands its mind

2009-06-17T18:25:00.011-05:00

I have looked at many Eliza implementations and most only match a single word in the sentence, or a word in the middle and the rest after the match, so I took it a bit further, I can now match stuff like: "I %% flowers **" and also stuff like "I %% and %% flowers **". This would catch several multi-word patterns in between and whatever goes behind flowers. Given that Eliza does pattern matching but the humans employ several words for the same thing if only your Eliza could group by similarities for example by ["love","like","fond of","don't hate","don't dislike"] . Another example: how many Elizas can tell that these are equivalent to a human: ["MAC", "MACs", "Macs", "Mac", "Macintosh", "mac", "macintosh", "apple", "Apple", "Macbook", "mac mini", "Macmini"] ?? So how do you catch different ways of expressing tastes with different nouns, that tell you more or less the same thing ??

Answer: pattern expansion !! I expanded Eliza's linguistic abilities to match similar patterns, you only need to create a pattern list of synonyms and specify that list in another pattern as an expansion pattern like "I ## ##" where ## stands for the synonym list, here is how:

To recreate the above example you would create 3 records in the Eliza file, the first is the pattern and the other 2 are merely variables.

{["I ## ## **"],
["I live inside a computer","I can live inside a Mac","Maybe I reside in a Mac right now"],
{subject,computers,["##1", "##2"],["like","Mac"],[]}},
{["##1"], ["love", "like", "fond of", "don't hate", "don't dislike"], {internal,synonyms,[],[],[]}},
{["##2"],
["MAC","MACs", "Macs", "Mac", "Macintosh", "mac", "macintosh", "apple", "Apple", "Macbook", "Ibook", "Macmini"],
{internal,synonyms,[],[],[]}},

So now the pattern "I ## ##" is expanded against both variables ["##1", "##2"]. which creates in this case 150 patterns in run-time, but your file only required a simple entry and again Eliza returns some values that your AI can now chew on. Those variables can obviously be reused in other patterns and it's needless to say that I will be able to teach her new stuff in run-time.

Things that I find are still needed be it either on Eliza's or the AI's side is to catch specifically nouns and verbs, maybe by matching against another list and trying to find a corresponding context for them, like when your application knows nothing about flowers, but flowers are plants and maybe your AI knows something about plants or maybe it lacks info on Mac's but knows about computers. And Eliza should also be able to ask the user and remember its question. But for the time being, I am quite happy with the implementation, it will suit SMASH well to have NPCs that the user can actually talk to. The current version is stand alone, future ones will likely put their info on KVS* for easy data access and replication.

Cheers,

Sunweaver

Eliza speaks Erlang with a twist

2009-06-13T17:17:00.045-05:00

As I said in the previous post, Eliza should be fairly easy to implement in Erlang and so it was, I present to you the first Eliza that speaks Erlang:

The image on the left shows the Erlang Shell where you can see some examples of Eliza being called as eliza:start(Text) [also possible: eliza:start(Text, Bot) (not shown) to choose some kind of personality]. Eliza will try to load its text file for interpretation from disk.
Eliza prints out in the shell and returns a tuple with: the input string, context information, extra words from the pattern matching and the answer from Eliza.

What for ?? A normal Eliza implementation has no idea what you are talking about, it does no provide or save the context of the conversation, so my implementation has a little twist, you can provide some additional info to each question-answer list, meant to be used by an AI implementation, so like above when stating your name, Eliza answers with {name,player} and the player name, so your AI could register that Eliza just got the player's name or whatever the player tells Eliza, like dislikes, likes, emotions, hobbies or whatever, your AI could then save the data to a database or you could also load a different Bot context file to chat more specifically on a topic. I will incorporate this in my AI modules in the future and in the meanwhile this will serve for some "dump" entertainment in SMASH. A miniature dictionary to showcase Eliza is included in the code. I also see a use for this on the CNPC's to detect automatically abusive or foul language, as mentioned on many previous posts.

If there is anybody interested in the code, let me know and I'll publish it here on the Blog, it's deceptively simple, just under 200 lines and not yet optimized, but it's a first step towards implementing AI in Erlang.

And now back to CAM.

Cheers,

Sunweaver

Huh ?? Did you SAY something ??

2009-06-09T23:33:00.029-05:00

As CAM goes forward, I am defining user permissions, channel permissions, guild structures and user levels which go from guest to game developer with each level having unique commands, which of course brings us to the point of what the final command structure will look like, I believe it's going to be something like: {CHANNEL:REQ:PARAMS} where each REQ that comes from the socket is seen as a request for action by the CC component before being sent to its Action handler (AH) and from there to the channel controlling NPC (let's call it CNPC for future reference). Only the CNPC's can return commands that need to be applied to each CC. Some commands will need to take into account distances, which will be done in the CC, commands like "SAY", movement and position data or Area of Effect spells, Thor gives in his book an example on how to do that kind of calculation easily and without spending much on calculations. Apocalyx will need to translate the server packages from {CHANNEL:WHO:{CMD1:PARAMS}{CMD2:PARAMS}{CMD3:PARAMS}} into action on the client and WHO is obviously derived from the CC.

Note to self: Commands are better off as binary structures or at least send the commands list with a number to the client and have the client use a number to shorten sent IP packets. Maybe SOX will translate and package this at some point into a binary format.

Once again I find that prototyping takes so much longer than actual coding, because it really pays off to think before you program and avoid conventional code at all cost !! Erlang rewards those who create the application logic by pattern matching.

Which reminds me, a nice little Eliza implementation for a CNPC might be fun to start with, for those who don't know Eliza, it was invented as a program that would apparently answer with AI to a sentence by analyzing the sentence structure, recognizing certain words and answer with premanufactured phrases, making people think they were talking to a person, well, in the 70's it was surprising, that is. It's still fun though and with little modifications, KVS could provide a short term memory for Eliza to keep track of what users talked about. You may notice that pattern matching is precisely what makes Eliza strong ;) and hence should be trivial to implement in Erlang.

One last side note, I have not yet replaced Mnesia in SMASH given that I am still debugging KVS*.

That's all for now folks,

Sunweaver

Tuples here, there and everywhere

2009-06-08T08:47:00.014-05:00

As I am trying to make some progress on the authorization module, I find that the required databases are getting more and more complex and with them the record structure, meaning that tuple pattern matching is quickly becoming a logistical nightmare, stuff like {_,_,_,_,Variable,_,_,_,_,_,_,_,_,_,_,_,_} = Some_tuple is not very practical. Erlang has a native record definition, but it has some important restrictions, so I had to create a record structure for KVS2, that can correctly create, modify and read records in KVS2 by naming the fields, for this you need to create a list entry on the 2 DB structure tables. After that, you can simply call kvs2:crtrecord(DB,Key,Data), where DB is the table & table structure, Key is the record name and Data is a list of tuples of {field_name, Value}, you can read with getrecord(DB, Key, [Fields]) and modify an entry with updrecord(DB,Key,Data), which works similar to crtrecord, you only need to specify the fields that will be modified. This greatly simplifies data management and allows to change the database format in the future, too. You can also use this functionality to manage any kind of tuple with crttuple & updtuple. This should provide some more flexibility than native Erlang to manage structured tuples. With this addition KVS2 can now handle Key-Value pairs with structured value tuples.

This tuple management and the magical pattern matching will make authorizations ... almost ... a breeze.

BTW, for those of you who can already employ version R13, note that there is a new function called: erlang:make_tuple(Arity, Default, InitList) which can create tuples with initial values, very nice addition, unfortunately Ubuntu has no support for R13 yet (unless I compile manually , yuk!), well, you can't have everything I guess.

Soon more on the Chat Authorization Module (CAM).

Cheers,

Sunweaver

Regarding Status Post

2009-05-26T16:28:00.006-05:00

I believe the Status Post (below) was long, long overdue. It describes in more detail the structure used in SMASH. Comments are forbidden on it, so please do so on this post, the status report will stay linked on the right and change according to progress obtained.

They say a picture says more than a 1000 words, so I hope that this makes all the other postings crystal clear and show what piece fits where or maybe it eliminates the last bit of clarity you had on SMASH.

I believe another post with screenshots of Apocalyx and development pictures might be nice, too, maybe the next post.

Cheers,

Sunweaver

SMASH Status

2009-05-26T10:30:00.026-05:00

Last edited on 25.06.2009

Introduction
This document is a more technical description of where SMASH stands, describing the internal flow and architecture. Please make sure you have read the Introduction first. If you are looking for an interesting read, I also recommend Thor Alexander's two books "Massively Multiplayer Game Development", I use them very heavily for SMASH's development, knowing the books will make it easier to get a grasp on certain concepts. This document will change over time as progress is made, so today's read will be different from tomorrow's and then again, maybe it won't.

The Goal of SMASH
Once SMASH is finished you will be able to create a Massive Multiplayer Online Game (MMO), a messenger-, a dynamic webpage-, an IRC- or SMS-network or anything that requires sending messages across groups of users on different servers, while being completely independent of the platform, as long as Erlang compiles on the OS. SMASH can be upgraded in run-time without missing a beat. The application behavior is dictated by the channel controlling NPCs or Sim Proxies (see below). This is why you will hear me call SMASH a "framework" here and there, because it's mostly a generic communication network on which you can build whatever functionality you require.

The SMASH Concept
At its very heart, SMASH is a chat network, which was inspired by Joe Armstrong's example from "Programming Erlang: Software for a Concurrent World". Joe be at ease, I replaced all your good code with all new, lousy code.

But anyway, every event in SMASH is essentially a message sent from one participant to another. Messages are always tell commands, either to another player or a virtual player, also called NPC. The means of interaction with several other players and with the game world is via chat groups, which are controlled by NPCs, those NPCs are fundamentally the same object than a player proxy, but with a different plugin to control its behavior. To chat in channel "X" the player sends a tell to the NPC "X" which validates, if needed, the intended message and then relays it to all other subscribers of that channel "X". So, for simple chat channels those NPCs might be doing nothing else than relaying text or they might filter offensive messages and raising flags or mute a player for a while. NPCs will handle zone behaviors, by receiving requests from players, validating them and relaying resulting actions; other NPCs in that zone will be implemented as simple scripts or as other NPCs, that receive messages like any other player and act upon them. At one point in the far future, all NPCs might be virtual players, have a decent AI and really "live" in this simulated world.

Zone NPCs might want to communicate with a Physique server to validate movement action or with other zone NPCs for overlapping zones, or maybe you have a server where a simulation runs and have that simulation feed its data into SMASH; Leo, author of Apocalyx has done a great many demos, so maybe I will modify some of his demos to transmit their actions via TCP/IP and players would be able to watch one of his soccer simulation, or car races or robot fights. Time will tell.

How does a single SMASH node work

In practice a single commands in the Erlang shell starts the master node which triggers certain start-up functions and initializes the framework. Each new node starts up as a slave node, the services get started from the master node automatically.

A client connects to the SOX server (SOX) which returns a client socket for each new connection and notifies the chat server (CS) module of it. The CS starts a new chat client (CC) or client proxy, which is the representation of a player in the framework, so if someone disconnects his virtual representation persists for a programmer defined interval. So if someone disconnects during a fight, his toon will persist and most likely die, disconnecting won't save him, some online games omit this. The CC can have a plugin (action handler) linked to it that will validate incoming messages, so one moment the toon is under player control, the next it might be doing some scripted action, or a NPC lives in this game world driven by this kind of plugin support.

The-soon-to-be authorization module will automatically join the player to the channels he needs to be in and grant him the authorized commands with which he can operate in this virtual world. For each group a specialized NPC is started that controls the behavior of this channel (Sim Proxy). It might load a Logic Module for it, connect to a Physique server, load scripts to simulate objects in the zone, a dictionary to filter offensive words or even connect via TCP/IP to an application written in your favorite programming language. At some point some NPCs will need to start by themselves to simulate an on going economy, weather or connect to another simulation.

The command structure will be {CHANNEL:REQ:PARAM1;PARAM2; ... PARAMn} from the client side, where the client can only send requests, the CC will validate if a user is authorized to that request command and if so forward it to the CNPC, which will be the unique instance to convert that request into some action and send a chat command back to each client in its channel. The CC will pick this up, modify its value on the serve side and send back the command to the client, which handle the commands with the following structure: {CHANNEL:WHO:{CMD1:PARAM1;PARAM2; ... PARAMn}{CMD2:PARAMS}{CMD3:PARAMS}} . This protocol is currently plain text for easy trouble shooting.

Underlying to the whole framework are some services like the Observer (OB) and Load Balancer (LB) that will restart broken services, auto start code on new nodes and decide where processes should run according to available resources. The lowest layer so far is the database, which will be migrated from Mnesia to KVS*, the Key Value Services. The envisioned structure is KVS for temporary NPC data or to cache behind the result of a function call to speed up certain things. Its data is also replicated among all nodes by KVS2. KVS2 on the other hand is a RAM resident, self replicating database. It saves data periodically and reorganizes by itself should the master node fail. In the future the OB and LB will follow that behavior to create a fully self reorganizing network. All relevant data will be thrown from local loop data to KVS/KVS2 to enable a swift restart of fallen services without any data loss. KVS* requires that SMASH follows the WORM principle (write once, read many) where only one writes a record, but any other can read it, several processes writing to the same record will mostly certainly lead to disaster and chaos. On the other hand KVS* enables SMASH to have a fast replicating database without write locks. KVS3 does not exist as of now and maybe never will. The 3rd database layer is always handled in a separate module, programs must never write directly to the master database. Currently this 3rd layer is mnesia, in the future it may be KVS3 or a separate database server cluster which will be accessed through a TCP/IP, bundling access to that database into one module makes it easy to change without breaking everything. This layer is meant for persistent data that requires 100% accuracy and has no special speed requirement, like billing data, realm data, account information etc. This layer will be fully implemented when everything else works.

As you can see, there are not all that many pieces to the puzzle, the intention is to have as little and as few but versatile parts as necessary. And from those right-sized buildings blocks we shall make something large, yet manageable (hopefully). Still missing here are AI, state machines, scripts, ... you name it, a lot of stuff.

How do several SMASH nodes work together

As stated above SMASH is meant to handle many servers as one unit, so on each node the same services run and communicate through the unique Sim Proxy to each local chat group object, which in return notifies each player or virtual player (maybe smart and not zone bound NPCs).

Erlang-wise these nodes need to be able to ping each other. This inter-node behavior already works. What's more, you can compile and update all of the code in run-time without missing a beat and given that Erlang code is platform independent, SMASH couldn't care less if a node runs on Windows, Mac, Linux, BSD or "Your-favorite-OS-here", a small tip here, linux sockets have proven to be roughly 3x faster than Windows sockets and beginning NT4 Workstation Windows sockets will only serve up to 10 connection concurrently (on desktop versions), while others will stay idle, I don't think this has changed, so you will need to use a server version to handle many clients in parallel or use some other platform for massive amounts of connections.

The vision of the finished SMASH framework

This picture shows clients connecting to a global TCP/IP server which reads global data and decides where the user was at last. So each SMASH cluster might be a zone, an instance, a realm, a site or a different game altogether. The clients would always see the same IP it connects to, hiding the full infrastructure behind it, although this is still talking about the far future, this might well be the culmination of the whole project, a sort of "SMASH Game

s Googolplex", let's call it SMAGG in short for future reference.

At this point I have no idea how easy it will be to write a game in SMASH, once the CC plugin support is due, I will see to this, SMASH will most certainly not be for script kiddies, but rather for seasoned developers with advanced skills, but I sure hope that the framework will lift the tons of weight for you to make a large scale project happen. Once finished I will write a Database editor that will hopefully have the power to create a full game, we'll see.

The current size of SMASH is under 1MB (source +compiled programs), so as you can see Erlang rocks when it comes to code size and I believe even once finished the compiled programs will be around 1 MB. The largest part will be most definitely the database.

Currently undecided is whether it's going to be open-sourced, if it will be worth while writing a book about or if I'll just scratch everything and plant a tree or two.

Contributions or how you could help

Well, while I have no real time table, given that this is just another hobby, you might be able to help, if you have a/some public domain ... :

1) city sized BSP level(s) I might use as a zoned demo level(s).
2) action figures for the demos, be it cowboys, spacetroopers, aliens, elves, etc.
3) any kind of 3D objects, animals, furniture, etc.

I am not really a 3D artist, so any free material might help, if not I'll just use Apocalyx demo material and when releasing a first playable demo ask the user to use all of Leo's .DAT files.

Signed,

Sunweaver

There is always another way .. again

2009-05-18T09:51:00.022-05:00

Ulf was right, io:format(user,Format, Args) does print on the console it's running on, which works out so much better for me than having to install any code previously on a VM. There is always another way, nice trick, thanks !! I will leave the code below anyway as a proof of concept that I did at least once in my life OTP compliant Erlang code :) !!

if anybody out there knows how to make a node ping the master node from the commands line, please let me know, so far I can only make this work with -mnesia extra_db_nodes [] -mnesia start. I tried -net_adm ping , but it does not work.

My favorite fun function of the month:
[net_adm:ping(X) || X <- nodes(known)]
While I could not have any exact documentation on what "known" stands for, it seems to be a list of all nodes from previous start ups, hence this function pings any previously known node and the result looks like a Wild West Showdown =).

But anyway, with the above modification in place, once a new server connects to the master node (where the load balancer runs) the code gets copied and started automatically, the new KVS2 database gets sync'ed and the output is now relayed locally, it works like a charm. KVS2 updates now always use the most recent record.

Currently in the works is the authorization scheme to channels and what users can use what commands. I am "erlagnizing" the concept to make pattern matching easy. I believe it's going to be divided into 4-5 table, a user table with groups/channels, guild info, allowed commands and revoked commands, an account level, factions and other stuff. Another table with the channel permissions, a table for account levels to differentiate guests from normal players or supervisors, a factions table, a guild table and last not least a commands resolver table that assigns commands permitted according to account level and channel permissions. Commands, will be the heart piece of the framework to enable players to move, fight, eat, sell, chat and other stuff, or supervisors to create new contents in run-time. The revoked commands are meant as a feature f.e. to silence a player, or make drinking during fights impossible, immobilize a player and other mean administrator things.

Cheers,

Sunweaver

New remote process starter module

2009-05-14T17:25:00.007-05:00

Although process starting has never been an issue, I find one thing not so cool in Erlang, all io:format output gets sent to the node where you started it, which is very inconvenient IMHO, so I created something simple and new: the smash proxy (SP), which is a remote process starter. Imagine you already have a set of servers running and want to join new ones with everything and starting the services on them remotely, so you monitor new nodes coming online, copy the code over to them and start processes on them, but this time around, all messages they generate will be displayed on their console window, not on the console you triggered that from.

When starting up a new node (slave1@pcname) start the SP with the command line options "-s sp" or "-s sp start ", the first option will start the service, but it won't join you to the other nodes, the second option will generate a net_adm:ping(

) saving you from having to do it manually. From this point on you can start processes from another node on this one by sending rpc:sbcast([slave1@pcname], sp, {start,{Mod, Fun, Arg}}) [note hat Arg must be a list, as it is passed on to spawn] to it and SP will start the process for you locally. So it's a remote local process caller.

Here is how:
1) Create a folder where all the other Erlang OTP applications reside, I called it "sp" (app root folder), underneath that create a folder called ebin and one called src.
2) create a file called "info" under that root folder and put these lines in it:

group: tools
short: A generic SMASH process starter

3) go under source and create a file called "sp.erl", then throw these lines of code in it:

-module(sp).
-compile(export_all).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
rpc(Mod, Fun, Arg) -> rpc({start, {Mod, Fun, Arg} }).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
start() ->
register(sp, spawn(?MODULE, loop, []) ).
start(Master) ->
register(sp, spawn(?MODULE, loop, []) ),
io:format("SP: Spawning ping to: ~p~n",[Master]),
?MODULE:rpc(net_adm, ping, Master).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Normal rpc call to this same node
rpc(Query) ->
sp ! {self(), Query},
receive
{kvs, Reply} ->
Reply
after 3000 ->
{error, noresponse}
end.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
loop() ->
receive
{Rpcpid, {start, {Mod, Fun, Arg}}} ->
P1=spawn(Mod, Fun, Arg),
Rpcpid ! {kvs, P1},
loop();
{start, {Mod, Fun, Arg}} ->
io:format("SP: Spawning ~p ~p ~p~n",[Mod, Fun, Arg]),
catch(spawn(Mod, Fun, Arg)),
loop();
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Standard interface
quit ->
io:format("SP: Smash Proxy shutting down=~p~n",[quit]),
{ok, true};
upgrade ->
io:format("SP: Smash Proxy upgrading ...~n"),
?MODULE:loop();
Any ->
io:format("SP: Smash Proxy received Msg=~p~n", [Any]),
loop()
end.

4) Now create sp.app under the ebin folder:

{application, sp,
[{description, "Serves to start applications remotely"},
{vsn, "1.0"},
{modules,
[
sp
]},
{registered, [sp]},
{applications, [stdlib, kernel, net_adm, io, mnesia]},
{env, []}
}
]}.

5) Create a file sp.appup under ebin:

{"1.0.0.0",[],[]}.

6) Compile the erl file and make sure that the generated beam ends up under the ebin folder.

You are done now, call the proxy at start up as described above and you can now do remote starts with local display of data. The src folder is not required on each node, only the ebin, so make sure to throw it into each new node installation and you are done. Or take the erl/beam file only and start the Erlang VM from the same folder, it does the same job. If you ever need to upgrade the code in runtime, then replace the beam file and send a command like this to it: sp ! upgrade to renew the code or do a rpc:sbcast to all nodes, as you please. I hope it helps for your projects, too.

Cheers,

Sunweaver

KVS2 continued

2009-05-11T12:22:00.017-05:00

I have extensively tested the stability of KVS2 and I am almost satisfied. The infrastructure is rather difficult to kill now, which is a nice thing.

All data written now stores when the last update happened to it and when the data is due to expire, to enable short term memory data functionality and in the future sync'ing data among master nodes. Now let's invent a new term, each database has a certain foundation and principles it's based on, mnesia f.e. defines that as the ACID properties, eg. atomicity, consistency, isolation and durability. Given the design choice of KVS2 we define:

The WORM principle: or "write once, read many"
Only one single process writes a record, every other process can read from it. If several processes write to the same record you will have at some point inconsistent data, given the lack of write locks and random order writes.

It would be more appropriate to call it " one writes, many read", but who can remember the acronym OWMR ??

So, f.e. loot data must be implemented on a zone level and hence be managed through the "zone player" process and auction house handling must be implemented as if it was a zone, WORM for the win.

If we apply the WORM principle to our programming style, in theory, we preserve the ACID property that mnesia enforces on application level and we add an "S" for speed to ACID, becoming ACIDS, because mnesia fails to live up to the part where it should be real-time.

Writing a distributed database is a lot of work and fun. I wonder, how long will I be able to do without write locks ?? Hmmm !!

Improvements needed:
1) Make sure that a newly elected master node is really running --> with a separate process on each node that verifies the masters list, its status and trigger a reelection process when needed
2) During start up, ping all previously know nodes, try to find running instances there, get more node names, ping them and only then start up or else you get several masters elected

Cheers,

Sunweaver

Unintended and useful feature

2009-04-29T17:32:00.005-05:00

As mentioned in the PD below now that both modules KVS and KVS2 are working it is very easy and actually extremely useful to integrate both. I believe the best way to do that is to make KVS post into KVS2 and once the cache is expired, delete the data again, hence creating a distributed "short term memory". This looks like complete awesomeness for NPC memory data, imagine a NPC being killed over and over by the same player, he could remember him for a while or to avoid NPC grieving, maybe it elevates its level each time, quest givers remembering for some time who visited them, a reminder when somebody casts a spell or the spell cooldown time. Sounds like fun and it was fairly easy to implement, too. A couple of lines to check for KVS2 and then shoot the data to it. It had not occurred to me before, but our brain also has like 6 different memory types, so why not simulate it. And of course, the data is replicated to the other nodes that run KVS/KVS2. Now I am almost ready to replace mnesia.

And on another note, since I migrated to Ubuntu, programming is so much more fun, this is how my desktop looks like, you can see the cube spinning with an aquarium in the mid, a text windows for code edition and on each plane a separate Erlang VM, very nice, programming in 3D, not on the level of Swordfish, but well almost. Oh and the background is made from an Stargate Atlantis "the Sunken City" wallpaper I modified to become a very nice looking seamless skydome.

Laters,

Sunweaver

Replicate this !!!

2009-04-24T08:26:00.005-05:00

Before we start, I installed the SMASH framework on a Linux machine and boy are those sockets fast, the message sync error commented in the post below does not occur, curious, latency speed gain 3:1 over Windows sockets, I am impressed.

But that is no the main topic of today, I have gone over the edge and I am currently programming a 2nd Key Value Server (KVS2). The original KVS server is a Key Value (or Name server), that caches function call results for x amount of time recaching every n seconds, the KVS2 is different. KVS2 is designed to be a database, aka mnesia, replacement.

Mnesia is a very cool design and I like it, I especially love the fact that f.e. RAM nodes can access the DB just by declaring where the master DB is, you can fragment a DB over several nodes, it has transaction capabilities and it can reside on several nodes. If you have a lazy look up application that is awesome, but I found in real life that usually the different nodes cannot keep up with transaction volume of a MMO framework in absolutely no way, if I create one master node then I face the challenge that when a slave node does a transaction it can take up to several seconds to get a result back and if the master goes down, well, so does the whole system, if you define several masters, synchronization takes too long, way to long and if one node goes down the other starts complaining and blows up.

So here comes the salvation, I hope, the birth of KVS2. The code is distributed through the LB to the code and auto started. The first node in the system reads its data from text files and other slave nodes copy the tables over, each table runs as a named process. When you change or delete data that node will issue a broadcast to the other nodes to do the same transaction, hence they sync and each node enjoys having a local cache, speed, speed, speed. If the master goes down, big deal, the other KVS2 servers are linked to the master and detect its death, so they choose a new one, so far they pick the new master simply by name, I could also very easily ask the LB and cache with KVS the value, so that they all get the same answer, but ehm, I don't. This node is then declared the new master and end of story. The master saves the data every 5 min to disk and you can even force a node to be the master, like when the original comes back up. A word of warning here, if several nodes write to the same record, you can have confusing results, as there is no check that all nodes have the same data, if messages for the same record arrive in different order, you can have different records on each node, be careful, this general scheme assumes that only one process manages a certain record, like for example in SMASH, only supervisors write records. Yet when accuracy is critical, it is very easy to send a message to the master node, the rpc interface handles that, so it can send it to the master and the master will then send it around, an easy way to ensure transactions or uniqueness without conflicting records. Simple, short and powerful. Obviously this still needs to be tweaked and debugged, but it is already working. I will later on integrate a time stamp to enable servers to sync data if someone has more recent data, but for the time being I believe this new scheme solves my database requirements nicely, I have a local cache, self replicating data and a system that will work until the last node is shut down, removing all mnesia draw backs.

I will also integrate, somehow, preferred servers to be master nodes, which should make it easier to know where the files get saved to. The keen observer may have noticed, that we lose any kind of indexes and filters, for the time being. Filters are easy to implement actually given the [X || {Y,X} <- List] functionality, normally we know the key "Y" and get "X" as the value, but we can filter like this:

Value=[Pid || {Pid,Node} <- Answers, Node=:=Filter]

grab the PID from a tuple, when we only know the node name passed to the function as "Filter". Here we grab the value as the search value.

And there will be no indexes, it is cheaper to define a small routine that creates a separate table instead. So, a couple more days until I have the KVS2 debugged and then off to replace mnesia completely with calls to KVS2, I have to replace something like 100 calls, quite doable actually.

And then I will implement the authorization scheme, urgh !!!

Cheers,

Sunweaver

P.D. Debugging is almost done, all possible things seem to be accounted for, given the intrinsic structure of KVS and KVS2, they work like a short and long term memory, hmm I might call them both in the future Skynet Database, sounds better than Key Value Server, doesn't it ??

There is always another way

2009-04-17T09:37:00.009-05:00

"There's an Italian painter, named Carlotti, and he uh, ahem, defined beauty. He said it was the summation of the parts working together in such a way that nothing needed to be added, taken away or altered [...]" Cris Johnson (Nicholas Cage) in Next

Good thing he did not talk about our code because each time I look at it I find another way of doing things, Erlang is really much faster when it matches patterns rather than using traditional programmatic code, so perfection or beauty is still a long way down the road, but anyway.

RTFM or ... how I should really read the manual more often, I had completely overlooked the function net_kernel:monitor_nodes(true) . When you subscribe to this function, in our case the load balancer (LB) does as of now, you receive a message {nodeup,Node} the instance a new node comes up so now the code gets copied and started instantly on new nodes, it couldn't be faster than this, if you blinked you just missed the copy+start process. A message of {nodedown,Node} tells you when one disconnects.

I also stumbled upon another very useful function called rpc:sbcast(Name,Msg) which sends a message across all connected nodes to a registered named process. Until now the group process caches a list of all groups on the different nodes and new groups can take a while (up to 20 sec. so far) to be seen by others nodes, but given that we now register each group on each node, this function could speed this process up, without relying on cached mnesia data, my only questions is, what happens when 2 nodes do not yet see each other ??

Speed-Ups: Every time I look at pattern matching I learn an easier way to do something, when I first started I was tempted to use traditional loops like the function lists:foreach then I switched to:

lookup(G, [{G,Pid}|_]) -> [Pid];
lookup(G, [_|T]) -> lookup(G, T);
lookup(_,[]) -> [].

I find that structures like the one here are even faster than the other two:

[X || {Y,X} <- L] where X is our return value , L our list to look up a value from and Y our variable to pick the correct tuple from a list. This function is so fast that it is causing me some synchronization problems among my processes for reasons yet unknown. This function will also serve as a command interpreter, when the client sends a string of commands, like "{cmd1:arguments}{cmd2:arguments}", I convert that list with a function I designed into a list like this [{cmd1,"arguments},{cmd2,"arguments"}] and I can then use the above [fun(Y,X) || {Y,X} <- L] to run some function on each element in the list. I might even displace Mnesia altogether, just load data in lists into the KVS, look up / filter values with pattern matching and sync the nodes with rpc:sbcast. I could use the module dict to create a dictionary instead of a table or list, too. Each dictionary could run in its own thread, hmm interesting. This sounds so crazy, it might actually work.

Remember that Mnesia is nice when speed is not critical, but when you need to do thousands of look ups per second across "n" nodes then the generated overhead is just too much, mnesia can't keep up, no matter how I structure it, be it several disk nodes, 1 disk node several ram nodes, the worst thing is the lack of a local cache on a ram node, hence the KVS working as a local cache. From a mnesia centric application framework I am getting more and more towards a KVS centered data structure, the above scheme might eliminate the caching errors and I do not require really the supertight transaction property that Mnesia offers, I'll gladly trade that for speed.

Designing the authorization module (AUT): it will mean a large impact, because I will need to define a whole lot of logic around this, including what commands each user can perform, guild channels, raid & instance channel, guild structures and more, so this will take some time to complete, but the design isn't finished yet, I am still prototyping the logic. Some nice features that will most likely come up are f.e. you will be able to define as many guild channels as you like and define their authorizations to differentiate the general channel from officer, class lead, raid or other group channels below the guild. I will also consider instances and raid channels with ownerships, to prevent problems like an invited outsider stealing raid IDs, a documented problem that happened to other large MMOs. Under the devised scheme an invitee will not have ownership or admin rights and it should be an easy thing to do that certain instances must be started by the owner/admins and hence avoid getting a started raid instance stolen. The user commands wil also be important to avoid that a guild master leaves the guild without reassigning ownership to somebody else, so a GM will lack a command of GQUIT, but have a GOWN, while other members will have GQUIT, but none afiliated will have the GJOIN command sombody unafiliated will have. The authorization module will need to combine speed with functionality so a huge fun part is coming up here.

With the speedups like the ones above, I am now facing sync problems between threads (that should not exist on the first place), the first or first few messages from a client get lost in cyberspace, I know that this is due to me cacheing values from mnesia, so I am thinking that the easiest way to avoid lost messages will be to implement a ping-pong protocol to allow the client to ping the server (once authenticated) until he gets a pong response and only then start asking the server for more stuff or start chatting. The cacheing delay can be anything from unnoticeable to several seconds.

Another curious fact is that the more functionality I design the smaller the modules get.
The module CS which was in the beginning the master of the whole framework is becoming less and less important, with groups registering now, it is not much more than central resource locater/creator. The CG module is now the local node relay station, down from channel admin, the logic has gone to the channel NPC. Both are still very much required, but their role has drastically changed during development, curious.

Cheers,

Sunweaver

Auto Config ... baby steps

2009-04-13T09:55:00.004-05:00

While bug hunting my code I came back to one thing I always wanted, have new nodes initialize themselves, like Skynet, becoming aware of new resources and start using them. Erlang provides a so called boot server, but no documentation exists that would clarify as to how it works, only one developer posts like half the instructions as to how set it up, but I could not make it work, I get to the point where the boot server starts, it sends the boot file and the remote node exists with an initialization error (and yes I did follow the instructions and the boot file oly contains the Erlang required modules), even analyzing the source code did not make a difference, although I found some undocumented calls, like erl_boot_server:add_subnet(mask,IP) and :would_boot that add a whole subnet to the boot server, compared to the normal function that requires you add each single IP address, which makes no sense to me and would_boot tells you if a certain IP could connect or not, fun functions, if only somebody would comment on how to set it up the whole way.

Anyway, I said to myself, there must be another way and of course I found one. The Load Balancer (LB) on the master node now checks for node changes every 10 sec. and reports on nodes lost (without any action so far, in the future it might restart lost processes on other nodes), nodes that stayed up (they stay untouched) and new nodes. If a new node is found it verifies rather crudely if SMASH is loaded on that node, if not it copies the code to it and starts afterwards a slave node and tries to fire up the socket server as well, this will fail for secondary nodes on the same machine, but that's ok, the supervisor table keeps track of which one stays up, so no problem here.

What this means for the framework is that you only have to start up and connect a node to the master and not worry about anything else, I love self configuring or so called zero configuration programs, so this is a step towards it; f.e. at some point when you want to fire up simulation parts and use the LB to pick a node, you know that the basic framework is already up and running, which is nice.

Notes to self: 1) keep a record of the internal and where possible external IP address of the SOX instances, maybe the No-IP Service can be helpful here. 2) Some work needs to be done now at the observer program to auto-restart lost processes. 3) we now create a player instance for each channel, so we probably only require to track those for the simulation, maybe every xx time we save their data to Mnesia and/or to the other group (CG) instances for pure safeguarding, both would work, Mnesia is probably the slowest yet cleanest approach.

Cheers,

Sunweaver

There is a bug in the electrical system !! (Agent J)

2009-04-07T22:05:00.004-05:00

With the new functionality in place some rewrites were necessary and some debugging of course, *sigh*. It's impressive how many bugs one can find in his own code, I did quite some trouble shooting these days and I have finally found some strange disconnect errors, where a client would randomly DC for no apparent reason, but my latest stress tests with a couple of test clients bombarding the SMASH framework are keeping up nicely, so after the first 30 Mio I will make the audacious assumption that the bugs are now gone, the counter will hit 100 Mio sent messages across the system by tomorrow, I hope for the best =) !! Good news also is that the system is much faster than before and the KVS is shining in our quest for speed lookups. Hmmm, now I will need to read about LUA code again and prep that Apocalyx client for testing.

Cheers,

Sunweaver

P.D. 100+ Mio messages ran through SMASH and it keeps running and running, the bugs have been eliminated and the framework is now more than ever a stable multi server soft real time messaging system, so far so good. =)

And: Action !!

2009-03-31T11:39:00.006-05:00

It is 2009 already and I have finally some time again to keep programming.

Action handling
After much thought on how to implement the application behavior, currently all chat was sent around without validation and a MMO needs to have an application logic behind each channel, so specific action handling needed to be implemented to process each client info before sending it around in the channel. I have come to the conclusion, that probably the best way to do that is to implement them as a player (CC) module with an action handler (AH), both are stand alone processes, so whenever a group gets started up, it will automatically seek in a table for the AH of that group or else start up the generic entry as the handler. ANY message to be sent in a group gets sent to that single automated player entity, which then decides how to proceed, so for chat channels one can implement a word filter, NPC data can be implemented from here or zone channels will be able to send the movement data first to a physique server (far future) before sending that data to the clients. This single AH per channel will also keep track of connected clients, no matter what node they come from.

Some lessons learned of programming in Erlang:
First lesson: there are so many ways of implementing a certain behavior, it's mind boggling, one can choose from creating vanilla subroutines, a cherry mono- or multi-tasked applet, a table entry, a DB table or whole self managed entities, the choice is really vast and the problem is that you can be quite dynamic, so I have to sometimes go back and reread where all the magic comes from, all or most choices seem to be on par speedwise. Once decided, implementing it is a snap.
Second lesson: right size your multi-tasking; it's so tempting to create many, many processes, but syncronization can sometimes be a pain, so double check when to MT and when no to. Even when sync-ing is no problem, don't forget the possible size of the outcome, f.e. in the very beginning I sent each chat command as a separate task, but when I did the stress test with 50'000 generated users over 5 servers I found that 2.5 Billon concurrent tasks was a tad too much to handle for the framework, I wonder why that would be, how lame !!! =P
Third lesson: And Erlang programmers will hate me for this: OTP sucks big time, I found that it is much easier to write my own functionality, including but not limited to code upgrades in runtime across all connected servers, which requires a Doctor title in OTP to get to work, but only little effort to programm oneself, nowadays I only kick in some very few lines of code and voila, it upgrades !!
Forth lesson: they say love comes quickly and so does complexity !! Programs in Erlang are short and concise, but when implementing something new I find myself rewriting code in 5-6 programs in parallel, which isn't harder, but very different from "normal" programming languages.

Next to come is a channel authorization table that decides at logon (or later) what client should connect to which channel(s). I will also implement channel authorizations for that, which will be interesting =O, some must be system assigned (zones and instances), others with password, by group assignment (like a guild f.e.), open or user managed, so this will be an important feature and also some piece of work as I need to rewrite some code for that. Before or after that I will implement the first NPCs with some limited actions, oh and I still need to get the object table into the Apocalyx client to handle mutiple objects. But after these tasks I have the basic capabilities for an MMO, that is good news.

And Hello World !! Since my last update I seem to be getting quite a lof of hits on this blog I can even google this blog, much to my surprise actually and scary, too. SMASH is not ready for prime time after all and it may still be a while, due to the fact that I only do this on my spare time. So for the newcomer: feel free to read the MISSION STATEMENT. This is still vaporware, but whatever is mentioned herein is actually implemented, so one day (hopefully this year) the framework will be up and running and I will set up a closed alpha server for some testing. But I will not haste anything, quality overrules time requirements.

That's all for now !!

Cheers

Sunweaver

No updates around here ??

2008-09-03T12:00:00.005-05:00

As it happens from time to time, excess of activities force us to put some of our projects on-hold during those moments, so for the moment there are no updates because of lack of time for development, but the project is not dead. Also, I recently stumbled across another MMO platform that comes conveniently with source code, which I am studying, too; the Mangos-Project. Interesting and inspiring stuff, although I won't implement their ideas into SMASH. I guess some more news will be out like at the end of September, with the multi-player display and some improved chat functions.

Cheers,

Sunweaver

Dynamic Environments

2008-06-02T16:48:00.005-05:00

There hasn't been any update in almost 2 weeks, I know, and there won't be much to talk about at the moment. I am trying to find out how to create Dynamic Environments, sort of "SimCity"-like dynamic scenarios, where you create buildings and everything. Simple as it sounds, this is quite a challenge for the noob-Apocalyx'er like me, so you can imagine the fun I am having here. What I have in mind is something like this:

1) Changeable height maps (at some point), at first the world is going to be flat as a flounder.
2) The Wild West: People build houses, villages, mountains, valleys, farming, dynamic trees, etc.
3) Mega Cities: Think of living in a Mega City like in the movie "The Judge" or "Star Wars" cities like Nar Shaddaa (cf. Wikipedia links).

Once we have the ability to just drop building blocks onto each other we can then build in runtime really large scenarios, with very high buildings, several levels, etc. This sounds very appealing to me and will allow for a more dynamic game play. What it implies is to create a collision detection per object, bounding boxes and a physique simulator, all of which are included in Apocalyx, it's just a question of finding out "how". =O

And well, I have yet to finish the routine to add multiple object to the scene, that will enable the above functionality in the first place, much work ahead and so little time, but then it's better to do it right from the beginning, instead of fixing it later on.

Cheers,

Sunweaver

And the changes go on

2008-05-19T10:35:00.005-05:00

There hasn't been much to show for the moment and for some reason. In order to progress on the GUI side it is much required to be able to know, who is present on the channel, so any object present in a zone needs to report it's there, so the CG was modified and now responds with the names of everyone present, this will now require an object database, where you can then look up what's what and who's who, so the client knows what to draw, most likely it will inquire each CC object to avoid DB lookup penalties. Once that is done, we can now proceed to create the SIM Object table on Apocalyx.

Also, another change was made to the CI routine, the one that compiles and installs new code on all connected servers, it now recognizes all Smash processes on all nodes and updates those processes that run old code, it does that by swiping the processes on all nodes and sending upgrade signals to the "old ones", just as a fail safe to make sure that all processes are ready for upgrade anytime. Updating the whole system in runtime has never been safer, remember that one of the goals is to update the system without ever shutting it down. And to all Erlang veterans: this is about a 1000x easier than using the OTP way, which requires a doctor degree to make upgrades happen. In this system just open a node for compiles, debug your code changes until no errors occur in the shell (or else running processes bomb out as per Erlang VM specs) and then type "ci()", which will distribute and upgrade all code on all nodes, now THAT is easy.

Cheers,

Sunweaver

Testing a BSP level

2008-05-12T10:59:00.011-05:00

As stated before, the next updates will be somewhat slower because of the obvious learning curve to understand LUA & Apocalyx. Some work has been going on to create a login screen, which is still incomplete, but it's underway at least a first glance: the background comes from the gui.lua demo and is using the GLGooey for the dialog (so far). The background will change, the important things is to try that GUI out.

As I tried to adapt a city level BSP, I ran into some noob problems, but fortunately Leo (creator of Apocalyx) helped promptly to solve them [ Thanks again Leo ], so here is what the same soldier looks like in a city:

What you see here is Leo's city from Urban Tactics. The toon's position now gets transmitted automatically every 200 ms whenever the position is marked "dirty", meaning, that there was a change and the 200 ms is the minimum interval permitted by Smash, send more often and your client will be in trouble, because the sockets are blocked for 200 ms (to avoid flooding) on the server side.

Currently underway is making the login work and creating the table structure for all network created objects like other players, NPCs, cars, trees, grass, boxes and whatever comes to mind. I am thinking of make certain objects client persistent, so the server sends those persistent objects once specifying model, position, orientation, size, expiration and an object name and all the following times the client only checks those objects by expiration date and object name only, instead of downloading the full set of data again, once I get the object table in place that is.

Live long and prosper,

Sunweaver

A first peek at SMASH

2008-05-07T08:33:00.012-05:00

A picture says more than a thousand words, so here it goes, this is the very first screenshot taken from the Smash Apocalyx Duo (should I call this SAD ??):

What you are seeing is a modified version of one of the basic model demos from Apocalyx with the console overlay displayed, where one can see the network information, in this case a successful logon, some positional data that the client sent and some other test clients who logged onto the server. Now we need to display other items, users and chat in this empty game world and a change on the framework (probably a whisper from each CG) to know who is on each zone channel, but we are getting there =) .

Laters,

Sunweaver

Still working on a first Apocalyx front-end

2008-05-05T09:39:00.004-05:00

Well, Apocalyx LUA is tougher than envisoned, we are still trying to produce a first GUI, but just not ... yet. A couple more days I suppose. So far we merged the MD3 demo with the network demo and the mini applet connects to the server, authenticates and the soldier runs around in circles, you can send position data at the push of a button and that's it so far. We are still fighting with the camera, which just won't stick to the avatar, but well. What's clear is that we need somehow a channel manager that logs who is on each server channel, so newcomers can query for other objects in the gaming world or we need to expand the CG threads to report that info back per node, it would be less error prone if something fails. And also required now is a vocabulary check at the socket server level, to filter out what commands a user can utilize, remember that users should be able to do different things according to their profile and/or authorizations. Also maybe a filter should be implemented that allows only one command of a certain category, like movement, chat, whisper, change equipment and such, or users might wanna cheat sending several movement commands at once and move faster than allowed, so that needs to be taken care of.

So the next steps for the week to come are:
* Sending Apocalyx movement, orientation, char and chat data around a chat channel
* Chose a BSP scenario, not just a blank endless terrain, maybe Genoa or city.bsp
* Modify CG behavior to include a report on who is on that channel
* Command permissions per user level and/or specific user
* Filter multiple commands to avoid cheating (+fix a CC send bug)

And that's plenty for now.

Sunweaver

The Load Balancer is ready for prime time

2008-05-03T10:37:00.004-05:00

The Apocalyx frontend is not yet ready, I am still messing around with that and well it's not showing yet anything, but soon, very soon. BUT, the Load Balancer (LB) is ready to be integrated into SMASH. The LB will check for speed on each connected node and how many processes are running, then it calculates what that means in terms of availability for the cloud. This game is a zero sum game, but for the final evaluation, every single node will count, negative values are not permitted, then the total percentage is done. For the end user there are 2 procedures to be called, remote rpc (rrpc) and remote spawn (rspawn). rrpc will spawn a procedure remotely and wait max. for one sec for the result or rspawn will spawn a procedure remotely and return the PID, in case that the procedure takes longer or you need to spawn a server. Those 2 procedures should do for the generic cloud service. This might even mean a strong impact on the Simulation itself that will run on the framework, the initial idea was to run a zone one a single node, but with the LB there is really no need for any kind of restriction, you could run the Sim threads all over the place. Let's see what that does for the framework :) .

Cheers,

Sunweaver

Some more remote utilities

2008-04-28T09:07:00.006-05:00

Some might know that SMASH has included into its belly an automatic updating system, which allows to compile all modules and reload them on all connected nodes in run time, without ever shutting the system down. On occasions this fails and a certain service shuts down. To get a hang on this, we have created some utility routines, one permits to execute commands remotely (on the other nodes) and the other shows if any module is executing old code, so very soon we will be able to first check what module on which node is executing old code before a new compile is allowed. So far the problem seems only to reside on longer lasting timers, which are not included in the update signal, all others update correctly (it seems). Also with these utility routines we could now easily start all preconfigured services (in the cluster file) remotely, without needing to touch then anymore. Some other fixes are that the CC proxy can now send tells or whispers without subscribing to any channel; KVS can now cache and forget, when you ask kvs:cache(M,F,A,Timer) it now will update every "Timer" ms and forget about the value after a minute, so the dictionary does not pile up with hundreds of unused values, this works like a garbage collection to keep KVS trim and slim. And more tests on Mnesia have been executed and curious enough on remote nodes the dirty function is slower than the normal database access with transactions and as to be expected KVS suffers from the initial Mnesia access, but afterwards blazes thru repeated accesses, so no matter how much we try to optimize the database access, the cached behind KVS approach is still the best when repeated access to data is required. Dedicated MMO literature also comes to the conclusion that processes on a multi server structure should update their data every 5 sec., less seems to be too much. And the cloud infrastucture has given some new ideas, Erlang already has the ability to spawn remote processes, so I guess, we just might implement our own flavour of it, by speed measuring each connected node and on the other hand check how many processes are already running there, this could be something like a routine task that executes per "X" interval and then when asked to spawn a process it checks for the best node to do so, a kind of "cloud_spawn(M,F,A)". This would make the SMASH ready to process any kind of task actually, not just a specialized Simulation Framework. It just might happen =) .

And Apocalyx can indeed use tables to store toon data, I played around with the Urban Tactics demo and it can easily handle some 200 toons running around there, very nice, now we need this networked. ;)

Laters,

Sunweaver

Does SMASH do Cloud Computing ??

2008-04-25T16:17:00.007-05:00

I was asked the other day if the SMASH does Cloud Computing or Grid Computing [cf. Cloud Computing from Wikipedia] and the answer is that while there is still a lot of ground to cover, in essence that is what the basic framework will do. The Load Balancer (LB) will check for the best resources to allocate (this is still future music) and distribute the simulation accordingly. The basic idea is the same, if you look at the graphic on Wikipedia, the cluster table (that we will have at some point and which will define how the cluster should behave) corresponds loosely to service catalog, the provisioning tool basically does what the LB will do and the Monitor is in essence what the Observer does. Once the pieces fall in place we will obviously also need a Systems Management component, which might be part of the cluster behavior.

So, given that this project was started like 4 years ago, the word "Cloud Computing" did not exist at that time, it seems to have formed during late 2007, but the idea of covering massive computations by networking them on many servers, instead of a large scale server is the same, so in essence I guess, SMASH is a Cloud Computing Framework and what's more, thanks to the unbelievable Virtual machine that Erlang runs on, we can already run cross-platform, the test cluster SMASH is being programmed on, consists of Windows, Linux and MacOS X systems.

Cheers,

Sunweaver

48 hours of endurance

2008-04-21T10:29:00.011-05:00

Alright, not much to report for now, except for some minor changes on the Observer side, which did not catch all required objects. And the channel management requires a major overhaul, whisper commands currently trigger being subscribed to that channel, which isn't exactly desirable, meaning that we will require authorization management, ouch. That's a major CG (chat group controller) rewrite, but anyway.

And now to the 48 hours test: yes the server holds up. The caching Key Value Server (KVS) makes the whole thing fly and no messages pile up, things have never been so fast, it does not matter where a client connects to, the SMASH framework relays the messages blazingly fast to other nodes, so we know by now. This concludes stability and speed tests for the time being.

Leo, if you are reading this, I sure hope that Apocalyx permits me to create each 3D object as a table entry and not just as a variable, like your examples showing variables like avatar0 - avatar6 and so on. I will be needing to create a LUA table and put each avatar as one entry and then update that table. I guess once your new Gun Tactyx is out I might try to "socket" it and permit people to connect to a fighting event, we'll see, so now back to learning Apocalyx. And yes you should be able to see some kind of graph here soon, to reflect what we got so far.

Stay tuned,

Sunweaver
EDIT: I shortened the post.

And there shall be speed

2008-04-17T08:42:00.006-05:00

It seems that the new caching KVS server does miracles, the message throughput of the whole framework went through the roof, up from a maximum of 7000 messages per second (averaging 2500 at heavy load) to a minimum of 150'000 messages, that's a 2100% increase, not bad. So, now we are indeed ready for the next step, Apocalyx here we come, the next thing will be to visualize our first GUI.

Sunweaver

Need for speed

2008-04-15T09:24:00.004-05:00

When doing the first message bombardment on the server and the news are mixed ones. On the one hand the server does not crash, which is nice, but after a while it starts to behave irrationally, like when logging out and logging back in, it does not seem to delete users from the DB anymore, so a kind of measurement is needed. Writing some quick benchmarks it turns out that the Chat Server (CS) can do 600'000 messages per second (mps) and the DB lookup in the chat group can only do about 6'800 mps, with 16 clients connected we get up to 2'600 mps, but I suppose that the constant bombardment becomes a problem at some point, because Erlang starts piling up messages and the missing client deletion from the DB is most likely the same cause, so a first conclusion is that Mnesia can't keep up with all those lookups on the DB and this causes messages to pile up like crazy. Fortunately this was likely to happen and now we have a legal excuse to use the KVS (Key Value Server), which has been completely dormant until now. The KVS will need to grab this kind of data from Mnesia and put it in a local table. The big difference now will be this, the KVS will only load the data from Mnesia every "x" time intervals, like maybe every second or every 5, meaning that we will lose precision in order to gain speed. We will now have to segregate our operations, those that require ultimate precision like close by moveable objects, banks, auction houses, trades and loot operations and others like chat, or even general environment operations that do not require the most precision. And we need to start with the Chat Group structure. This should be really easy to do, we will see if things speed up afterwards. A general throughput of 500'000 mps looks like our boy, but 7'000 mps does not.

And surprise, surprise, the Playstation 3 is not nearly as powerful as hoped for as a server, it's depending on the test 4-10 slower than a Dual core Windows Laptop, averaging so far like 6x slower, how very disappointing. In the CS test it scores at 95'000 mps, the DB lookup only gets 4'400 mps and it takes 24 sec. to create 10'000 CC clients, while the Laptop does that job in 7 sec., OUCH !!!!!

Long live the benchmarks.

Sunweaver

Notes on the work in progress

2008-04-09T08:53:00.013-05:00

Today we will get into some more detail as to what the WIP list means and check what comes next:

Work in progress (WIP) of the To-Do List:

* Create an action plug in template [easy / pending]
Currently a user logs on and a socket + chat client process are created, what is needed now is another process that will sort out what to do with the messages, what this means is that the socket is the non-transient part of the user, if he disconnects this process gets killed immediately, but the CC client stays there for "x" time, so a user cannot just logout when in danger and no harm will happen to him, this CC socket guarantees, that he cannot cheat by quickly logging out. The plugin is required to process his requests and whatever comes to him, this includes for example:

>> Action dictionary: what actions the user may execute, like chatting, moving, joining other channels, changing clothes and other stuff, so with that you could simply delete chat from his action list and mute the user, remove his movement ability and have him stand still or disallow a certain spell for a while.

>> Swap the action plugin: under certain circumstances you may want to put the user under script control, like for cut scenes or "mind control" a user by another user, so being able to switch the action plugin is helpful.

>> Put in the ability to have several plugins, this way we could f.e. log all chat activity, maybe this is not required.

* Internal user registration (to avoid same user on several nodes) [easy / partially done]
This is done for the CC processes, but needs to be done as well for socket processes, this shouldn't be too hard.

* Cluster behavior [medium / pending]
Here we might predefine what server does what task and what to do when that server is unavailable. This way we don't have to configure our cluster each time we start it up.

* Zone Simulation Manager [medium / WIP]
This part will include a chat manager for a certain zone and to extract from the database all the objects that are contained within it, like fixed objects and moving objects.

* TPC/IP server with authentication [medium / WIP] + protocol converter [medium / WIP]
This is basically done, what's missing is the above mentioned database registration for logged on users. The server has already functionality in place to block certain IPs and to avoid flooding by disallowing the port for 100ms, hence only 10 actions per second are possible.

* TCP/IP test client [medium / WIP]
First client is done, which is a simple text client that displays all messages and crashes *sigh* after minutes or hours because it can't handle all the display stuff, next a graphical client is needed, this will be done in Apocalyx.

* Apocalyx integration [medium / pending]
The first client in Apocalyx is right ahead, this is the next step that we need to create, hopefully not too much of a mess to do.

* Master node [hard / WIP]
The ultimate goal of this is to run a process on the master node that "thinks" for the rest of the infrastructure, like how many processes are running on each node, how many SOX users, CC users, Sims and so on and rebalance the equation, this will require a lot of work and will be most likely the last thing to do.

* NPC scripts [hard / pending]
This will represent a mayor effort to finish, given that this will have to handle scripted quest givers, movement from one place to the other and simulate AI to give the NPCs a certain life and methods of learning, exciting but difficult, given the overall broad approach.

* Subscription Server [hard / partially done]
The first part is already implemented, each user has a validity period after which he cannot log on anymore, you can already, per shell, increase the period, but we cannot handle prolonged subscriptions as of yet nor do we have like a web-based secure server in place for this.

* Interrealm connector [hard / pending]
In a word: Think BIG !! Each realm might be a separate instance of the same Simulation/Game or it might be a completely different game, so user data needs to be copied over, and obviously a domain/realm browser needs to be created. Sounds like fun.

* Physique server implementation [medium / pending]
This can be done the easy or the very hard way by duplicating all the Apocalyx logic into Erlang (sounds like disgusting) or to use Apocalyx itself. If we use Apocalyx, then we have to have a client (probably for each zone) that runs it and every user will send their requests like this:

game client --> SOX --> CC --> action plugin (sorting out movements) --> physique server (sort out the ACTUAL possible moves) --> action plugin --> CC --> SOX --> game client.

In order to make this work, the Physique server needs to do a simple rendering, that does not use any textures or coplex figures, so we might represent of all items only the bounding boxes for collisions and represent users as boxes, this should make this process painless and fast and if a collision is detected then only the real movement gets reported back.

So the next steps are the SOX database registration to avoid double logins and to create a simple Apocalyx client. After that the zoning and char room part with authorizations is due.

Warm regads,

Sunweaver

SOX now upgrades correctly

2008-04-04T09:18:00.006-05:00

One issue in the past had been, that the socket server could not be upgraded in runtime, but now it works, so we can as of now upgrade the code in runtime the whole system and not lose a heartbeat, no disconnects, nothing, the system keeps running. What this means is that we can have our game running, while you kepp customizing your code and then just compile+load the code on all connected servers while players are online and they won't even notice you changed something. This should greatly facilitate maintenance tasks and avoid any kind of downtimes. And now back to the other pending items.

Sunweaver

Tuning to perform

2008-04-01T13:04:00.007-05:00

The initial performance is being tweaked, while not bad we feel, that more can be done here regardng message passing and some first strange errors have appeared, no showstoppers, but something to deal with.

Also under current analysis is how to proceed with zones and where players show up, should one player appear in the next zone when he is close to the next one, that would mean like overlapping zones and if a player is in the center he only reports his position into one if he runs towards a corner he would potentially run into up to 4 overlapping zones ... or is it better to make smaller zones and always send the coordinates into all surrounding zones, which would mean always to transmit into 9 zones. Zones could also be done through range lists and handle everything like one large zone. In order to determine this, Apocalyx will be required, so next steps must include a graphical frontend already. More coming soon.

Work in progress (WIP) of the To-Do List:
* Create an action plug in template [easy / pending]
* Internal user registration (to avoid same user on several nodes) [easy / partially done]
* Cluster behavior [medium / pending]
* Zone Simulation Manager [medium / WIP]
* TPC/IP server with authentication [medium / WIP] + protocol converter [medium / WIP]
* TCP/IP test client [medium / WIP]
* Apocalyx integration [medium / pending]
* Master node [hard / WIP]
* NPC scripts [hard / pending]
* Subscription Server [hard / partially done]
* Interrealm connector [hard / pending]

Sunweaver

SMASH to Planet Earth ... ??

2008-03-24T10:34:00.006-05:00

While not finished by a long shot, we can now communicate between TCP/IP clients and the SMASH framework. The integration is not yet 100% complete, but one can now authenticate from any TCP client against the accounts table and create a chat client (CC: the player proxy). Any chat as of yet joins prefixed chat channels, this needs still a lot of work, like a channels DB, with authorizations to join, chat, listen, member list, password and so on, but at the very least you can now chat in 'general' and broadcast to any connected "player". This also works across nodes, so you can connect to different SOX servers that run on SMASH and even if the CC was created on a different node and still persists, as expected, the SOX thread correctly communicates with that CC, phew !!!

So, what's next ?? Now comes first of all a light test, after creating a scripted Delphi client, we will now do some tests over Apocalyx and create a VERY simple game world. After that there will be some very heavy work where we need to create a character selection / creation something and a channel table with all kinds of authorizations, cheers !!! This will be fun. And also upcoming then is a command parser, to enable the server to start validating stuff, because up to this point each client will decide positioning and capabilities, which is obviously not acceptable. During this step we will also have to implement a physique server, current thoughts are to put an Apocalyx client to do that, but to omit texture rendering and represent clients only as cubes (for line of sight), so we could verify very precisely physical conditions in the game without reinventing the wheel, but much lighter than a client side reproduction, let's see how that works out.

Stay tuned, a graphical test client is just around the corner.

Sunweaver

First benchmarks

2008-03-14T10:12:00.004-05:00

We have conducted the first couple of test series on a single-core notebook (sc), a dual-core notebook (dc) and the PS3, where creating 10'000 users takes 1:11 min., 0:22 min and 0:42 min respectiviely. We then proceeded to send chat messages between the 3 systems and the results are similar in ratio. What is very apparent is that the PS3 displays the text relatively slowly, so in my opinion given that the CEL processor should be very superior to the dual-core it seems that the display is what makes it slow. We will conduct a modified test series, without the display part to check on performance. There is also the aspect of the hard disk performance, but the both series, user creation and chat showed more or less the same difference and user creation requires hard disk access, while chatting does not, so the difference of the 5400 rpm on the PS3 and the 7200 rpm on the dc do not seem to make that much of a difference.

Last not least, it may be that Erlang is not really optimized for CEL processors and maybe it does not distribute the processes correctly, but our personal appreciation is that the display is the bottleneck.

We will see what happens next.

Sunweaver

PS3 + YDL + Erlang = ready for prime time

2008-03-11T09:59:00.006-05:00

Ok, it's been a long Sunday, but I did manage to install Yellow Dog Linux on my Playstation 3 successfully. It took much longer to backup the PS3 than it did to configure YDL. After that I installed VNC Server to remote control the machine and let me tell you, that the Gnome Desktop is a beauty. Installing Erlang was as simple as downloading and compiling it, so no new really on this end, either. In other words, the server platform is ready for deployment, I will now go ahead and run the SMASH framework on it to compare speed ratios among my test systems, let's see what happens next. This is for me a major milestone, given that THIS PS3 will be the mother of all SMASH development nodes. I will have some time next week as well to push the conection between the socket server (Mr. SOX) and the SMASH framework, so I definitely hope to have some empty game world that people can actually connect to by next week. If that holds true, we might do a very first game world logon, maybe, maybe, maybe.

Oh and by the way, some may ask, WHY Linux and PS3 at all ??? Very simple, the thought process behind that is, that I want to be able to provide a server frame work taht is inexpensive and powerful, because regular servers are definitely more $$$ heavy and not everybody will be able to afford buying Windows Server licenses, so nothing beats Linux moneywise and a U$500 PS3 server should make a nice duo. Now, that much said, many people are scared of Linux, because it's not exactly intuitive, well, given that we only need a DVD install and later on a simple download + install of Erlang, that should be fairly simple to do for everyone and ALL the others tasks can be done thru native Erlang, no more touching of the PS3 is required, I am planning on adminstrating everything from a more comfortable platform, without losing the power of the solution.

Stay tuned.

Sunweaver

What's next ??

2008-03-05T08:51:00.003-06:00

Alright, it's 2008 and I finally have some time to come back to the SMASH project. One of my goals was to buy a PS3 and install Linux on it, so at last I got me one and I will install Yellow Dog on it this weekend. Why a PS3 ?? The processor of the PS3 has internally 6 cores that are ready for use and it is said to be a very powerful server, so I will set up Erlang on it and see just how fast it goes. Erlang recognises multiple cores, so this should prove to be a powerful combination, a dual-core versus single-core scores about 40% speed increase, so 6 internal cores should make quite a difference. I will post how it went, both installation and benchmark.

I am also working on the TCP/IP server and I should soon have a first first alpha server up and running where people can connect to (probably a pretty empty world).

See you soon.

Sunweaver

Stability & endurance tests

2007-06-25T17:11:00.001-05:00

Alright, let's get to this week's updates. It is still to early to post any code over here, but we shall give some progress reports. In the last post we commented about next tasks, I took on these fellows:

* Internal user registration (to avoid same user on several nodes or processes) [easy] : Every user that creates a client proxy, automatically registers on the mnesia database to avoid errors. Done for the moment, but this needs to be revised when we get to account management.

* Cluster behavior [medium] : The cluster config will require several things, but the very first one to grab was to define a master. For this a special process called observer was created. The observer is a special registered named service "ob" that stands by and waits that every persistent or longer lasting process needs tro register with and like highlander there can be only one, you cannot run a second observer on your cluster. Once it receives a request it writes that into a public ets table and starts an asyncronous link to that task. If that task or node dies, it will be able to start a stored function, which may decide whether or not to restart or do some desaster recovery, but in the very least, it writes a special local table called monitor (the table does not get replicated for performance issues) to record all of the closed-while-monitored tasks, no matter why, this way we will ALWAYS know what happened, even if you close a whole node, actually, this is why I designed a central Observer, to catch crashed nodes, local receovery was not enough for my taste. The observer is currently a signle point of failure, because there is no backupo for it at this time, this requires improvement.

Now this topic will be ongoing, the cluster behavior obviously does not stop here, it got barely started with this, we will have to look at other topics like:
--> Server tasks (DB, Players, Simulation, NPCs, Chat, etc.)
--> Is a cluster dynamically or statically built, does it load-balance ??
--> We need to notify when a player is between two zones, in order to get smooth character display or else they would just suddenly pop into existence in the next zone,
--> Can we resize the managed zone in run-time ?? For densely populated areas f.e., this maybe rather complicated to implement as it requires continuous terrains.

TCP/IP server with authentication
I have started to develop the TCP/IP interface which worked actually easier than I thought. So far you can connect to it, the server sends back a challenge which in the future will be a MD5(random) sort of function and the client will have to answer with an account name, MD5(challenge), which the server will validate against the database, if the password is wrong the servers disconnects, if it's correct the server will check if the user is already online and so forth and finally create the client proxy, which will be the man in the middle between the TCP/IP interface and the chat client (the low level connector). I am undecided if I do an action plugin to the CC or do a proxy that just uses the CC to connect to the low level functions, most probably I will scratch the action plugin and replace that functionality with this client proxy, it sounds like a better idea, but it also means that per client we would have 3 processes running at the bare minimum. Advantage is that you can just write whatever small program and have it call the CC or call the CC directly from the console and monitor a certain chat channel. So many decisions, so little time :) .

Endurance test:
It seems somewhat early to do stress testing, but I find that most applications don't get off the ground because the backbone has been poorly designed, so thoroughful testing is of the essence. I have tried to open massive clients and I designed a special routine for this, so far I can very easily create 10'000 or 100'000 clients, which open without any problem and send messages between them, we are on the right track and if you close one of those large nodes you get a whole lot of notifications in the database, but no errors, everything is stable, even the code updates keep working, right on !!

Next steps:
Next we will go for the action plugin or client proxy and start designing the simulation behavior, this is a major decision, so a clean and logical design will mean the world (quite literally). Also, a backup behavior for the Observer is needed, given that all his information is stored in an ets table a restart should be quite easy to implement and a second master node needs to be coded. Also the TCP/IP server needs still a lot of work.

Next tasks on the To-Do List:
* Create an action plug in template [easy]
* Cluster behavior [medium]
* Zone Simulation Manager [medium]
* New: backup observer and restart design
* New: mnesia fallback behavior ( to have 2 master nodes, one in stand-by mode )
* TPC/IP server with authentication [medium] + protocol converter [medium]
* TCP/IP test client [medium]
* Apocalyx integration [medium]
* Master node [hard]
* NPC scripts [hard]
* Subscription Server [hard]
* Interrealm connector [hard]

Status: June, 13th 2007

2007-06-12T17:12:00.000-05:00

Project Scope
The complete implementation foresees multi-domains, account administration, zone-groups, NPCs and other simulation handling, but you have to start somewhere and that is from the ground up. In this case that means that we have to build the most basic components first to make the whole work.

Making it happen

Our first task here is to build a multi-server structure. There are two aspects to it, start using mnesia for self-replicating persistent data across the nodes and the next part is a sort of chat room server across the nodes the facilitate message passing among processes that have common interests, like for example guild chat, which obviously needs to function for members no matter on which zone and server they play or you could have your users connect to server X, zone Mainland, but start the NPCs for that zone on server Y, given that you can connect your servers at 1Gb speed, but might only have a 20Mb pipeline, speed is not the issue. This way you could also connect on server Y a service that records all the chat in a certain channel to scan for example for spammers and foul language. The mnesia and multi server chat is already done, next we need to do a sort of internal logon procedure and create an action plug in for the internal player/NPC proxies. The final implementation will open a client TCP process for every client connected, but there is going to be ALWAYS an internal client proxy, which will stay alive, even when the TCP client disconnects, so the proxy is part of the simulation implementing behavior through the action plug in, while the TCP client is just the external connector. The client proxy is going to be a fixed program implementing basic behavior regarding how to plug into the chat server, but we may want to be able to put the player in control of his char or even a "mind-controlled" NPC or disable user input for a while, when a small script runs, so the actual character behavior needs to be handled by a changeable action plug in. So the action plug in will determine how to handle requests, a user might send a request to move or chat to the proxy, the proxy sends thedata to the action handler which might determine, that the user can move or maybe he wants to chat, but given abusive behavior the administrator decided to mute this user for some time and the chat request is discarded. We can see that user handling is so much easier, but how do we start a whole zone ??

Zone handling
Let's give some initial thoughts on zones:
* Zones can be shared across servers with our multi chat room servers
* Some servers should be dedicated to user connections and not run NPCs or Sim events, they will be busy with filtering/handling requests and optimizing bandwidth usage
* If we assign zones to servers, how to handle load balancing ? Maybe assign fallback servers ?
* How to handle instances ? This needs to be coded right into the core structure
Thought: any zone is an instance, zones are linked instances, instances are stand-alone
* How to handle weather and global economic events ?
* How to handle NPCs that walk across regions ?
* How to handle flocks and armies ?
* When a node's down, where do we put his processes without interrupting?
The chat rooms stay the same, the sim needs to be restarted on a different node

It looks like we need a global resource manager and do some cluster configuration. So the cluster master needs to do some load balancing, know the optimal configs, backup configs and if else fails improvise. He will require to know number of running processes on each node, which ones are connection nodes (for users), which ones are sim nodes (for sim and/or NPCs), the main database node that writes to disk should not run anything and probably the master node, it is thinkable that the subscription master node runs password authentication on local mnesia tables which do not replicate on a separate and isolated node.

Next tasks on the To-Do List:
* Create an action plug in template [easy]
* Internal user registration (to avoid same user on several nodes or processes) [easy]
* Cluster behavior [medium]
* Zone Simulation Manager [medium]
* TPC/IP server with authentication [medium] + protocol converter [medium]
* TCP/IP test client [medium]
* Apocalyx integration [medium]
* Master node [hard]
* NPC scripts [hard]
* Subscription Server [hard]
* Interrealm connector [hard]

I will go through the list be degree of difficulty (which also reflects time required to do it).

Until next time,

Sunweaver

Welcome to the SMASH project

2007-06-12T12:37:00.005-05:00

Hello reader.
This blog marks the beginning of a project that has the objective of producing a massive multiplayer online project written in Erlang. Actually the project started in 2005 with a lot of research, but not much code was produced until 2007.

Is it Vaporware ?
For all those in knowing whether this is real or not, be calm don't argue, it is vaporware ... until the day we publish code, so in the meanwhile, don't get anxious or annoyed.

When you say MMO, do you mean MO or MMO?
MO stands for multiplayer online game and the second M puts the massive into it. We classify hereby that any engine that only supports some 15-20 users as an MO and engines that can handle 1000+ users as MMO. Nowadays you see many announcements of people claiming to have built MMO engines, when truly they start to choke on 20+ connections. This is mostly due to a bad initial design because of a platform that cannot handle massive requests and also due to limits of a single server being used. The goal of the project is to produce an engine that can handle multiple servers and at least 1000+ users in real-time, being the limit basically only the throughput of a LAN and CPU power or the speed requirement of the application, so if a given application can live with less updates per second, thousands of users must be supported simultaneously. We don't want another toy, we want a serious MMO engine instead.

Why limit the project to 1000 users ?
This is not really meant as a limit, rather a minimum requirement and something achievable, but one thing for sure, if the Smash framework can handle 1000 its uses are unlimited, because in order to produce a 2000 user framework, you only would have to communicate 2 networks, so assuming that 1000 works, a million is easily within our grasp, too. The current tests have shown that the framework does not even choke on 50'000 test users, although sending out 50'000 messages takes some time, so 1000 really isn't a problem to accomplish and overshoot, but we need to set a target, so 1000 is the initial target, that's all.

Time frame
There is no specific time frame, which is why we classify ourselves as vaporware. The complexity of this endeavor is quite high, so trying to extrapolate a date is too difficult at this time.

Project components
Defining the development platform is vital, so Erlang is defined as the server platform and Apoxalyx for the GUI.

Why Erlang ?
* It runs on several OS like windows, Linux, BSD, MacOS and others.
* It is concurrent on the very heart, creating threads is easy and independent from the OS
* It's soft real-time, which is what we need
* One can share easily information across threads
* The Mnesia database replicates across the connected nodes
* You can update the application during normal operation
* It can be operate fault-tolerant
To mention a few advantages of Erlang, more on http://erlang.org.

Why Apocalyx ?
Because it is easy to use through LUA scripts, it's fast and it's pretty, more on http://apocalyx.sourceforge.net.

Why this blog ?
It sometimes helps to write stuff down as you go and often a loving comment of a reader/listener can help to improve things, so for now I will write here on progress and ideas and if there is something you want to contribute, please message me, note that flame posts or any unrelated posts will be deleted without question.

Bye bye,

Sunweaver