ANNE gets more emotional

It's been a while since my last post and that is due to the fact that I have been testing ANNE and expanding her emotional range. I am trying to set up a neural network for a "Smith and Ellsworth" style model, which reflects the feelings of happiness, sadness, anger, fear, disgust, surprise, boredom, challenge, hope, interest, contempt, frustration, pride, shame and guilt by determined by the external factors of pleasantness, responsibility, certainty, attention, effort and control. Given that I don't have access to the model I am making up the matrix as I go. It takes around 15 min (!!) to train ANNE, but evaluations are instantaneous (Erlang is speedy), oh well if you need more precision, it takes 149 microseconds (µs = millionth of a second) on the average, a mean of 1µs and a maximum time of 15900µs (most likely the first call), so you could perform over 6700 calls per second to it, not even the most emotionally unstable human could evaluate that many ;) . What this shows is that a single process running this ann could easily evaluate emotional feedback for thousands of NPCs in the game world. Believe it or not, this rather simple matrix of (6, 20, 15) takes up 7 KB !! That's quite a mouthful of data, but anyway. Random tests show very logical although on occasion rather unexpected results, remember that anns do pattern matching, not value matching.

Here some random inputs with E being the emotions ann, note that sometimes even when changing 2-3 inputs they still result in in the same emotion. Curious, but consistent and no worries, it really does evaluate to all possible emotions depending on the training matrix and input. -1.5 denotes the lowest possible value, 1.5 the highest possible one, so f.e. -0.3 pleasantness means a little bit unpleasant, 0.5 control , means some control and so on. The output shows how certain ANNE is regarding the highest match, results near 1 mean very certain, while f.e. a "0.40 surprised" could mean a little bit surprised so you really get an emotion and a degree, the higher, the stronger. I am starting to love neural networks, this is so much fun :) . I am thinking, that animals probably don't need all these emotions, nor all the input factors, which might lead to different anns for different entities.

Well, it seems to me that I will still spend some time with the fine tuning, so be patient with me. If there is enough interest, I might publish the source code, too.

Cheers,

Sunweaver

Fleeting thoughts on AI

No doubt about it, AI is a fascinating topic and there are no clear cut answers, possibilities are simply endless. AI may depend first of all on the game genre, so a platform game will have a different AI compared to an evolutionary game.

Nowadays from what I have seen most seem to have some kind a Finite State Machine (FSM), where certain conditions are analyzed and actions are taken accordingly, this works similar to IF..THEN..ELSE statements. So a NPC just stands around forever on the same spot doing nothing or walking senselessly around day after day without rest, at most the NPC is going through its FSM and that's it. When aggroed, as pointed out, most NPCs keep a hate list, where the NPC sorts who has caused how much aggro, be it through heals or damage dealt and decides who to hit on. Run out of range and the NPC resets, the hate list is wiped, it's got a clean slate, boy do I envy those NPCs !!!

Well, IMHO I find this kind of NPC extremely boring. I envision a bit more and I'll mention an example inspired by Matt Buckland's book "Programming Game AI by Example".

A story from the fictitious mining town "Backwater":

Miner Irwin goes every morning out to the mine to fill his pockets with gold, during this process he will get thirsty, hungry and sleepy, but he also appreciates digging up a certain amount of gold, once satisfied he goes to the bank and deposits the money, if he gets thristy he visits the local saloon, has a drink and talks to his buddies, if he gets hungry he goes home to have his lady who serves him some stew (both will need to communicate as well), if he gets sleepy he will lay down, when rested he gets up and at it again. Let's add to this some more sense, he may be receiving input from his surroundings, so f.e. he one day detects within 50 yds. an orc, he also determines his level and notes that he has weapons equipped, his emotional ann determines fear and he runs back to town and alerts the guards. His goals may temporarily change in favor of self preservation, Irwin goes to the local saloon for more drinks until the danger is over. Guard Erl hears him shout and stands ready to fight the orc. Erl loses the fight to the orc with the rusty spoon.

-HALT- Design decision ahead -

Option 1: Normal MMO AI's would wipe their memories, Irwin engages in the fight before perishing, the new Erl is just as "savvy", as he was before, and both will just respawn and surely be killed again and again and again.

Option 2: Erl preserves his short term memory through death, he remembers that the orc with the rusty spoon was 3 levels higher than him, he modifies his internal state and makes a note to himself: if dead by attack and enemy_level + 3 then update in table fights[self+3] values pleasantness - 0.1 and control - 0.1 . When next confronted with that orc the ANNE with emotional responses will evaluate the situation differently, there will be a moment when anger becomes fear and he may just run for his life and/or call for additional help, this orc William is just too much for him alone. The internal goal seeker of Erl might determine that when he has lost 3 battles in a row, he needs further training and emerges as a level+1, he will now be tougher than before, after some cycles, the orc finally loses the battle and Erl, 4 levels higher and 20 generations later maybe feels invincible again. He keeps standing around for a while and his abilities might slowly degrade to the original skills/levels (or he might stay at his level), this learn and forget setup would avoid players farming certain NPCs but ensures that on the long run the game world is somewhat stable. He will slowly forget that he lost 8 battles against orc William and won 2. However he just might remember in his long term memory how many battles he lost and won sorted by enemy levels and how many attacked him. He may also communicate this to his peers, in case of SMASH maybe on the same city chat channel or a special guards channel. Erl will also change guards with others and not just stand there forever, but swap with Marlo, while Marlo stands guard, Erl goes for lunch or dinner and sleeps a while. Meanwhile our miner Irwin either asks the guards for dangers and/or notices that the threat has gone away, he no longer perceives orc William in the surroundings scanning some 100 yds (just to be sure) and goes back to his original goal of getting his pockets full of gold.

If enough orcs have attacked the city, the council members might stage an all out attack against neighboring orc villages to protect themselves.

While this is still future music, this is what I have in mind for the AI, a simulation rather than just a cheap FSM.

This requires an agent capable of perceiving its surroundings (sight and words), a certain personality to evaluate perceptions, a goal seeking Hierachical FSM (HFSM) capable of sorting goals, a neural network (-> ANNE) to analyze what emotions result from its current situation applied to the personality and a HFSM that decides on memory, learning and thresholds whether to pursue its goals or take alternate actions and then carry them out. Let's call this the perception system (PS), goal seeker (GS), personality profile (PP), ANNE you already know, the memory records (MR), the decision maker (DM) and the action machine (AM). If you wanted to implement a hive mind, you would need to sync the goal seeker, the memory and probably the personality profile or if you do it in Erlang, just call the processes that represent those per hive mind.

I believe this design should enable for different actions according to each NPC's personality: f.e. when insulted, Eliza might feel sad, run home and cry, while guard Erl while take his sword and apply a subtle lobotomy. When attacked, Irwin might run for his life, while Erl fights back. When applying evolutionary rules, their behavior might even change over time. Most literature does not recommend to enable long term learning when you go life, because results will inevitably be unpredictable, so only time will tell if long term evolution is a good choice or not.

I highly recommend downloading and running "AI Planet" from sourceforge.net and also read the author's design notes, it is fascinating to watch how creatures evolve over time. Imagine this in the settings of a MMO and I am sure it will create immersion for the player, but it would also mean the game to be different according to when you joined, tough call.

In the end though, some restrictions may apply, according to how much processing power you have on the server side, if you have 200'000 NPCs in your game world, you'll probably downsize the AI a bit to match CPU power versus NPC numbers, I am eyeing the shared process structure as mentioned above.

Cheers,

Sunweaver

News on ANNE

The toughest task in a programmer's life is not the actual algorithm but the bug hunting, I noticed that something was still weird in ANNE and I found some more bugs in my code, but well.

I implemented some improved trainers for XOR operation and for the emotion part, but this time not by running a fixed number of epochs, but by checking for the LMS (least mean squares) of the training sets, which actually works nicely so. When setting the trainer to 0.01% LMS: ann structure = input:(hidden layers): output

* XOR works after 11422 epochs with 2 hidden layers and it takes 11855 epochs with 1 hidden layer, 2:(2:2):1 versus 2:(2):1. Precision seems to be on par of both by 1%.

* Emotions are trained after 25201 epochs with 2 hidden layers and 18387 epochs for 1 hidden layer, the first is 6:(6:6):6 and the other is 6:(7):6. Their precision is about on par for learned values, but the first is much more precise towards one or the other emotion when interpreting new values, the second would allow for some more varied emotions (like primary and secondary).

My interpretation here is that while for most practical purposes a second layer is not required, it does add a lot of precision if you care for it and if you don't mind the extra time for the training.

I believe ANNE is now working correctly. Now I will start trying different AI scenarios for NPCs, it seems that ann's are suited to be the brains of a NPC, let's see.

Regards,

Sunweaver

ANNE to ELIZA: Think before you talk !!

As I said before, there are so many, many components that need to be built for the complete SMASH, one that I have barely mentioned so far is the AI part, no game ever would be really complete without some intelligence. So let's talk smart today.

Eliza will most likely be part of the NPC perception system and and it will be important for stateful conversations with the added context info (which will later form a sort of decision tree) and when I coded Eliza, I had the idea of also including something else, one thing that I find about basic Eliza is that it feels too cold, there are no emotional reactions behind it. We humans associate words with feelings, like it or not, so I started looking into that.

Now, how do we translate input into emotions ??

In order to do that association I dug up a couple of AI books on my shelf and found a nice model about how a NPC (I recommend the book: Programming Believable Characters for Computer Games by Dr. Penny Baillie-de By) can translate situational information into emotions, which turns out to be an artificial neural network (ann), so after some real messy programming sessions ANNE was born: Artificial Neural Network Entity.

ANNE is a fully connected multilayer feedforward network with normal error backpropagation. I will spare you yet another explanation of how ann's (also called sometimes multilayer perceptrons) work, mathwise, I found however this very insightful page: Introduction to Neural Networks which does an awesome job at explaining them.

While it's tempting to implement each neuron in Erlang as a process, I didn't, because I do not like to have processes just laying around doing nothing 99% of the time, rather, I implemented this as a list of lists, which can now very conveniently be saved to KVS*. This way you don't have to retrain it every time, just save a trained version on KVS2 and with KVS* replication, you now have the model ready to use across all SMASH nodes.

I wanted to see just how many neurons ANNE could support, so after some partial testing I can say that I was able create f.e. an ann with 500 inputs, 2 hidden layers with 5000 neurons each and 500 outputs, let me tell you that this a hell of a list of lists with a total of 30 Mio internal weights, so be patient during the training process. Most literature also does not speak about how many iterations are required to train an ann, but it would seem to me that it requires about 50'000+ epoch (epoch = 1 run with all training sets) to perceive a convergence of the weights. Erlang impressed me yet again, I was unsure if it could hold THAT much data in a single list .... but it did.

Now, let's get a feel for it: My first test was as suggested by Mrs. Penny a simple XOR operation (with 2 inputs, 2 hidden layers neuron in a single layer and 1 output neuron) which after sufficient training works just fine. Once it passed the initial runs I grabbed that emotional model described in the book looks like this (chapter 7.5.2):

What you see here on the left is the lower part of the trained ann with 6 inputs, 1 hidden layer with 7 neurons and 6 output neurons. The ann is then trained with 6 x 6 samples for about 50'000 epochs, returning the new trained ann, the output values should be loosely binary as intended, meaning 5 ceros and a single 1 value.

The call to anne:ann([INPUT], ANN) accesses the anns knowledge, so I fed the values from the book into it, knowing that those values will need to be interpreted and hoping that the 4th value is clearly the highest one of them, .... indeed the 4th parameter is the highest with 0.88 ... which is correct !!! In this example it represents: "fear", also note that the 3rd value is 0.24 which stands for "anger", the beauty of an ann is that it states the most prevalent emotions, but you could also take into account the 2nd strongest one, so maybe this input to the NPC will cause it to run away in fear cursing angrily.

I now need to find some example with 2 hidden layers to test ANNE on, but so far everything works just fine.

I have not decided yet how a fully grown NPC will have to work, it might be a sort of hierarchical finite state machine (HFSM) with modules of Eliza and ANNE integrated into it or just ANNE + Eliza, but I believe this is good progress already on the NPC AI.

One more nice thing is, if you embed an ann into a looped process you would only need a single instance on all of the SMASH servers for all NPCs to interpret that input. Maybe several anns will feed its data into a HFSM and make it trigger its actions and responses.

ANNE is still an uncooked meal, but the ingredients look fine. And that's all for now folks. Until next time.

Cheers,

Sunweaver

Eliza expands its mind

I have looked at many Eliza implementations and most only match a single word in the sentence, or a word in the middle and the rest after the match, so I took it a bit further, I can now match stuff like: "I %% flowers **" and also stuff like "I %% and %% flowers **". This would catch several multi-word patterns in between and whatever goes behind flowers. Given that Eliza does pattern matching but the humans employ several words for the same thing if only your Eliza could group by similarities for example by ["love","like","fond of","don't hate","don't dislike"] . Another example: how many Elizas can tell that these are equivalent to a human: ["MAC", "MACs", "Macs", "Mac", "Macintosh", "mac", "macintosh", "apple", "Apple", "Macbook", "mac mini", "Macmini"] ?? So how do you catch different ways of expressing tastes with different nouns, that tell you more or less the same thing ??

Answer: pattern expansion !! I expanded Eliza's linguistic abilities to match similar patterns, you only need to create a pattern list of synonyms and specify that list in another pattern as an expansion pattern like "I ## ##" where ## stands for the synonym list, here is how:

To recreate the above example you would create 3 records in the Eliza file, the first is the pattern and the other 2 are merely variables.

{["I ## ## **"],
["I live inside a computer","I can live inside a Mac","Maybe I reside in a Mac right now"],
{subject,computers,["##1", "##2"],["like","Mac"],[]}},
{["##1"], ["love", "like", "fond of", "don't hate", "don't dislike"], {internal,synonyms,[],[],[]}},
{["##2"],
["MAC","MACs", "Macs", "Mac", "Macintosh", "mac", "macintosh", "apple", "Apple", "Macbook", "Ibook", "Macmini"],
{internal,synonyms,[],[],[]}},

So now the pattern "I ## ##" is expanded against both variables ["##1", "##2"]. which creates in this case 150 patterns in run-time, but your file only required a simple entry and again Eliza returns some values that your AI can now chew on. Those variables can obviously be reused in other patterns and it's needless to say that I will be able to teach her new stuff in run-time.

Things that I find are still needed be it either on Eliza's or the AI's side is to catch specifically nouns and verbs, maybe by matching against another list and trying to find a corresponding context for them, like when your application knows nothing about flowers, but flowers are plants and maybe your AI knows something about plants or maybe it lacks info on Mac's but knows about computers. And Eliza should also be able to ask the user and remember its question. But for the time being, I am quite happy with the implementation, it will suit SMASH well to have NPCs that the user can actually talk to. The current version is stand alone, future ones will likely put their info on KVS* for easy data access and replication.

Cheers,

Sunweaver

Eliza speaks Erlang with a twist

As I said in the previous post, Eliza should be fairly easy to implement in Erlang and so it was, I present to you the first Eliza that speaks Erlang:

The image on the left shows the Erlang Shell where you can see some examples of Eliza being called as eliza:start(Text) [also possible: eliza:start(Text, Bot) (not shown) to choose some kind of personality]. Eliza will try to load its text file for interpretation from disk.
Eliza prints out in the shell and returns a tuple with: the input string, context information, extra words from the pattern matching and the answer from Eliza.

What for ?? A normal Eliza implementation has no idea what you are talking about, it does no provide or save the context of the conversation, so my implementation has a little twist, you can provide some additional info to each question-answer list, meant to be used by an AI implementation, so like above when stating your name, Eliza answers with {name,player} and the player name, so your AI could register that Eliza just got the player's name or whatever the player tells Eliza, like dislikes, likes, emotions, hobbies or whatever, your AI could then save the data to a database or you could also load a different Bot context file to chat more specifically on a topic. I will incorporate this in my AI modules in the future and in the meanwhile this will serve for some "dump" entertainment in SMASH. A miniature dictionary to showcase Eliza is included in the code. I also see a use for this on the CNPC's to detect automatically abusive or foul language, as mentioned on many previous posts.

If there is anybody interested in the code, let me know and I'll publish it here on the Blog, it's deceptively simple, just under 200 lines and not yet optimized, but it's a first step towards implementing AI in Erlang.

And now back to CAM.

Cheers,

Sunweaver

Huh ?? Did you SAY something ??

As CAM goes forward, I am defining user permissions, channel permissions, guild structures and user levels which go from guest to game developer with each level having unique commands, which of course brings us to the point of what the final command structure will look like, I believe it's going to be something like: {CHANNEL:REQ:PARAMS} where each REQ that comes from the socket is seen as a request for action by the CC component before being sent to its Action handler (AH) and from there to the channel controlling NPC (let's call it CNPC for future reference). Only the CNPC's can return commands that need to be applied to each CC. Some commands will need to take into account distances, which will be done in the CC, commands like "SAY", movement and position data or Area of Effect spells, Thor gives in his book an example on how to do that kind of calculation easily and without spending much on calculations. Apocalyx will need to translate the server packages from {CHANNEL:WHO:{CMD1:PARAMS}{CMD2:PARAMS}{CMD3:PARAMS}} into action on the client and WHO is obviously derived from the CC.

Note to self: Commands are better off as binary structures or at least send the commands list with a number to the client and have the client use a number to shorten sent IP packets. Maybe SOX will translate and package this at some point into a binary format.

Once again I find that prototyping takes so much longer than actual coding, because it really pays off to think before you program and avoid conventional code at all cost !! Erlang rewards those who create the application logic by pattern matching.

Which reminds me, a nice little Eliza implementation for a CNPC might be fun to start with, for those who don't know Eliza, it was invented as a program that would apparently answer with AI to a sentence by analyzing the sentence structure, recognizing certain words and answer with premanufactured phrases, making people think they were talking to a person, well, in the 70's it was surprising, that is. It's still fun though and with little modifications, KVS could provide a short term memory for Eliza to keep track of what users talked about. You may notice that pattern matching is precisely what makes Eliza strong ;) and hence should be trivial to implement in Erlang.

One last side note, I have not yet replaced Mnesia in SMASH given that I am still debugging KVS*.

That's all for now folks,

Sunweaver

Tuples here, there and everywhere

As I am trying to make some progress on the authorization module, I find that the required databases are getting more and more complex and with them the record structure, meaning that tuple pattern matching is quickly becoming a logistical nightmare, stuff like {_,_,_,_,Variable,_,_,_,_,_,_,_,_,_,_,_,_} = Some_tuple is not very practical. Erlang has a native record definition, but it has some important restrictions, so I had to create a record structure for KVS2, that can correctly create, modify and read records in KVS2 by naming the fields, for this you need to create a list entry on the 2 DB structure tables. After that, you can simply call kvs2:crtrecord(DB,Key,Data), where DB is the table & table structure, Key is the record name and Data is a list of tuples of {field_name, Value}, you can read with getrecord(DB, Key, [Fields]) and modify an entry with updrecord(DB,Key,Data), which works similar to crtrecord, you only need to specify the fields that will be modified. This greatly simplifies data management and allows to change the database format in the future, too. You can also use this functionality to manage any kind of tuple with crttuple & updtuple. This should provide some more flexibility than native Erlang to manage structured tuples. With this addition KVS2 can now handle Key-Value pairs with structured value tuples.

This tuple management and the magical pattern matching will make authorizations ... almost ... a breeze.

BTW, for those of you who can already employ version R13, note that there is a new function called: erlang:make_tuple(Arity, Default, InitList) which can create tuples with initial values, very nice addition, unfortunately Ubuntu has no support for R13 yet (unless I compile manually , yuk!), well, you can't have everything I guess.

Soon more on the Chat Authorization Module (CAM).

Cheers,

Sunweaver

Regarding Status Post

I believe the Status Post (below) was long, long overdue. It describes in more detail the structure used in SMASH. Comments are forbidden on it, so please do so on this post, the status report will stay linked on the right and change according to progress obtained.

They say a picture says more than a 1000 words, so I hope that this makes all the other postings crystal clear and show what piece fits where or maybe it eliminates the last bit of clarity you had on SMASH.

I believe another post with screenshots of Apocalyx and development pictures might be nice, too, maybe the next post.

Cheers,

Sunweaver

SMASH Status

Last edited on 25.06.2009

Introduction
This document is a more technical description of where SMASH stands, describing the internal flow and architecture. Please make sure you have read the Introduction first. If you are looking for an interesting read, I also recommend Thor Alexander's two books "Massively Multiplayer Game Development", I use them very heavily for SMASH's development, knowing the books will make it easier to get a grasp on certain concepts. This document will change over time as progress is made, so today's read will be different from tomorrow's and then again, maybe it won't.



The Goal of SMASH
Once SMASH is finished you will be able to create a Massive Multiplayer Online Game (MMO), a messenger-, a dynamic webpage-, an IRC- or SMS-network or anything that requires sending messages across groups of users on different servers, while being completely independent of the platform, as long as Erlang compiles on the OS. SMASH can be upgraded in run-time without missing a beat. The application behavior is dictated by the channel controlling NPCs or Sim Proxies (see below). This is why you will hear me call SMASH a "framework" here and there, because it's mostly a generic communication network on which you can build whatever functionality you require.



The SMASH Concept
At its very heart, SMASH is a chat network, which was inspired by Joe Armstrong's example from "Programming Erlang: Software for a Concurrent World". Joe be at ease, I replaced all your good code with all new, lousy code.

But anyway, every event in SMASH is essentially a message sent from one participant to another. Messages are always tell commands, either to another player or a virtual player, also called NPC. The means of interaction with several other players and with the game world is via chat groups, which are controlled by NPCs, those NPCs are fundamentally the same object than a player proxy, but with a different plugin to control its behavior. To chat in channel "X" the player sends a tell to the NPC "X" which validates, if needed, the intended message and then relays it to all other subscribers of that channel "X". So, for simple chat channels those NPCs might be doing nothing else than relaying text or they might filter offensive messages and raising flags or mute a player for a while. NPCs will handle zone behaviors, by receiving requests from players, validating them and relaying resulting actions; other NPCs in that zone will be implemented as simple scripts or as other NPCs, that receive messages like any other player and act upon them. At one point in the far future, all NPCs might be virtual players, have a decent AI and really "live" in this simulated world.

Zone NPCs might want to communicate with a Physique server to validate movement action or with other zone NPCs for overlapping zones, or maybe you have a server where a simulation runs and have that simulation feed its data into SMASH; Leo, author of Apocalyx has done a great many demos, so maybe I will modify some of his demos to transmit their actions via TCP/IP and players would be able to watch one of his soccer simulation, or car races or robot fights. Time will tell.



How does a single SMASH node work

In practice a single commands in the Erlang shell starts the master node which triggers certain start-up functions and initializes the framework. Each new node starts up as a slave node, the services get started from the master node automatically.

A client connects to the SOX server (SOX) which returns a client socket for each new connection and notifies the chat server (CS) module of it. The CS starts a new chat client (CC) or client proxy, which is the representation of a player in the framework, so if someone disconnects his virtual representation persists for a programmer defined interval. So if someone disconnects during a fight, his toon will persist and most likely die, disconnecting won't save him, some online games omit this. The CC can have a plugin (action handler) linked to it that will validate incoming messages, so one moment the toon is under player control, the next it might be doing some scripted action, or a NPC lives in this game world driven by this kind of plugin support.

The-soon-to-be authorization module will automatically join the player to the channels he needs to be in and grant him the authorized commands with which he can operate in this virtual world. For each group a specialized NPC is started that controls the behavior of this channel (Sim Proxy). It might load a Logic Module for it, connect to a Physique server, load scripts to simulate objects in the zone, a dictionary to filter offensive words or even connect via TCP/IP to an application written in your favorite programming language. At some point some NPCs will need to start by themselves to simulate an on going economy, weather or connect to another simulation.

The command structure will be {CHANNEL:REQ:PARAM1;PARAM2; ... PARAMn} from the client side, where the client can only send requests, the CC will validate if a user is authorized to that request command and if so forward it to the CNPC, which will be the unique instance to convert that request into some action and send a chat command back to each client in its channel. The CC will pick this up, modify its value on the serve side and send back the command to the client, which handle the commands with the following structure: {CHANNEL:WHO:{CMD1:PARAM1;PARAM2; ... PARAMn}{CMD2:PARAMS}{CMD3:PARAMS}} . This protocol is currently plain text for easy trouble shooting.


Underlying to the whole framework are some services like the Observer (OB) and Load Balancer (LB) that will restart broken services, auto start code on new nodes and decide where processes should run according to available resources. The lowest layer so far is the database, which will be migrated from Mnesia to KVS*, the Key Value Services. The envisioned structure is KVS for temporary NPC data or to cache behind the result of a function call to speed up certain things. Its data is also replicated among all nodes by KVS2. KVS2 on the other hand is a RAM resident, self replicating database. It saves data periodically and reorganizes by itself should the master node fail. In the future the OB and LB will follow that behavior to create a fully self reorganizing network. All relevant data will be thrown from local loop data to KVS/KVS2 to enable a swift restart of fallen services without any data loss. KVS* requires that SMASH follows the WORM principle (write once, read many) where only one writes a record, but any other can read it, several processes writing to the same record will mostly certainly lead to disaster and chaos. On the other hand KVS* enables SMASH to have a fast replicating database without write locks. KVS3 does not exist as of now and maybe never will. The 3rd database layer is always handled in a separate module, programs must never write directly to the master database. Currently this 3rd layer is mnesia, in the future it may be KVS3 or a separate database server cluster which will be accessed through a TCP/IP, bundling access to that database into one module makes it easy to change without breaking everything. This layer is meant for persistent data that requires 100% accuracy and has no special speed requirement, like billing data, realm data, account information etc. This layer will be fully implemented when everything else works.

As you can see, there are not all that many pieces to the puzzle, the intention is to have as little and as few but versatile parts as necessary. And from those right-sized buildings blocks we shall make something large, yet manageable (hopefully). Still missing here are AI, state machines, scripts, ... you name it, a lot of stuff.



How do several SMASH nodes work together

As stated above SMASH is meant to handle many servers as one unit, so on each node the same services run and communicate through the unique Sim Proxy to each local chat group object, which in return notifies each player or virtual player (maybe smart and not zone bound NPCs).

Erlang-wise these nodes need to be able to ping each other. This inter-node behavior already works. What's more, you can compile and update all of the code in run-time without missing a beat and given that Erlang code is platform independent, SMASH couldn't care less if a node runs on Windows, Mac, Linux, BSD or "Your-favorite-OS-here", a small tip here, linux sockets have proven to be roughly 3x faster than Windows sockets and beginning NT4 Workstation Windows sockets will only serve up to 10 connection concurrently (on desktop versions), while others will stay idle, I don't think this has changed, so you will need to use a server version to handle many clients in parallel or use some other platform for massive amounts of connections.



The vision of the finished SMASH framework

This picture shows clients connecting to a global TCP/IP server which reads global data and decides where the user was at last. So each SMASH cluster might be a zone, an instance, a realm, a site or a different game altogether. The clients would always see the same IP it connects to, hiding the full infrastructure behind it, although this is still talking about the far future, this might well be the culmination of the whole project, a sort of "SMASH Games Googolplex", let's call it SMAGG in short for future reference.

At this point I have no idea how easy it will be to write a game in SMASH, once the CC plugin support is due, I will see to this, SMASH will most certainly not be for script kiddies, but rather for seasoned developers with advanced skills, but I sure hope that the framework will lift the tons of weight for you to make a large scale project happen. Once finished I will write a Database editor that will hopefully have the power to create a full game, we'll see.

The current size of SMASH is under 1MB (source +compiled programs), so as you can see Erlang rocks when it comes to code size and I believe even once finished the compiled programs will be around 1 MB. The largest part will be most definitely the database.

Currently undecided is whether it's going to be open-sourced, if it will be worth while writing a book about or if I'll just scratch everything and plant a tree or two.



Contributions or how you could help

Well, while I have no real time table, given that this is just another hobby, you might be able to help, if you have a/some public domain ... :

1) city sized BSP level(s) I might use as a zoned demo level(s).
2) action figures for the demos, be it cowboys, spacetroopers, aliens, elves, etc.
3) any kind of 3D objects, animals, furniture, etc.

I am not really a 3D artist, so any free material might help, if not I'll just use Apocalyx demo material and when releasing a first playable demo ask the user to use all of Leo's .DAT files.



Signed,

Sunweaver

There is always another way .. again

Ulf was right, io:format(user,Format, Args) does print on the console it's running on, which works out so much better for me than having to install any code previously on a VM. There is always another way, nice trick, thanks !! I will leave the code below anyway as a proof of concept that I did at least once in my life OTP compliant Erlang code :) !!

if anybody out there knows how to make a node ping the master node from the commands line, please let me know, so far I can only make this work with -mnesia extra_db_nodes [] -mnesia start. I tried -net_adm ping , but it does not work.

My favorite fun function of the month:
[net_adm:ping(X) || X <- nodes(known)]
While I could not have any exact documentation on what "known" stands for, it seems to be a list of all nodes from previous start ups, hence this function pings any previously known node and the result looks like a Wild West Showdown =).

But anyway, with the above modification in place, once a new server connects to the master node (where the load balancer runs) the code gets copied and started automatically, the new KVS2 database gets sync'ed and the output is now relayed locally, it works like a charm. KVS2 updates now always use the most recent record.

Currently in the works is the authorization scheme to channels and what users can use what commands. I am "erlagnizing" the concept to make pattern matching easy. I believe it's going to be divided into 4-5 table, a user table with groups/channels, guild info, allowed commands and revoked commands, an account level, factions and other stuff. Another table with the channel permissions, a table for account levels to differentiate guests from normal players or supervisors, a factions table, a guild table and last not least a commands resolver table that assigns commands permitted according to account level and channel permissions. Commands, will be the heart piece of the framework to enable players to move, fight, eat, sell, chat and other stuff, or supervisors to create new contents in run-time. The revoked commands are meant as a feature f.e. to silence a player, or make drinking during fights impossible, immobilize a player and other mean administrator things.

Cheers,

Sunweaver

New remote process starter module

Although process starting has never been an issue, I find one thing not so cool in Erlang, all io:format output gets sent to the node where you started it, which is very inconvenient IMHO, so I created something simple and new: the smash proxy (SP), which is a remote process starter. Imagine you already have a set of servers running and want to join new ones with everything and starting the services on them remotely, so you monitor new nodes coming online, copy the code over to them and start processes on them, but this time around, all messages they generate will be displayed on their console window, not on the console you triggered that from.

When starting up a new node (slave1@pcname) start the SP with the command line options "-s sp" or "-s sp start ", the first option will start the service, but it won't join you to the other nodes, the second option will generate a net_adm:ping() saving you from having to do it manually. From this point on you can start processes from another node on this one by sending rpc:sbcast([slave1@pcname], sp, {start,{Mod, Fun, Arg}}) [note hat Arg must be a list, as it is passed on to spawn] to it and SP will start the process for you locally. So it's a remote local process caller.

Here is how:
1) Create a folder where all the other Erlang OTP applications reside, I called it "sp" (app root folder), underneath that create a folder called ebin and one called src.
2) create a file called "info" under that root folder and put these lines in it:

group: tools
short: A generic SMASH process starter

3) go under source and create a file called "sp.erl", then throw these lines of code in it:

-module(sp).
-compile(export_all).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
rpc(Mod, Fun, Arg) -> rpc({start, {Mod, Fun, Arg} }).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
start() ->
register(sp, spawn(?MODULE, loop, []) ).
start(Master) ->
register(sp, spawn(?MODULE, loop, []) ),
io:format("SP: Spawning ping to: ~p~n",[Master]),
?MODULE:rpc(net_adm, ping, Master).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Normal rpc call to this same node
rpc(Query) ->
sp ! {self(), Query},
receive
{kvs, Reply} ->
Reply
after 3000 ->
{error, noresponse}
end.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
loop() ->
receive
{Rpcpid, {start, {Mod, Fun, Arg}}} ->
P1=spawn(Mod, Fun, Arg),
Rpcpid ! {kvs, P1},
loop();
{start, {Mod, Fun, Arg}} ->
io:format("SP: Spawning ~p ~p ~p~n",[Mod, Fun, Arg]),
catch(spawn(Mod, Fun, Arg)),
loop();
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Standard interface
quit ->
io:format("SP: Smash Proxy shutting down=~p~n",[quit]),
{ok, true};
upgrade ->
io:format("SP: Smash Proxy upgrading ...~n"),
?MODULE:loop();
Any ->
io:format("SP: Smash Proxy received Msg=~p~n", [Any]),
loop()
end.

4) Now create sp.app under the ebin folder:

{application, sp,
[{description, "Serves to start applications remotely"},
{vsn, "1.0"},
{modules,
[
sp
]},
{registered, [sp]},
{applications, [stdlib, kernel, net_adm, io, mnesia]},
{env, []}
}
]}.

5) Create a file sp.appup under ebin:

{"1.0.0.0",[],[]}.

6) Compile the erl file and make sure that the generated beam ends up under the ebin folder.

You are done now, call the proxy at start up as described above and you can now do remote starts with local display of data. The src folder is not required on each node, only the ebin, so make sure to throw it into each new node installation and you are done. Or take the erl/beam file only and start the Erlang VM from the same folder, it does the same job. If you ever need to upgrade the code in runtime, then replace the beam file and send a command like this to it: sp ! upgrade to renew the code or do a rpc:sbcast to all nodes, as you please. I hope it helps for your projects, too.

Cheers,

Sunweaver

KVS2 continued

I have extensively tested the stability of KVS2 and I am almost satisfied. The infrastructure is rather difficult to kill now, which is a nice thing.

All data written now stores when the last update happened to it and when the data is due to expire, to enable short term memory data functionality and in the future sync'ing data among master nodes. Now let's invent a new term, each database has a certain foundation and principles it's based on, mnesia f.e. defines that as the ACID properties, eg. atomicity, consistency, isolation and durability. Given the design choice of KVS2 we define:

The WORM principle: or "write once, read many"
Only one single process writes a record, every other process can read from it. If several processes write to the same record you will have at some point inconsistent data, given the lack of write locks and random order writes.

It would be more appropriate to call it " one writes, many read", but who can remember the acronym OWMR ??

So, f.e. loot data must be implemented on a zone level and hence be managed through the "zone player" process and auction house handling must be implemented as if it was a zone, WORM for the win.

If we apply the WORM principle to our programming style, in theory, we preserve the ACID property that mnesia enforces on application level and we add an "S" for speed to ACID, becoming ACIDS, because mnesia fails to live up to the part where it should be real-time.

Writing a distributed database is a lot of work and fun. I wonder, how long will I be able to do without write locks ?? Hmmm !!

Improvements needed:
1) Make sure that a newly elected master node is really running --> with a separate process on each node that verifies the masters list, its status and trigger a reelection process when needed
2) During start up, ping all previously know nodes, try to find running instances there, get more node names, ping them and only then start up or else you get several masters elected

Cheers,

Sunweaver

Unintended and useful feature

As mentioned in the PD below now that both modules KVS and KVS2 are working it is very easy and actually extremely useful to integrate both. I believe the best way to do that is to make KVS post into KVS2 and once the cache is expired, delete the data again, hence creating a distributed "short term memory". This looks like complete awesomeness for NPC memory data, imagine a NPC being killed over and over by the same player, he could remember him for a while or to avoid NPC grieving, maybe it elevates its level each time, quest givers remembering for some time who visited them, a reminder when somebody casts a spell or the spell cooldown time. Sounds like fun and it was fairly easy to implement, too. A couple of lines to check for KVS2 and then shoot the data to it. It had not occurred to me before, but our brain also has like 6 different memory types, so why not simulate it. And of course, the data is replicated to the other nodes that run KVS/KVS2. Now I am almost ready to replace mnesia.


And on another note, since I migrated to Ubuntu, programming is so much more fun, this is how my desktop looks like, you can see the cube spinning with an aquarium in the mid, a text windows for code edition and on each plane a separate Erlang VM, very nice, programming in 3D, not on the level of Swordfish, but well almost. Oh and the background is made from an Stargate Atlantis "the Sunken City" wallpaper I modified to become a very nice looking seamless skydome.

Laters,

Sunweaver

Replicate this !!!

Before we start, I installed the SMASH framework on a Linux machine and boy are those sockets fast, the message sync error commented in the post below does not occur, curious, latency speed gain 3:1 over Windows sockets, I am impressed.

But that is no the main topic of today, I have gone over the edge and I am currently programming a 2nd Key Value Server (KVS2). The original KVS server is a Key Value (or Name server), that caches function call results for x amount of time recaching every n seconds, the KVS2 is different. KVS2 is designed to be a database, aka mnesia, replacement.

Mnesia is a very cool design and I like it, I especially love the fact that f.e. RAM nodes can access the DB just by declaring where the master DB is, you can fragment a DB over several nodes, it has transaction capabilities and it can reside on several nodes. If you have a lazy look up application that is awesome, but I found in real life that usually the different nodes cannot keep up with transaction volume of a MMO framework in absolutely no way, if I create one master node then I face the challenge that when a slave node does a transaction it can take up to several seconds to get a result back and if the master goes down, well, so does the whole system, if you define several masters, synchronization takes too long, way to long and if one node goes down the other starts complaining and blows up.

So here comes the salvation, I hope, the birth of KVS2. The code is distributed through the LB to the code and auto started. The first node in the system reads its data from text files and other slave nodes copy the tables over, each table runs as a named process. When you change or delete data that node will issue a broadcast to the other nodes to do the same transaction, hence they sync and each node enjoys having a local cache, speed, speed, speed. If the master goes down, big deal, the other KVS2 servers are linked to the master and detect its death, so they choose a new one, so far they pick the new master simply by name, I could also very easily ask the LB and cache with KVS the value, so that they all get the same answer, but ehm, I don't. This node is then declared the new master and end of story. The master saves the data every 5 min to disk and you can even force a node to be the master, like when the original comes back up. A word of warning here, if several nodes write to the same record, you can have confusing results, as there is no check that all nodes have the same data, if messages for the same record arrive in different order, you can have different records on each node, be careful, this general scheme assumes that only one process manages a certain record, like for example in SMASH, only supervisors write records. Yet when accuracy is critical, it is very easy to send a message to the master node, the rpc interface handles that, so it can send it to the master and the master will then send it around, an easy way to ensure transactions or uniqueness without conflicting records. Simple, short and powerful. Obviously this still needs to be tweaked and debugged, but it is already working. I will later on integrate a time stamp to enable servers to sync data if someone has more recent data, but for the time being I believe this new scheme solves my database requirements nicely, I have a local cache, self replicating data and a system that will work until the last node is shut down, removing all mnesia draw backs.

I will also integrate, somehow, preferred servers to be master nodes, which should make it easier to know where the files get saved to. The keen observer may have noticed, that we lose any kind of indexes and filters, for the time being. Filters are easy to implement actually given the [X || {Y,X} <- List] functionality, normally we know the key "Y" and get "X" as the value, but we can filter like this:

Value=[Pid || {Pid,Node} <- Answers, Node=:=Filter]

grab the PID from a tuple, when we only know the node name passed to the function as "Filter". Here we grab the value as the search value.

And there will be no indexes, it is cheaper to define a small routine that creates a separate table instead. So, a couple more days until I have the KVS2 debugged and then off to replace mnesia completely with calls to KVS2, I have to replace something like 100 calls, quite doable actually.

And then I will implement the authorization scheme, urgh !!!

Cheers,

Sunweaver

P.D. Debugging is almost done, all possible things seem to be accounted for, given the intrinsic structure of KVS and KVS2, they work like a short and long term memory, hmm I might call them both in the future Skynet Database, sounds better than Key Value Server, doesn't it ??

There is always another way

"There's an Italian painter, named Carlotti, and he uh, ahem, defined beauty. He said it was the summation of the parts working together in such a way that nothing needed to be added, taken away or altered [...]" Cris Johnson (Nicholas Cage) in Next

Good thing he did not talk about our code because each time I look at it I find another way of doing things, Erlang is really much faster when it matches patterns rather than using traditional programmatic code, so perfection or beauty is still a long way down the road, but anyway.

RTFM or ... how I should really read the manual more often, I had completely overlooked the function net_kernel:monitor_nodes(true) . When you subscribe to this function, in our case the load balancer (LB) does as of now, you receive a message {nodeup,Node} the instance a new node comes up so now the code gets copied and started instantly on new nodes, it couldn't be faster than this, if you blinked you just missed the copy+start process. A message of {nodedown,Node} tells you when one disconnects.

I also stumbled upon another very useful function called rpc:sbcast(Name,Msg) which sends a message across all connected nodes to a registered named process. Until now the group process caches a list of all groups on the different nodes and new groups can take a while (up to 20 sec. so far) to be seen by others nodes, but given that we now register each group on each node, this function could speed this process up, without relying on cached mnesia data, my only questions is, what happens when 2 nodes do not yet see each other ??

Speed-Ups: Every time I look at pattern matching I learn an easier way to do something, when I first started I was tempted to use traditional loops like the function lists:foreach then I switched to:

lookup(G, [{G,Pid}|_]) -> [Pid];
lookup(G, [_|T]) -> lookup(G, T);
lookup(_,[]) -> [].

I find that structures like the one here are even faster than the other two:

[X || {Y,X} <- L] where X is our return value , L our list to look up a value from and Y our variable to pick the correct tuple from a list. This function is so fast that it is causing me some synchronization problems among my processes for reasons yet unknown. This function will also serve as a command interpreter, when the client sends a string of commands, like "{cmd1:arguments}{cmd2:arguments}", I convert that list with a function I designed into a list like this [{cmd1,"arguments},{cmd2,"arguments"}] and I can then use the above [fun(Y,X) || {Y,X} <- L] to run some function on each element in the list. I might even displace Mnesia altogether, just load data in lists into the KVS, look up / filter values with pattern matching and sync the nodes with rpc:sbcast. I could use the module dict to create a dictionary instead of a table or list, too. Each dictionary could run in its own thread, hmm interesting. This sounds so crazy, it might actually work.

Remember that Mnesia is nice when speed is not critical, but when you need to do thousands of look ups per second across "n" nodes then the generated overhead is just too much, mnesia can't keep up, no matter how I structure it, be it several disk nodes, 1 disk node several ram nodes, the worst thing is the lack of a local cache on a ram node, hence the KVS working as a local cache. From a mnesia centric application framework I am getting more and more towards a KVS centered data structure, the above scheme might eliminate the caching errors and I do not require really the supertight transaction property that Mnesia offers, I'll gladly trade that for speed.

Designing the authorization module (AUT): it will mean a large impact, because I will need to define a whole lot of logic around this, including what commands each user can perform, guild channels, raid & instance channel, guild structures and more, so this will take some time to complete, but the design isn't finished yet, I am still prototyping the logic. Some nice features that will most likely come up are f.e. you will be able to define as many guild channels as you like and define their authorizations to differentiate the general channel from officer, class lead, raid or other group channels below the guild. I will also consider instances and raid channels with ownerships, to prevent problems like an invited outsider stealing raid IDs, a documented problem that happened to other large MMOs. Under the devised scheme an invitee will not have ownership or admin rights and it should be an easy thing to do that certain instances must be started by the owner/admins and hence avoid getting a started raid instance stolen. The user commands wil also be important to avoid that a guild master leaves the guild without reassigning ownership to somebody else, so a GM will lack a command of GQUIT, but have a GOWN, while other members will have GQUIT, but none afiliated will have the GJOIN command sombody unafiliated will have. The authorization module will need to combine speed with functionality so a huge fun part is coming up here.

With the speedups like the ones above, I am now facing sync problems between threads (that should not exist on the first place), the first or first few messages from a client get lost in cyberspace, I know that this is due to me cacheing values from mnesia, so I am thinking that the easiest way to avoid lost messages will be to implement a ping-pong protocol to allow the client to ping the server (once authenticated) until he gets a pong response and only then start asking the server for more stuff or start chatting. The cacheing delay can be anything from unnoticeable to several seconds.

Another curious fact is that the more functionality I design the smaller the modules get.
The module CS which was in the beginning the master of the whole framework is becoming less and less important, with groups registering now, it is not much more than central resource locater/creator. The CG module is now the local node relay station, down from channel admin, the logic has gone to the channel NPC. Both are still very much required, but their role has drastically changed during development, curious.

Cheers,

Sunweaver

Auto Config ... baby steps

While bug hunting my code I came back to one thing I always wanted, have new nodes initialize themselves, like Skynet, becoming aware of new resources and start using them. Erlang provides a so called boot server, but no documentation exists that would clarify as to how it works, only one developer posts like half the instructions as to how set it up, but I could not make it work, I get to the point where the boot server starts, it sends the boot file and the remote node exists with an initialization error (and yes I did follow the instructions and the boot file oly contains the Erlang required modules), even analyzing the source code did not make a difference, although I found some undocumented calls, like erl_boot_server:add_subnet(mask,IP) and :would_boot that add a whole subnet to the boot server, compared to the normal function that requires you add each single IP address, which makes no sense to me and would_boot tells you if a certain IP could connect or not, fun functions, if only somebody would comment on how to set it up the whole way.

Anyway, I said to myself, there must be another way and of course I found one. The Load Balancer (LB) on the master node now checks for node changes every 10 sec. and reports on nodes lost (without any action so far, in the future it might restart lost processes on other nodes), nodes that stayed up (they stay untouched) and new nodes. If a new node is found it verifies rather crudely if SMASH is loaded on that node, if not it copies the code to it and starts afterwards a slave node and tries to fire up the socket server as well, this will fail for secondary nodes on the same machine, but that's ok, the supervisor table keeps track of which one stays up, so no problem here.

What this means for the framework is that you only have to start up and connect a node to the master and not worry about anything else, I love self configuring or so called zero configuration programs, so this is a step towards it; f.e. at some point when you want to fire up simulation parts and use the LB to pick a node, you know that the basic framework is already up and running, which is nice.

Notes to self: 1) keep a record of the internal and where possible external IP address of the SOX instances, maybe the No-IP Service can be helpful here. 2) Some work needs to be done now at the observer program to auto-restart lost processes. 3) we now create a player instance for each channel, so we probably only require to track those for the simulation, maybe every xx time we save their data to Mnesia and/or to the other group (CG) instances for pure safeguarding, both would work, Mnesia is probably the slowest yet cleanest approach.

Cheers,

Sunweaver

There is a bug in the electrical system !! (Agent J)

With the new functionality in place some rewrites were necessary and some debugging of course, *sigh*. It's impressive how many bugs one can find in his own code, I did quite some trouble shooting these days and I have finally found some strange disconnect errors, where a client would randomly DC for no apparent reason, but my latest stress tests with a couple of test clients bombarding the SMASH framework are keeping up nicely, so after the first 30 Mio I will make the audacious assumption that the bugs are now gone, the counter will hit 100 Mio sent messages across the system by tomorrow, I hope for the best =) !! Good news also is that the system is much faster than before and the KVS is shining in our quest for speed lookups. Hmmm, now I will need to read about LUA code again and prep that Apocalyx client for testing.

Cheers,

Sunweaver

P.D. 100+ Mio messages ran through SMASH and it keeps running and running, the bugs have been eliminated and the framework is now more than ever a stable multi server soft real time messaging system, so far so good. =)

And: Action !!

It is 2009 already and I have finally some time again to keep programming.

Action handling
After much thought on how to implement the application behavior, currently all chat was sent around without validation and a MMO needs to have an application logic behind each channel, so specific action handling needed to be implemented to process each client info before sending it around in the channel. I have come to the conclusion, that probably the best way to do that is to implement them as a player (CC) module with an action handler (AH), both are stand alone processes, so whenever a group gets started up, it will automatically seek in a table for the AH of that group or else start up the generic entry as the handler. ANY message to be sent in a group gets sent to that single automated player entity, which then decides how to proceed, so for chat channels one can implement a word filter, NPC data can be implemented from here or zone channels will be able to send the movement data first to a physique server (far future) before sending that data to the clients. This single AH per channel will also keep track of connected clients, no matter what node they come from.

Some lessons learned of programming in Erlang:
First lesson: there are so many ways of implementing a certain behavior, it's mind boggling, one can choose from creating vanilla subroutines, a cherry mono- or multi-tasked applet, a table entry, a DB table or whole self managed entities, the choice is really vast and the problem is that you can be quite dynamic, so I have to sometimes go back and reread where all the magic comes from, all or most choices seem to be on par speedwise. Once decided, implementing it is a snap.
Second lesson: right size your multi-tasking; it's so tempting to create many, many processes, but syncronization can sometimes be a pain, so double check when to MT and when no to. Even when sync-ing is no problem, don't forget the possible size of the outcome, f.e. in the very beginning I sent each chat command as a separate task, but when I did the stress test with 50'000 generated users over 5 servers I found that 2.5 Billon concurrent tasks was a tad too much to handle for the framework, I wonder why that would be, how lame !!! =P
Third lesson: And Erlang programmers will hate me for this: OTP sucks big time, I found that it is much easier to write my own functionality, including but not limited to code upgrades in runtime across all connected servers, which requires a Doctor title in OTP to get to work, but only little effort to programm oneself, nowadays I only kick in some very few lines of code and voila, it upgrades !!
Forth lesson: they say love comes quickly and so does complexity !! Programs in Erlang are short and concise, but when implementing something new I find myself rewriting code in 5-6 programs in parallel, which isn't harder, but very different from "normal" programming languages.

Next to come is a channel authorization table that decides at logon (or later) what client should connect to which channel(s). I will also implement channel authorizations for that, which will be interesting =O, some must be system assigned (zones and instances), others with password, by group assignment (like a guild f.e.), open or user managed, so this will be an important feature and also some piece of work as I need to rewrite some code for that. Before or after that I will implement the first NPCs with some limited actions, oh and I still need to get the object table into the Apocalyx client to handle mutiple objects. But after these tasks I have the basic capabilities for an MMO, that is good news.

And Hello World !! Since my last update I seem to be getting quite a lof of hits on this blog I can even google this blog, much to my surprise actually and scary, too. SMASH is not ready for prime time after all and it may still be a while, due to the fact that I only do this on my spare time. So for the newcomer: feel free to read the MISSION STATEMENT. This is still vaporware, but whatever is mentioned herein is actually implemented, so one day (hopefully this year) the framework will be up and running and I will set up a closed alpha server for some testing. But I will not haste anything, quality overrules time requirements.

That's all for now !!

Cheers

Sunweaver