Preventing the server to crash and burn at release

I wanted to shape and explain my idea a bit better so that I’m able to show more clearly how and why it can work. Just as an exercize.

Requirements: Obviously the idea is possible only if the programmers are able to implement it. From my point of view it’s nothing fancy but there are non-trivial parts. The biggest issue to solve is that the databases need to communicate between each other. Some of the data of a character will be moved (cut and paste, not simply copied) from a database to another and the problem is a possible data loss during the transition. Now I’m not a programmer but I imagine that it’s something solvable in a creative way, for example using a system similar to the journalized filesystems (ext3, XFS etc…). There shouldn’t be other problems since what I propose is simply based on the possibility of that operation/transition. So, once this problem is solved, the rest should work.

The goals: There are three different goals. The first is to regulate the load on each server/shard, so that the population is spread equally on the servers, avoiding overcrowded, crashing servers and totally empty servers. The second goal is to regulate the balance, so that the population is more even between the factions of a PvP environment. The third goal is to insert the previous two into an in-game mechanic/gameplay. So that this system is part of the frame of the game, within the frame of the roleplay and not just an Out Of Character mechanic based on the technical data coming from the math on the servers.

Plus there are positive side-effects, like the possibility to have finally a real global and massive environment without the need to encapsule strictly the players inside air tight spaces.

How it works: For this example I decided to simplify more and more my idea to show how it works in the core. All the rest are layers allowing a more precise control but the core is what it makes it a valid idea. For now I don’t need fancy data, what World of Warcraft shows in the server login screen is enough. It just tells the load of a server in three different states. Low, Medium, high. That’s all I need. At release it’s obvious that I cannot achieve the third goal I explained above (transforming the system into an in-game PvP action) so for the first month it will work in a “special” status.

The idea is that the world is still differentiated into cloned shards. Each shard has a perfect identical copy of the landmass of the game-world. On each shard you can find two types of portals. One to leave the shard you are in, like an exit, and one to arrive from an external shard. To understand better the server structure you need to look this diagram:

As you can see the shards aren’t directly connected between each other. The exit portal doesn’t bring directly a character to another shard, instead it brings it to a “limbo area” working like an “hub” (similar to Guild Wars or Tabula Rasa, but the hubs are unique and not instanced). So the transition is:

Shard(a) => Static Plane => Shard(a-z)
From a shard you can only exit to one of the planes/hubs, from a hub you can exit to every shard in the gamem (if the portal is open).

Now. As I said we know just the load of each of the shards in a three-way status. The portals simply work on the following way:

  • If a server is flagged as “low” population, both “in” and “out” portals are open.
  • If a server is flagged as “medium” population, same as the first case.
  • If a server is flagged as “high” population, the “in” portal is closed, the “out” portal is open.

At release when you create a character you cannot choose a server. The game will randomly pick a “low” population server and send you there. Once you are in the game you can freely leave the shard and follow the portals rules as I explained them here. At any time players in a “high” population server can leave to migrate to a less stressed shard. Noone can get in that high populated zone till the population number will decrease under a set limit. This means that the server will never crash for server-load issues. Once a server is capped the players can leave, but not come in. The new characters instead will be sent to “low” population servers. This will help to have the players equally distributed.


This is how the idea works at its core. Then there are a lot of “complications”. The first complication is about the real math formulas used. We cannot simply calculate the load of a server in a precise moment. Instead we’ll have an algorithm that will keep track of:

  • The load on a precise moment
  • The average load in the last hour
  • The average load in the last twelve hours
  • The average load in the last day
  • The average load in the last week

The server will then combine this data and decide (giving to each a percent of relevance) if a server has “high, medium or low” load. This means that you won’t see these three statuses change sharply as the players log in and out. Instead it will be a relaxed movement. The formula isn’t done, though. Because the server also needs to track the number of unique characters it holds. This because we cannot forbid a player who logged off on a shard to not log back in because the server is set as “high”. So another percent of relevance must be granted to the number of passive (not logged in) characters inside the server. And that value must be considered once again in the calculations to sort out the status.

At this point we have a strong system to enforce always a precise load on the servers. Avoiding overcrowding issues and obtaining an even load on each server. But this isn’t the end. The complete system that will trigger after the first month after release will have three progressive “checks”:

  • The first check is the one I explained above, unchanged. Each server/shard calculates its load. If it’s “red” the “in”-portal is closed. The other two checks aren’t needed. The portal is marked “red” and cannot be unblocked in any way. The players can only move out. Instead if it’s “green” or “yellow” (the load isn’t heavy) we move to the second check. At this phase the portal is still marked “red” and blocked at the eyes of the players.

  • The second check is about the PvP balance. The server/shard calculates the proportion of the various PvP factions of the game, following a similar algorithm used to calculate the server load. It’s at this point that the players could see the portal change its status. If the players belong to a prevalent faction the “in”-portals are blocked even if the server load is low. Instead if you are in a faction with lower numbers AND the server load isn’t high (previous check), you’ll see the portal marked as “yellow”. It still isn’t open and usable.

  • The third check is more complex because it’s about the third goal I explained above. From the player perspective the first check is hidden, passed or not. Then we pass to the second if the first was passed. If the second check fails (still red due to PvP population unbalaces) the players still have the “in”-portal blocked and red. But if the second check goes ok, the portal changes its color and becomes yellow. Now the portal CAN be opened but still isn’t opened. And we are at this third check. In this phase the system becomes gameplay. The players need to organize and conquer (PvP) power nodes inside the shard where they play. When they own enough power nodes the portal finally becomes green and can be used.

This is how the system works in the long run. We achieve the first goal because the server load is always under control, we achieve the second goal because the PvP faction population is as equilibrate as possible (within a threshold), we achieve the third goal because the portals are an in-game mechanic requiring you to play. To go out, take part in the PvP, conquer the land and then be able to move to the limbo areas where there will be access to an instanced form of high level PvE (the “adventures” in the diagram I showed above).

Have fun ;p

Are you a programmer?

He said he’s not (paragraph 2).

HRose, you lost me at the PvP part. Are you saying that PvP should be a mechanism whereby players can open up spots in overpopulated shards so that they can move there–i.e. that, when you are killed by another player, you lose your server spot (if your server is overpopulated) and get shuffled off to an underpopulated server where you can then try to battle your way back onto the one you like?

He said he’s not (paragraph 2).

[/quote]

Sorry, it was so long my eyes kind of glazed over.

If you’re interested in this topic, a quick google on “load balancing” and MMORPG will turn up a boat load of references, including several white papers, doctoral theses and research papers.

Nope.

Once the server passes all its checks (server load+faction balance) it will flag a portal as “yellow”. Yellow means that you (as a character) can unblock it by conquering power nodes. Then the portal is activated and you can use it to travel between the shards. At release this part doesn’t exist because all the portals that pass the check are open by default, without requiring PvP efforts.

The server NEVER moves you. Once you are inside a shard you’ll log in exactly as in every other game. You cannot loose your spot or be ported against your will. What you have is the possibility to travel (using portals) from the shard where you are to another one.

Killing other players has zero relevance on this system. You need to conquer land and power nodes to control and enable a portal, not directly kill enemies.

I’m going to focus on goal one, server balancing. Goal two is a subset of that and goal three can be made irrelevant based on the solution to the server balancing problem.

Problems with the HRose solution:

  1. It doesn’t solve the namespace problem. All the servers must share the same namespace, or else people can’t actually move because of name collisions. If namespace issues weren’t relevant, than CoH, for example, would have one giant city instead of having a dozen individual realm-cities.

  2. If all the servers share the same namespace, then there’s almost no reason not to allow near-complete freedom of movement between any pair of servers.

  3. There’s no natural incentive under a system for players to move from heavily-populated servers to lightly-populated ones. The incentives that do exist (i.e. overcamped spawns, lag, etc.) should be taken care of by good game design, not by trying to get the players to load-balance your server population for you.

  4. The idea makes things difficult for players for no good reason. If I want to join the WoW QT3 guild, I need to go to one specific server where people are trying to congregate. In other MMOs this was trivially easy. Your system makes it a serious group-cohesion problem by making it difficult for people to congregate except under a server that really is always underloaded - and such a server isn’t likely to stay that way if the game is growing at all.

  5. The PvP part of the scheme makes no sense to me. I particularly don’t like the idea of forcing the losing side to conquer what will quickly be the most valuable piece of real estate in the game just so that they can get more players in. Your statements about power-node conquests leading to instanced PvE adventures for high-levels also makes no sense - are these power nodes supposed to allow people to enter the shard or to leave?

Problem already solved in Guild Wars. Add surnames to names and use UI shortcuts to allow quick commands. And not only. In my idea each player can move between the servers but he can only born once.

Since you can born once your home server will never change and your name is flagged after that precise home server. So the name record is: [server+name+surname]

  1. If all the servers share the same namespace, then there’s almost no reason not to allow near-complete freedom of movement between any pair of servers.

Each world is a PvP environment. The travel is restricted so that you cannot jump back and forward at will joining who is winning. This isn’t Planetside.

  1. There’s no natural incentive under a system for players to move from heavily-populated servers to lightly-populated ones. The incentives that do exist (i.e. overcamped spawns, lag, etc.) should be taken care of by good game design, not by trying to get the players to load-balance your server population for you.

On a PvP space where you can conquer and own the land? That’s enough to make entire guilds move and settle down where they choose.

  1. The idea makes things difficult for players for no good reason. If I want to join the WoW QT3 guild, I need to go to one specific server where people are trying to congregate. In other MMOs this was trivially easy. Your system makes it a serious group-cohesion problem by making it difficult for people to congregate except under a server that really is always underloaded - and such a server isn’t likely to stay that way if the game is growing at all.

Totally wrong. My system prevent you to join a server when this server is swamped. The difference is that in WoW you enter a queue, in my game you are out of that server till the queue goes to zero. If your friends want you to join the can simply leave the overcrowded server and meet somewhere else. Of the 84 servers of WoW basically only 10 or so had queues. And for the first week. It’s THAT hard to move to a shard that isn’t swamped? In WoW you don’t do this because you loose the progress and need to restart from zero each time you want to join someone else. In my system you can switch when you want to, without loosing any progress. Even after five months of play you can meet a friend and decide to go play with him. In any other game you need to restart from zero and make choices because or you play there or you play somewhere else.

  1. The PvP part of the scheme makes no sense to me. I particularly don’t like the idea of forcing the losing side to conquer what will quickly be the most valuable piece of real estate in the game just so that they can get more players in. Your statements about power-node conquests leading to instanced PvE adventures for high-levels also makes no sense - are these power nodes supposed to allow people to enter the shard or to leave?

Should I point out again that the shards are cloned? Travelling isn’t that valuable. You do when you need to join a community but in general once you settle down you’ll stay in the place and play the game. The portals are opened because they give access to hubs and from hubs to private instances. It’s an important part of the development of a character.

Conquering power nodes has various effects on the game. The opening of the portal isn’t the only reason. Power nodes affect the magic and affact the zones you own. Then they also open the portals. Once a portal is unblocked it is unblocked both ways (if it passes the first two checks). The players will open portals 90% of the times to go to the instances.

When a portal is open there will be players coming in, players going out to open PvE instances and players travelling to other shards but the purpose of the whole thing is mostly tied to PvP reasons in a similar way the relics work in DAoC.

Instead there are various mechanics so that the guild won’t travel, so that the population on the server will be steady and solid. You cannot own land in more than one server and to mantain your territories you need to stick to that specific server you choosed. Each time you move you need to start over.

Nitpicking for the win.

Someone has way too much free-time…

K

HRose, you’re not a programmer. I think you need to understand about databases, synchronization, and load balancing, and program a few things before you’re going to get much buy-in on your ideas.

I like driving cars, but I’m not qualified to post on a forum about my new ideas in drivetrain design.

This on a very different level than drivetrain design. The papers written on load balancing are mostly not very related to this particular kind of load balancing and the specific problems we see in MMORPGs (although there are obviously some papers). When it comes to databases and synchronization he only assumes it is possible (it is), if it is worth the trouble depends on how bad the load balance is. I like this kind of post better than plain whining at least.

Definitely true.

Consideration: Given the far-from-insignificant complexity a scheme like this adds, is it actually worth doing? How does the cost-benefit analysis come out?

Take WoW – how many subscriptions did the launch headaches really cost Blizzard? I’d guess the number is in the hundreds, tops, if it even cracked a hundred.

Now – how much would it cost to add a sophisticated system along the lines of HRose’s? It’s gonna be many, many orders of magnitude higher than what you’re losing by not having it.

Over the life of the product, what’s going to give the developer the most revenue? More in-game content with a slightly bumpy launch, or a smooth launch, with a commensurate drop in the amount of in-game content due to the shift in development resources?

HRose, I agree, a lot of these issues could be engineered out; however, it makes the back-end so much more complex, it’s just not worth doing perfectly. You find the point that’s good enough, and roll with it.

Yes, but in general I “design” packing in different long term goals. What I propose here is a fancy system to regulate the balance but it does a lot more.

For example it eases (even if not solving completely) the balance of the factions in a PvP environment. This is a critical issue. The other critical issue is that the load balancing isn’t just about the servers, it’s about the players. The game world, to be a good game world, needs to have players. Enough to build groups easily and not as much to swamp each zone.

Let’s see what will happen in a few months in WoW with 84 servers. You think you’ll be able to play the game during off peak? The number of active+passive accounts that each server holds are low. Lower than, say, DAoC or EverQuest. With the time, even if the subscription numbers will be steady or will go up, peoples will log less often and the world will be barren. The PvE will become hard, the PvP impossible.

To this I add the fact that I find horrible to have a large community, with a lot of potential, shattered between 84 servers. Right now it is simply impossible to gather with your friends. You have to make choices and these choices are definitive. Each time you want to move to meet a new friend or another community you have to go back to zero.

I don’t know but from my point of view what I propose tries to address the CORE of the issues of massive game worlds. It solves the load on the servers, it solves the population balances in PvP, it solves the socialization problems, it helps to build a community all together and not shattered and it helps to put all this into a gamesystem integrated within the game. Easy to grasp for new players and interesting as a PvP mechanic, providing already purposes and goals.

What I did was about repositioning all the parts of a mmorpg in the right place.

Oh, and then I haven’t listed other fun possibilities that become easy in this scenario, like instanced PvP battlefields where you can host official tournaments between different servers :)
(think to something like a yearly festival where all the guilds of the game can sign up to this tournament and then join this massive event that could span a whole month. Something like a “Super Bowl” but within the game world)

The point is that the structure is everything. Once it is strong and shaped in the right way, the possibilities are endless.

HRose, with all of the hours you have spent pulling abstract design concepts out of your ass, you could have learnt a programming language or three, maybe some basic art and sound design, and be well on your way to developing your own games. Instead you invest all that effort into trying to get people on a message board to tell you that you were right. If you don’t have the conviction to try to put all your ideas and concepts into action, then why should anyone else give a damn about them, regardless of whether they are solid ideas or not?

In other words: Go make a game.