Understanding how your server ticks underneath can be helpful when diagnosing problems regarding slow performance. We’ll cover several topics.
- An overview of the game logic loop
- How this all applies to modded servers
- An explanation of transient problems
- A guide to profiling your server
- A macro-view of improving performance
The game logic loop
The most important part of Minecraft is the game loop. In there is a clock keeping everything synchronized, going tick-tock at an ideally constant rate: twenty times per second, with each tick or tock lasting hopefully at most 50 milliseconds. During each cycle, game elements have to do their work — skeletons have to figure out where to walk, minecarts have to travel forward, grass has to spread, and light problems have to be checked. That’s a lengthy to-do list, considering how many blocks and entities are on a loaded world at at time (thousands). This tick-tock interval is called the tick rate or “TPS,” and you can get some mod or plugin to give you your server’s rate.
Quite a few things are on the to-do list, but as far as the “main game elements” go, those items are blocks, entities, and tile entities. Blocks include simple things such as sandstone, cobble, sugar cane, and things of that nature. Entities are your standard animals and monsters, in addition to paintings, item drops, and “objects that don’t fit perfectly on a grid.” Tile entities are a special kind of block, as they can hold any type of data (think storing an inventory, something sugar cane doesn’t have to do), and sometimes they must think separately. As previously mentioned, each loop has a budget of 50ms and each portion consumes some part of this pie. Any leftover time is considered idle.
Usually on an un-loaded server, it works out and you have a nice slice of idle time, but what happens when it doesn’t “work out?” That “idle” slice in the picture above is nowhere to be found — it’s simply skipped over. That means that there’s no unused time anymore, and each clock cycle consumes more than 50ms. Things just slow down because they happen less frequently (including the passing of time!). While bits of Minecraft are multi-threaded (chat, for example), the game logic loop, the part you really care about, happens in one thread only.
Fundamentally, the problem is that of “too many items” with “each item taking 10 nanoseconds too long.” You have two approaches:
- Reduce the number of entities and blocks loaded at any time.
- Optimize the code for each block or entity to do more in less time.
One very important thing to note is that the number of loaded chunks correlates strongly with the number of blocks and entities. A reduction in the number of loaded chunks is a reduction in the number of offending objects!
How do you go about reducing numbers?
- Unload chunks more often and more reliably. Surprisingly, doing this makes a world of a difference because Minecraft tends to “leak” chunks. This means that over time, chunks continue to stay loaded, adding more entities and blocks. It’s an extremely old bug that’s been known around for forever. Spigot is one third-party server (based on Bukkit) that provides an option to do this, though several Bukkit plugins have provided it in the past. You can also restart the server occasionally to “fix” this problem.
- Reduce the chunk load radius. While this should be a last resort option, the view-distance option in server.properties adjusts the chunk load distance, which exponentially decreases the number of chunks to load every time you reduce the value by 1. This can be extremely effective.
- Design your gameplay around reducing numbers. Your choices determine what your players will do, which may include building massive sugar cane farms that create hundreds of sugar cane blocks that drop thousands of sugar cane items.
Contrary to a popular belief, the number of players on a server does not correlate that strongly with “lag.” Players themselves do not use that much CPU, and players in the same area share the same loaded chunks. The number of chunks is what you should concern yourself with.
Just a note: While not many people can do this, turning off monsters (and animals) basically erases a big part of the pie, which has a massive benefit but the high cost of losing those lovely creepers.
Optimizing the game
Okay, so you’ve gotten the number of sugar canes down, but each sugar cane is still taking a little too much! Each one might take only 0.01ms, but when you still have hundreds of them, they add up.
The process of optimizing the entities and block themselves is not that easy; we’re talking about reducing entity or block “think time” by mere nano-seconds. However, there are people who work on the problem, to which you can benefit from their progress. For example, the Spigot project has a few of these tweaks, and if you run a vanilla server, those changes can make a bit of a dent in reducing lag on your server.
Other portions of the loop
While I’ve mentioned the blocks and entities, there are other things that go on during the loop that you need to be mindful of, and a few things you may want to do are:
- Possibly consider disabling random “re-lighting.” As some may remember, Minecraft used to have numerous lighting bugs (dark spots) all over the place after its last major lighting update (barring the newest 1.5 update). The “fix” was to re-check lighting constantly across all chunks. Disabling this (with Spigot or by commenting out a few lines in the logic loop) can have a massive benefit with an acceptable level of lighting bugs. I can say that I’ve been doing this since this update came out (the impact was quite noticeable) and no one has really complained. (Note: Minecraft 1.5 has a new lighting system and this point may be obsolete).
- Look into multi-threading chunk loading. Loading chunks requires them to be read from disk (a potentially very slow operation for magnetic disks), uncompressed, and then re-constructed into chunk, entity, etc. objects. Remember that chunks are constantly being loaded (and saved), because players are always moving even slightly across chunk borders. If you are using Bukkit, you already have the benefit of multi-threaded chunk loading. Otherwise, the best vanilla and Forge server users can do is to at least put the server’s world files on an SSD or RAM disk, and deal with the performance hit of decompression and object initialization.
- Be mindful of chunk generation. Chunks have to be generated as the world is explored, which can, depending on how many people there are, eat quite a bit of CPU. The older your world is, the less of a problem this is, but if you are starting a new one, you may want to pre-generate chunks, and for all other cases, you may want to set a maximum world border.
- Take note of the plugins that you use. Sometimes they just have to do what they need to do too (i.e. a skeleton can’t think in “zero” time either) to keep synchronized with the game, and you have to be aware of the costs involved.
There are other things going on in the loop, but they are not as worth mentioning. I may visit them in the future.
Now, modded servers?
If you do not run a modded server, you can skip to the next section.
If running a vanilla Minecraft server was not enough of a headache, it doesn’t get any better with a modded server (using mods from FTB or Tekkit, for example). Aside from the incredible number of exploits that come and go, the sometimes endless number of world crashing bugs, the unhelpful personalities that you sometimes have to deal with, getting the server to run smoothly can give you a few bald spots. But hey, it’s more fun.
With mods, the main change is that you know have a lot more entities and/or tile entities. (Though if you use the Forge server, you may also lose out on the improvements in Bukkit’s ecosystem, unless you use MCPC-Plus.)
Here’s the problem though: it’s a lot more. You still want to reduce the number of items and optimize each one, but it’s much difficult when you have hundreds of new blocks and entities to manage (plus you often don’t readily have access to the code of mods). Rather, you have to take a differing approach when dealing with modded additions:
- Understand what the mods you install do, choosing wisely. Some mods, simply because they do more things, require more CPU. A flower mod? Probably doesn’t use that much CPU. A mod that simulates a tornado? Possibly a problem.
- Design gameplay around the more CPU-expensive features. Sometimes you still want the mod, even if it’s expensive. The best course of action here is to design your server around them to discourage people from using more CPU-expensive features. You can also downright disable features, but that makes for sad players.
What makes a mod expensive?
Many mods have the potential to be both extremely cheap and expensive, performance wise, because it all depends on how the mod is utilized. To begin considering the potential cost of using a mod,
- Consider what the additions in a mod actually do, and what “algorithm” might have to run on every tick. For example, an item that has to travel through a pipe has to find destination pipes and then inch forward every tick. That’s not actually that bad, but contrast this with liquid pipes, which may have to “balance” liquid throughout the pipes, although liquid pipes are still simpler than many other things.
- Profile the Minecraft server (and see function call elapsed times) so you know what actually does take up the most CPU, either confirming or disproving your theories. We’ll get into this later.
Here are some examples:
- Logistics Pipes is a wonderful mod that automatically can route items between machines and inventories to do useful things such as keeping machines stocked with fuel, or distributing items across a network of chests. This, however, means that each item that enters the system requires that a destination for it be found among possibly a spider web of pipes. Supplier pipes, which routinely request from the network an item to stock a chest or machine with, can eat a lot of CPU time.
- RedPower2 allows the creation of “frame machines” that can move many blocks at one time. Combine this with its block breakers to make an automatic mining machine. Besides the obvious problems this involves, imagine the load demanded by a machine that is moving across the world and mining large numbers of blocks (which themselves trigger physics events).
- Greg’s Floodlights allow the lighting of long tunnels up to the light level of skylight. As Minecraft’s lighting system is fairly simple, the way the floodlights accomplish this is by placing invisible light blocks along the length of the beam. Lighting, for one, is expensive in Minecraft, and to add to that, the flood lights can be triggered by redstone.
That’s not to say that there is anything wrong with those mods. You probably might want to install them! They are fun. Instead of removing them outright, design your server’s gameplay around the more problematic features so as to discourage people from using them. For example, some possibilities would be:
- Making Buildcraft quarries cheaper to entice people to prefer them over RedPower2 frame borers (although this is not as effective as you’d like).
- Providing alternative ways to make cobblestone for IndustrialCraft2 scrap generation that do not require RedPower2 block breakers (which can get very expensive when you make a 100-breaker array!). Thermal Expansion provides a lag-free method of generating cobble, for example.
We’ve so far discussed problems that are always ongoing throughout the lifetime of a server, but some issues come and go. There’s no point trying to tackle the ongoing problems until you have a handle on issues that may be pretty easy to solve. Some examples include:
- Item drops; They too are entities, and they consume CPU time like any other entity. With thousands of item drops on the map (which may occur due to massive sugar cane farms, for example), this can have a very significant impact on the server. Those who run modded servers are very familiar with this problem.
- Massive numbers of creatures; Massive mob grinders or animal farms can collect hundreds of entities that all eat up some slice of the CPU’s pie. You may need to occasionally prune monsters that are bunched together, or otherwise design gameplay mechanics accordingly to discourage people from making these farms.
- TNT; Massive TNT explosions may or may not have an impact on your server, depending on the server and the set of plugins that you use. A lot of servers outright disable TNT too. If you happen to allow TNT, you might want to conduct some tests.
- RedPower2 tubes; (modded servers only) RedPower2 tubes never drop items, which is great, but sometimes the items get stuck bouncing around the pipes. This is not that easy to do, but once in a while, someone does it on accident and it will completely murder your server (and also consume a ton of network bandwidth).
That’s merely a sampling of the number of transient problems that you may have. Usually one of the best ways to identify transient problems is to profile your server, which we will cover next.
Be aware that WorldGuard does have a feature, /stoplag, that is basically the heavy-handed solution to a lot of vanilla transient issues. It will wipe all dropped items and monsters continually (even in newly loaded chunks), until you turn it off with /stoplag -c. (Warning: It will wipe all animals, as well as NPCs if you use Noppe’s NPC mod.)
Wouldn’t it be great to see actual numbers?
One plus with Java is its large ecosystem of tools for inspecting a running Java application. JVisualVM is one popular utility, and its “sampling” feature is extremely beneficial to identifying exact sources of server lag. What it does is simply to very periodically ask a Java process “what are you doing?” and then record the results, which would show, for example, if skeletons, were using 5% of the CPU time. Sampling is not accurate as “profiling” (which instead records every call as opposed to asking periodically), but it’s more than sufficient in this case (and a whole less server killing).
UPDATE: You can now also use WarmRoast instead of JVisualVM, which may be more convenient for you. The following instructions are for JVisualVM.
Note: This guide assumes that you are running JVisualVM on the same system as the server, or at least have access to a desktop environment, but this is not an option for many people. In my case, I have a very light VNC setup on the server to access JVisualVM directly, but you can try passing JVisualVM over SSH (which doesn’t seem to work very well for me). Another option is to use X11 over SSH, but I find this horrifically slow. There’s also the option to use FreeNX. If you are using a game hosting provider, you are completely out of luck.
You can get JVisualVM by downloading the Java Development Kit. Once you install it, you will find “jvisualvm” or “jvisualvm.exe” (for Windows users, look in C:\Program Files\Java\jdk#.##\bin, or in “Program Files (x86)” for 32-bit).
You’ll notice on the left side a list of currently open Java applications, and you should find your server somewhere on there. Double click the process.
Open the “Sampler” tab and click the “CPU” button to start collecting data.
You will want to wait for as long as you are willing to wait to get more accurate data, and then click “Snapshot.”
What you are interested in is Server thread. It is where the game logic loop resides. Expanding that node should give you the first function called (remember, in a program, functions call functions, which calls functions, which calls more). You will want to expand it until you get to the meaty part (but stick to expanding the nodes that are consuming the most CPU time).
It’s noteworthy to see that sleep[native]() is consuming 95.5% of the CPU time in the above screenshot. This is good — it is that “idle” part of the game logic loop, because I generated this screenshot on a server with nothing on it. You’d be in a great shape if your sleep() call showed up on the results with such a high percentage!
Why are the names strange? What does “r()” do? “b()”? Be aware that Mojang obfuscates the names of the code in the game (presumably to slightly complicate code theft), you will have to deal with this problem. Once you start profiling your server more often, you will learn to know what goes where. Please note that I ran this on a Bukkit server, and CraftBukkit de-obfuscates the class names (so you see MinecraftServer of MinecraftServer.r()), but you would see something like ayy.r() on a vanilla or Forge server. (If you’re on a Forge server or vanilla server, you can look at the files in conf/ from MCP to translate the names.)
Those are screenshots of an empty server: what would a real server look like? Here’s one on MCPC-Plus (which is a combination of Bukkit and Forge), and you can see easily what is consuming the most CPU time. It’s a modded server, and so you will see modded additions in the screenshot below:
I’ve expanded it to the level where entities and tile entities are processed. What we can tell from this screenshot:
- Buildcraft (which is a mod) is eating 21.9% of a tick’s time on its pipes.
- However, most of that time is actually spent on the Logistic Pipes mod.
- 11.6% of the time (md.j_) is spent on living entities (skeletones, sheep, etc.). “md” corresponds to the living entity class. I used MCP to translate the name, as MCPC-Plus uses obfuscated class names.
- 12.7% is spent on RedPower2 motors. This corresponds to mostly RP2 frame boring machines, which is consuming about 12% of the entire server’s CPU time (in the main thread),
- Even though entities take up 17.5% (yc.g()), much of that percentage under that is taken up by items that take less than 1% of the time. That’s the problem of just “too many items.”
- You could go on and analyze more of what is going on, but for now, that should serve as an example.
Powerful information, isn’t it?
Can I blame java?
No, sadly not. While Java (or rather, the Java Virtual Machine) is known to use more memory than it may need to, CPU performance wise, it is very comparable to native code that is generated by C++ or C. That means that a port of Minecraft would not do you any good! (A rewrite would, but so would a rewrite of MC in Java.)
One thing I have neglected, however, to mention is that Java may need to run the “garbage collector” (that identifies unused objects and frees up memory) and it would tie up the game logic loop too, although it is not part of the loop. Before your eyes light up any further through, the JVM is defaulting to the multi-threaded garbage collectors now, meaning any lag that you may be experiencing is likely caused by other issues (unless you are on a single-core machine). That said, you can tweak the garbage collector in use, or even explicitly set one.
There are many articles and resources on how to tune the Java GC, but a lot of the options are more suited to optimizing long-running “enterprise” Java applications with uptimes of 10 months. That is not to say that you won’t benefit, and it’s something worth to try. Off the bat, I only suggest at minimum two flags (which, as I said, may be redundant):
java -server -XX:+UseConcMarkSweepGC -jar minecraft_server.jar
You’ll find a lot of the Minecraft-related threads on the Internet about what flags you can use, and even I wrote one a long, long time ago, but make sure that you understand what the flags you are using do, as sometimes people even suggest putting conflicting flags together!
Note: Java 7 has a new garbage collector named G1. I’ve heard good things about it, from people who run enterprise applications, and terrible things about it, from people who use it on Minecraft. I’ve had a poor experience using it on Minecraft myself. The flag I presented above does not use G1, but instead chooses the older Concurrent Mark and Sweep GC.
The big picture
We’ve been tackling the problem of performance on a micro level — looking into internals and what not — but let’s step back and see what else you can do.
- Throw more money at the hardware.
- Configure your operating system accordingly to reduce overhead.
- Multi-thread the game logic loop so it can be split among several processors or cores.
- Run several servers and link them together.
Hardware counts, somewhat
There is a limit to how much money you can throw at the hardware, at least until Minecraft’s game loop is multi-threaded. Your problem is that you do not need more cores — you need cores with faster clock speeds (think 10 Ghz), but you’re not going to find those kind of processors to buy, and at best, you can try overclocking (for home-hosted or colocated servers). If you have the funds, buy the fastest processor that you can get your hands on (preferring clock speed over the number of cores).
Aside from the processor, RAM does matter, but it’s more of a “minimum” that you need. If you do not have enough RAM and the server has to be swapped to disk (put into virtual memory), the performance hit is massive because the disk may be too slow, and you’d be in contention with the chunk loading. Once you have enough RAM, throwing more RAM at the server won’t do much at all.
However, hard disk speed, especially if you do not have multi-threaded chunk loading, can have a very big impact. This is actually where a lot of virtual private servers (VPS’s) and game server providers (GSPs) fail to meet requirements, because a lot of chunk loading and saving is being blocked by waiting on the disk (which also consumes CPU time, counted as “I/O wait”). If you’re intending on purchasing a new VPS (or even a dedicated server) for Minecraft, you should test its read/write speed (and several times over a period to find an average).
You can get around disk speed problems by using a RAM disk, which essentially makes a part of your RAM into virtual disk space. RAM is absurdly fast, but it comes with its own problem: everything in RAM is lost on server reboot, and so you need some way to frequently copy the server back to disk. If you use a RAM disk, make sure to not underestimate the amount of space that your world will need too. You can use an SSD instead, which is nowhere as fast, but still considerably quicker than magnetic disk (but be aware, SSDs don’t like a lot of writes, which Minecraft is a prime user of).
Reducing system overhead
Even with the right hardware, you may want to tweak your system to reduce a bit of overhead. These tips are Linux-oriented, but I’m sure users of other operating systems may have some similar (or related) courses of action:
- Consider enabling Linux’s “hugepages.” It reduces the overhead of accessing memory by increasing the size of Linux’s fundamental memory block size. Check your search engine on how you can do this on your particular distribution.
- Reduce Linux’s swappiness value. Even if you have a massive amount of RAM available, you may notice that sometimes Linux still swaps programs to disk. If you have plenty of RAM to blow (with usually a safe amount of free RAM, to prevent sudden excruciating swaps), you can try setting the swappiness factor to 0.
- Tune tcp_wmem/tcp_rmem accordingly. This was suggested to me (by /u/chiisana), and I’m afraid that I’ve never considered this! See this article for more information. It might help a bit overall.
Multi-thread all the things
You can multi-thread the game logic loop, but this is not an easy task (and worth hundreds of hours of work). The reason is that making algorithms parallel is quite hard work, as seen below just making a simple counter parallel:
Now apply that to hundreds of game blocks and entities, especially if you have mods. None of them expect it, and if you were to put them into different threads, they would all step on each others’ toes and inevitably corrupt something. Some have nevertheless tried (see nallar’s TickThreading project), and a few have made progress. Unfortunately, it is largely a hack until the entire game is redesigned from the group up with multi-threading in mind. (You can try TickThreading if you wish though, but sadly it did not work for me on a large pre-existing world.)
Lastly, another solution is to run several servers and link them together. Many people have done this in the past, by making portal mods, shared chat mods (something I used to do), and linking serves through one common proxy server (see BungeeCord). At some point, this is your only option, and so you may want to consider organizing your server to be cash flow positive in order to pay for more servers.
As a summary of this post,
- Make sure you are running on good hardware (esp. for shared systems).
- Ensure you have a dual-core (or processor) or better CPU available, with the fastest (and reliable) clock speed that you can get your hands on.
- Make sure you have enough RAM so that Minecraft isn’t swapped to disk.
- Make sure that your hard disk throughput is high enough, or consider using a RAM disk.
- Consider tweaking your operating system to reduce overhead.
- Make sure to identify and remove transient problems. If your server starts lagging, look for the transient problems first and foremost.
- Are people building things that drop too many items?
- Is there some giant collection of 300 sheep somewhere on the server?
- Did someone sent off a ton of TNT?
- Is there some massive flow of water or lava somewhere?
- For a modded server, are people building poorly-designed pipe systems? Frame borers?
- The next step is to tackle ongoing problems, and to repeat the bolded entries from above:
- Install Spigot if possible.
- Add in an auto-restarter if you do not have something to unload chunks automatically.
- Reduce the view-distance setting in server.properties.
- Design your gameplay to incentivize people to build less demanding things.
- Disable random “re-lighting” of chunks if possible (with Spigot, possibly).
- Use Bukkit if you aren’t already for multi-threaded chunk loading
- Pre-generate chunks if necessary, and/or set a world border.
- Go through mods and plugins to see if one of them is spending more time than you can handle at the moment. Contact their developers and see what options they can provide you (and be nice! writing fast code is not as easy as it might sound sometimes.)
- When in doubt, use JVisualVM (or another profiler, of several available, but JVisualVM is free) to find out the exact cause. In fact, you should use a profiler whenever possible.
- If you just can’t squeeze enough performance out, consider making several servers.
(On a relevant note, if you play with mods and your frame rate is really low, you can try using JVisualVM on the client too to figure out which mod is using the most CPU and eating into rendering!)