Snapshot 6 Changelog (and it's private)

The new streamer is working surprisingly well. I just put it to the test under some interesting test scenarios which I am going to describe soon in another post and it has proven both stable and advanced enough to handle stuff it was not designed for.

Still, it is too early to release it into the public and since this eat up most of my time I still didn't bugfix Snapshot 5 100%, so Snapshot 6 will be kept private. It is feature complete, but buggy and I don't want to knowingly release buggy software if it can be avoided.

Here is a screenshot with it:

This screenshot was done on a computer with and integrated cheap Intel GPU without any dedicated RAM. The new engine has higher requirements but it can just about manage to slug along on such low hardware. So the upgrade in requirements is minor. 

Actually, I would like to do an experiment soon. Model only 2-4 objects at the standards of (the end of) 2012, with a polygon count perfectly suited for the complexity of the object and 2048x2048 textures. Try the engine in first person mode and see how it looks. And how it performs. I'll try with an 8800 GT and a 540M GPU.

Here is the changelog for Snapshot 6 (I won't go into detail because I've talked about a lot of this stuff in previous posts):

• Uses the second generation 3D engine
• Graphics overhaul. The current level and the one bellow are rendered with high quality blending to create the illusion of a less cube based terrain. Most meshes have been fine tuned for for better visuals and better LOD behavior.
• Up to 60 levels can be rendered.
• Streaming biased LOD switcher implemented. The system uses some simple heuristics to try and prioritize changes that have the potential to create a better image, while changes that don't have such impact or only free up resources are pushed to the back. Item building almost two times faster.
• Crafted items have quality ranging from poor to epic (when it makes sense).
• Cleaned up and bugfixed the item description panel.
• Quick navigation keys: press "]" to go to the highest level of the current map, thus rendering the entire map. Press "[" to go to the lowest level of the map, thus rendering only that level and the one bellow.
• Trees are back. They are still too high poly and represent a massive performance bottleneck. Animations had to be turned off. You can toggle the rendering of tree canopies on or off with the "T" key. Turning them off offer a performance boost and can even help you navigate the map easier because canopies are quite large.
• Implemented wall smoothing.
• Fixed horrible screen tearing in full screen mode with vertical synchronization turned off.
• Smoothed out jerky movement when scrolling or zooming. Actually, this effects everything, from time passage to animations, but scrolling was where it was the most visible. Currently bugged if vertical synchronization is on.
• When in first person mode and the game window is not active the mouse cursor is no longer grabbed and continuously repositioned at the center of the screen. Furthermore, while the window is not active the game is paused and the game will display the last render it had without consuming any resources except for RAM.
• Better terrain generation: wold is made out of random thickness layers. A layer is predominately a single kind of material but has random chunks of other materials inside. The system still isn't as sophisticated as the one present in the 2D version.
• New containers: armor stand and weapon rack.
• New furniture: wooden chair, table and bed.
• Improved item interaction consistency. Placing block on top of items used to work, but placing items on top of blocks didn't. Now all items and block are considered to be in the same category, so you can replace anything with anything without having to delete something from a given location.
• Zooming now works with numpad +/- too.
• Mouse wheel scrolling support to the engine. Scroll with the mouse wheel to zoom and hos shift and scroll to tilt the camera in tot down mode. In first person mode you can control the camera FOV with mouse wheel scrolling. 
• Introduced a new super safe API for common tasks. This new API is very slow but it checks for most inconsistencies and reports them. I started using this API for most interaction tasks that are not performance critical. This is a developers tool meant to keep the number of bugs low and as the engine matures this new API will be slowly phased out.
• The "X" key now also deletes all walls in selection, not just items.
• When starting a new map the engine will try to position the camera upon an interesting location with a clearing rather than randomly in the middle of nowhere. Alternatively it is capable of placing you on the highest point of the map. But most of the time it will just put you in a corner with a clearing.
• A few extra options added to the launcher to control mip mapping and filtering.
• Experimental: The game can now be compiled and executes correctly without having to modify the Irrlicht package.
• BUGFIX: When replacing a block with a block of the same type but a different texture, sometimes visually the update would only trigger after a secondary block has been replaced.
• BUGFIX: Placing new blocks would not always update the height map.
• BUGFIX: Items would sometimes be created over the upper border of the map if the map was too high.
• BUGFIX: Placing items at world geometry seems can cause visual issues and even a crash.
• BUGFIX: When changing levels in first person mode the camera no longer behaves in a disorientating way.

Second generation 3D engine done!

It's about time! For future generations I have planned some light multithreading, at least some gray blobs behind objects to simulate shadows and indoor lighting. 

But let's talk about how the second generation engine came to be and what it can do.

Like I said in the previous post, I am raising the quality of graphics just a little. So I set up a heavy stress test and started tweaking the objects. The problem was that I was getting heavy snagging: while scrolling the map, the scroll animation would freeze for just a fraction of a second, but enough to be jerky and disturbing.

There are many reasons for this: it was a stress test so item density was high. I tested on a map without elevation and elevation uses less resources than items, so the load was higher. But there are two main reasons for this. The first is that I fixed scroll animation to be smooth. In the past it was never smooth, not even with an empty map. So with jerky scrolling you did not notice the extra snags. The second reason is that the new LOD system was designed with the first person camera in mind. I said back then that for top down the new one is a little bit slower and I did not get around to do something about this. Ironically, the primary camera for world interaction that also has the best performance has the worst LOD implementation. For first person camera, the snags are not an issue.

So I ignored this issue for starters and went though the entire item list optimizing LOD levels to obtain the best possible visuals. This meant going to a LOD switch border, taking a step forward, one back, from different angles and observing the change. Then adjusting items: adding a face her, a loop cut there, removing some faces, changing shading, scaling, moving, etc. It took me about six hours to do so for the 14 items I have right now. So I don't want to touch Blender again for at least a week.

So now I have a good looking set of items with all LOD levels 95% optimized. I went for visuals, not performance, so GPU requirement is a little bit higher, as planned.

This is my reference point. This is how graphics will look and how many resources they will eat. So I must make sure that for these levels the engine works fine and is snag free.

In order to remove the load on the CPU I implemented a streaming LOD switcher. Streaming means that the system determines the work load and has a speed at which it sets of to execute that work load in the background. This way even huge tasks can be executed, they only take longer. The speed must be set in such a way that it is a good compromise between CPU time and the resulting pop-in. Because streaming also means extra popin. With streaming you can walk for a while, then stop, and a few second after the engine is still working, optimizing what you see.

I did two fun experiments that were visually interesting for me. The first was creating a map but not pre-populating it, so in the first second in was completely empty. With a streaming speed set to low it was fun to see the world be built little by little for a couple of minutes. The second experiment was the opposite. I created the world and pre-populated it with maximum LOD levels. It ate up almost 1 GiB of video memory. Then I watched as the LOD switcher reduced detail for distant objects little by little, eventually reducing the memory consumption to 150 MiB.

I fine tuned the values for what I expect my target hardware to be and got this result:

To make things more interesting I set the game to ultra and set antialiasing to 32x CSAA. This is why performance is so low. With 4x antialiasing and without FRAPS performance is at normal levels.

What can't be seen in the video is what I did next. I replaced the simple streaming LOD switcher with a biased streaming LOD switcher. This means that it uses heuristics to try and prioritize updates that would lead to a better quality render.

Combining all these techniques I get a really spectacular engine for first person navigation. There is nothing I can think of that would improve it right now, except for adding free form height transitions.

In top down mode things are not that great. Streaming greatly reduced snags, but they are still present. Making them shorter and less frequent actually made things worse, because you are not expecting a snag.

So the next step was to do some heavy profiling to determine what causes the snag. I created for these tests very small fixed size maps with an uniform distribution of barrels to eliminate all randomness. I scaled things so the streaming LOD switcher would interpret this as the worst case scenario with the highest work load.

To my surprise the snag seemed present and of similar length even on such small maps. This meant that map size was not an issue, and the snag was constant per work unit. So you could determine the approximate snag duration by multiplying the work load with the cost of the work unit.

There were two important measurements to determine. I won't go into detail of what these represent because I can't explain it without fully explaining the streaming LOD switcher with formulas and all. It is suffice to say that for each LOD switch for the maximum work load I will give two numbers in milliseconds. Both numbers need to be as low as possible, but if the second is not that low it is not a big deal.

For the game compiled in release mode I got an average of 29 / 4 ms, and for the debug mode 88 / 12 ms. Having a 30 ms delay when scrolling when you are rending a lot of frames per second seems like a good candidate for the snags. Even if you have just 100 FPS, a frame takes 10 ms to render. So while pressing a directional key we have 10 ms, scroll, 10 ms, scroll, 10ms, 30 ms, scroll, 10 ms, scroll, 10 ms, scroll. You see what the problem is? Once in a while a frame takes 4 times as much to render and this makes makes scrolling jumpy. And the higher your framerate, the worse it gets. I did a test and I confirmed that making this delay shorter directly affects the smoothness of the scroll.

The first thing I needed to do is disable hardware buffers and see if the results remain consistent. I heavily use hardware buffers and if they are the cause for the length of the update operation, there is no way to fix the snag. Not without going into the Irrlicht code at least. Keeping my fingers crossed...

Disabling hardware buffers causes a 6-7 times drop in FPS, but the operations seem to take as much as before. Maybe times are sometimes lower by 1-2 ms. So this is great news. It seems that hardware buffers don't cause this, so it can potentially be fixed!

Then I change over to shallow building. Shallow building is very fast but can only be used under very limited circumstances. Shallow building changes the 29 ms to around 10 ms. This is not that good. It is not possible to get better results than shallow building for any build operation. To further improve upon that we need assembly and/or multithreading. But from 29 to 10 there is a big difference, and even bigger form 88 to 10 (it is still 10 under debug mode), so let's see if we can't improve upon this a little. These measurements were taken under worst case scenario, so a sizable but insufficient improvement here could translate in what is needed for normal scenarios.

The first step is to get rid of some vestigial shadow support calculations. Shadows probably won't make it into the engine for quite a while so no use having them around. This brings zero gain, but it is cleaner.

The second step is to try and optimize bounding box calculations. First we fully disable all such calculations to see what results we can get: 22 / 3 ms. Not bad! That's 7-8 ms spent just calculating bounding boxes. Of course, without bounding boxes frustum culling is dead and performance goes down while rendering is no longer correct. So let's optimize! After a 30 line optimization that pre-computes far more efficiently the bounding boxes using vector math we get 22 / 3 ms.  To make sure that everything is correct I keep both methods on for a while and added an assert to see if they produce different results while playing around on a full map. The new calculation gives the same results 100% of the time. I could replace them with a calculation that is faster, less precise and thus making the GPU work harder to render, but I don't think this will help the performance of the LOD switcher. For low to medium item densities this change is already enough to reduce snagging noticeably.

Next we go for the colorizing code! After doing this we get a minor and improvement. Sometimes we get a -2 / -0.5, but sometimes we don't. Such small duration are hard to measure with accuracy. There is still one if per vertex that I would like to see gone. Luckily, I find a loophole: I make a change that slows down level change a little but the if is gone. I am getting now a consistent 18 / 2.75 ms. This optimization can be applied  to shallow building too so that has been sped up a little bit.

The obvious and simple stuff is out of the way an we are sitting on 18 / 2.75, down from 29 / 4. Now comes the hard part.

Using extremely low level pointer arithmetic I squeeze out every possible CPU cycle out of index updating and get down to 16 / 2.5. Other than this I have only one more idea, an idea that uses about about about 12 MiB of static cache memory, but I wont resort to that only if I am desperate.

For vertices I try something with memcpy and pointers, but this does not give any benefits, so I try the same tactic with pointers like I did for indices and... I think it is 15.

So we'll remain by the 15 / 2.5 for now. This is as much as I can optimize for now, but already this gives great results. When testing out on a normal map snagging is completely gone when testing in top down mode without the massive performance hogs: the trees. With trees I'm 95% it is gone. Sometimes I get the impression that there might have been a minor snag. First person never had any snagging issues and now it is probably even better.

This change also generally effect anything that creates or destroys item so it is a global performance gain. Item creation is now almost two times as fast!

The only bug I can see is that streaming and vertical synchronization do not mix well. So for now it is recommended that you turn that of. Vertical synchronization adds frame delay which compounds with my delay and things are jerky again.

So the streaming LOD switcher and this massive item creation optimization are the stars of the second generation 3D engine. If you have a weak CPU you should instantly notice the change. If you have a strong CPU you won't notice anything except for a smoother scroll that is caused by bug I fixed some time ago. I think I wrote about it. Not sure.

This second generation 3D engine will serve me well, but because of the massive changes a bug or two might have slipped in. I need a few days to play test and make sure that everything works as expected. 

So I can't really release Snapshot 6 yet. If I deem the engine stable I might make it on time, but if not I'll post the chagelog for it and get started on Snapshot 7.

Snapshot 6 also marks the finalization of creation mode stage 1, so I might do a release only with this followed by a few bugfix releases, but my mind already wanders towards the reintroduction of dwarves. And a very special HD fork.

I won't work on the engine any more right except for bugfixing, but one day when I feel ready for it, I'll try and create the third generation engine that should do all the tasks with a little help from multithreading. 

Performance disscussion & graphical updates

If you don't have the patience to read through this long post just skip to the end: there are two screenshots with the way the game is looking right now.

This is one of three posts in which I talk about some specific theme. Today about performance, some other time about the design of the game systems and key differences from DF and finally I'll detail all the playable races with great (anatomic) detail.

So I was working on improving terrain quality and was trying a new technique that would make it prettier than ever. Getting the color blending right was near impossible, but I was getting good results. The caveat was that this method was two times slower, used the CPU too much and not enough GPU and used three times as much memory. This was the first alarm signal that I must greatly mind the performance implications of what I implement because this is not a normal game.

The second alarm signal came when I put together everything that I have ever implemented (except for shadows): fully editable 300x300x100 maps with level changing, rendering up to 60 levels, great item diversity and density. This was the result:

Yikes! Sure, most of the new items and the trees don't have low LOD variants, but still, the performance is way too low. Ten million triangles? Almost 500 MiB of mesh memory? A very heavy round of optimization ensued after seeing this and you will see the results at the end of the post.

So why is performance such an issue?

First of all, this is a DF inspired game. Just creating a game with half as much depth as DF will be heavy on the CPU. This is the first challenge.

Creating a game with the same depth, but with a big scope, scale and frequency of events as in DF is the second challenge. This is one is very hard, almost impossible. No one man in the history of gaming has managed this yet. You may be thinking: but what about Toady? Here's the thing: he is very close, but the game is not out of alpha yet so you can't really tell. Are the anecdotal jokes about needing a super computer to run DF under a given load that circle around in geek circles caused by the game still being in alpha or is this an issue that can't really be resolved. Could multi-threading help? There are a lot of questions and right now DF might be 99% there, but the way to 100% is still very long. Ah, and don't forget about the bugs.

The third challenge is the 3D. Creating an engine that can render everything and provide an interactive environment is not that hard. Creating one that does these things with a big item density, no billboarding and unlimited view distance is very hard. Creating one that responds to these interactions when they happen multiple time a second, while still being responsive and while still running pathfinding, scheduling, time compression, adjusting the needs/mood of dwarves and doing world simulation in background is a huge challenge.

So in order to have good performance there are a few things I must do and a few that you must do.

I need to be very careful with my ambitions. The above mentioned pretty terrain renderer was too much. I replaced it with a new light on resources one. This one uses color blending and some visual illusions to give a less rectangular and Lego like aspect to the terrain:

Sure, the predominant theme is still rectangles, but now they seem to be varied in shape. Here is a close up where the black spot seems smaller than the yellow one, even though they are exactly the same size:

Once in  a while this system produces a rather ugly transition, but this is not a problem I'll worry about. I also removed blending from selections, making them perfectly rectangular and with clear edges. This reinforces the illusion that "this a free form shaped world where dwarves select perfectly rectangular shapes because this is how they think and plan their tasks" rather than "this is a world made out of rectangles":

The next thing I must do is have a very clear idea of a target performance on the target hardware and make sure not to sacrifice visual quality in order to achieve higher and unneeded performance. My engine can handle 90000 items and more. But a normal map won't have such and item density. Rather than scaling my engine to handle absurdly high item densities well and have extremely good and unneeded performance under normal densities, I'll scale it so it has very good performance under normal densities and runs acceptably with absurd densities.

The performance will also be based on the main camera: the top down one (and later the isometric one). Top down is a lot faster than first person because it has a narrower field of view and frustum culling works extremely well with it. Furthermore, in top down you can't really turn on very extreme rendering ranges because you won't be able to see what you need to if you do.

The first person camera is a bonus. A very useful bonus! In first person mode you'll see farther but have lower performance and you may wish to increase the rendering range because it benefits you, unlike in top down mode.

So for hardware compatible with the minimal setup (which I have not precisely determined yet), my target is to have a smooth 60+ FPS in top down mode. In first person mode I'll accept lower performance, that is 40+ FPS. This is the bare minimum. If you have the required hardware and you don't have this performance I did not do a good job with the engine because I did not reach my target.

If you are curious, on my laptop I am getting 200-400 FPS in top down mode and 80-150 FPS in first person mode. These measurements are for fullscreen.

Also, in the future I'll make sure to cater for people with a thousand dollar video cards too. This means better quality models, postprocessing effects and very GPU intensive but pretty LOD switching profile.

So taking these into consideration I perfected the way of mixing smooth and flat shading in the same model to obtain the best look possible. Analyzing the item density requirements I am now comfortable with using meshes with up to 120% polygon count. I don't want to sacrifice a minimum level of visual quality for some unrealistic performance expectations. Normal games won't have 90000 items at once. So I made sure to obtain the best looking barrels and also managed to mostly eliminate the seem:

If you look carefully you can still see it, but I am fine with it. Annoyingly enough, from a distance it is more visible than from close up.

I did not manage to get antialiasing to work yet with Irrlicht. But the engine is capable of it. Using the NVidia Control Panel I forced the game to use antialising. Here is a set of barrels as rendered normally:

You can notice the aliasing effect at the edges of the barrels. And now with anti-aliasing:

This looks a lot better! There is a noticeable performance hit, but my laptop that routinely runs the game at 150+ FPS under normal circumstances does not care. I'll try to add options for direct antialising, but for now using the NVidia Control Panel will have to do.

I remodeled the mesh for the dawnsider:

I am not 100% happy with it, but it is better than the previous one. Here is an alternative dawnsider when bearing fruit.

I adjusted the weapons rack. It has the same concept, but now the polygon count is in line with the rest of the objects:

The armor stand was very hard to see from top down view so I made it thicker. Anyway, these are dwarves, not elves! They need thick and bulky armor stands. Maybe I'll increase the width even more:

I also modeled a very basic bed. It is just placeholder and has no sheets on top, but here it is:

That's about it for what I modeled, but then Bryan sent this to me:

A workshop! A very high quality and detailed workshop! Thank you Bryan! The patch that finally brings workshops to Dwarves & Holes has long since been heralded as the Messianic patch. If you have an interactive world with raw resources and workshops to process them, you are just about done! Am I right? Am I right? Guys? Am I right? Hello?

In all seriousness, workshops will be a major milestone, and while they are not planned until version 0.3.5-0.4, starting to think and plan a little the modelling part for it even now can't hurt. This workshop is too high polygon for starters. While you will never have a huge amount of workshops at once, this workshop has at least ten times the polygon count as it should. I'll reduce the polygon count and send a modified model back to Bryan when I have time. Another concern is the height of it. The little legs are probably not a good idea and I'll adjust it to suit the height of a block and relative size of a dwarf. That is if I still allow for single Z level workshops. Another concern is that I might add levels to workshops. When starting out, you create a low level and less productive workshop. Latter when you have time and resources, you will upgrade it. I'm thinking improvised, normal and finally masterwork workshops. The way this is modeled, with all the detail, it does not look like something a dwarf threw together in a few hours. This looks like the highest level workshop possible with all the detail. And a final concern is that while I want workshops to have a fixed size, all workshops will come with their built in variable sized rectangular stockpile area. This is major change and departure from the DF workshop model and I will detail it some other time.

Short version: the model looks great, hits all the right ideas with the detail, but I need to remove the feet, scale it on the Z axis and reduce polygon count.

So putting things together, after a healthy dose of optimization, here is how a first person fly by render of the game looks with a 300x300x100 map, with the engine trying to render up to 60 levels:

As you can see polygon count is not so high and memory consumption is lower. While item density has been  adjusted a little (there were too many trees before) this scene is largely equivalent to the previous one that ran poorly and had the white trees. The trees are too dark because I did this on a very bright monitor and did not notice the dark colors. An easy fix. Most of the new items have no low LOD meshes yet so the polygon count is a lot higher than it should be. This includes the trees. I need to figure out a way to do low poly distant trees that do not look like crap. 

In the next screenshot we have a more zoomed in view of the same map and thus performance is a lot better:

These two screenshots do not contain the adjusted dawnsider, weapons rack and armor stand, the bed and workshops are not present, but you can still make out most of the meshes.

So what do you think? I say with a little better terrain shading that takes into account the position of the sun this engine has the potential to look quite good! I'll create some screenshots with shadows too, but shadows are something that I did not manage yet to get working.

Screens of the day 23 - Some new model tests

I've got my desktop back! It was repaired a few days ago but we were hit by an uncharacteristically snowy winter. We did not have such a winter in a lot of years and I had little inclination to spend time on the streets, even if that meant recovering my PC. But anyway it is back and works apparently well. I missed it! I am going to give it a name now! The desktop has been repaired and has two extra coolers so now it is loud as hell and greatly effects the temperature in the room. Having used my laptop for development during this period I can tell that my laptop is an awesome piece of hardware with sufficient performance. Well, it is a 600€ laptop. Still, using a 22 inch monitor on my desktop is a lot more comfortable that the 15 inch laptop screen. Just need some earmuffs now...

When my desktop broke it killed two 650 GB (not GiB) hard disks. There is world wide hard disk crisis (which I feel is partially artificial because of retailers panicking and buying more hard disks than the real demand) so I won't buy new disks right now because they are 2-4 times more expensive than before the crisis. So I am stuck with a 320 GB hard disk on my desktop. I have the habit of creating a partition for my Windows and a separate partition for everything else so I can format and reinstall Windows at will without having to back up data. But this time I made a single 200 GiB partition for everything. What am I going to do with the rest of the 90+ GiB? Install a few version of Linux and see if I can get the Linux version working. Using virtual machines is not a good idea at first because my game requires a lot of GPU power and hardware support under Linux is hardware support under Linux. Sigh...

Anyway, Snapshot 5 is buggy as hell. I fixed most of the bugs but there are still a few big ones left. And I've been working on some fairly interesting things. One evening after fixing some bugs I started playing around with the game and without noticing 30 minutes have passed! Everything is starting to come together and I think I'm only 1-2 snapshots away from a version that puts together everything you have ever seen in one of my videos or posts regarding the 3D engine. And about 3-4 snapshot away from finishing stage 1 creation mode. This version will be officially designated as version 0.2.5, while a stage 2 creation mode will be version 0.3.

Most of the new features and tweaks are only half way done so I can't really talk about them yet. I already have a 14 point changelog. While there are a few very interesting changes there, one of the more minor changes overshadows I feel even the huge ones: scrolling and zooming has been smoothed out. No more jerky camera movement. And using fullscreen without vertical synchronization has significantly less screen tearing. Before screen tearing was so bad that turning off vertical synchronization in fullscreen mode was a very bad choice. Probably no one could play like that without rage quiting and probably getting a headache. Still, the amount of changes is comprehensive enough that I probably won't finish and test everything by Monday, so it may turn out that Snapshot 6 will be delayed/kept private. I don't want another super buggy Snapshot 5.

So what can I talk about? Hmmm...

Here, have some new model concepts:

The above thing is a weapons rack. It is a little bit high poly, but I generally like the shape. It is meant to be made out of wood, maybe with the little things that stick out make out of metal. for now I am not planning to add stone weapon racks, but if I ever do they need to look differently.

This model combines smooth shading with flat shading. I have no idea if my engine likes this and the item will turn out looking OK. This render uses shadows so in the engine it will look slightly worse because of the lack of shadows. I really need to find a way to bake the shadows into the textures and see if this looks good. This will not work with dynamic lights, but for now I have no such things. Actually, I have absolutely no idea how all the items I am going to show today look in game because I haven't had the opportunity to check them out yet. The items are modeled quite amateurishly and are not optimized.

Above we have an armor stand. Low poly but it looks good. I need to add a little rounding to the bottom of the central pillar because it looks too flat but overall I am happy with it. Made out of wood.

Above we have a wooden chair. Low poly again and this model really lends itself to the "diversity" approach, that is making a lot of alternative models. The proportions are a little bit of, but generally it is good.

And finally we have the wooden table:

I wanted it short and stumpy but I don't like the result. The wooden models were designed to have a different shape from the stone ones. No effort has been put in yet to make sure that items look good and are instantly recognizable from a top down view, because this is how you will spend most of your time interacting with the world.

Still no luck with finding a modeler. I really need someone with good artistic vision to make sure that all items are consistent and look like they were part of the same universe. If said artist can also create more unique aesthetics so that everything not only looks consistent but also has personality, that would be just spectacular. Meanwhile you'll have to do with my models.

In my next screen post I'll show you the items in game.

76 – Uber graphics

So how do I spend the first day of my short break away from Dwarves & Holes? By implementing the largest and most comprehensive graphics overhaul yet! Even if at the moment of writing this Snapshot 5 is still not approved on IndieDB. I submitted it yesterday.

The first step in this overhaul was replacing the textures. I did another run optimizing the texture atlas system and then used some textures from two distinct landscapes from Ryzom. The assets can be downloaded from here and are available under Creative Commons License Attribution-Share Alike 3.0. They do a fine job of being test textures so I can figure out more precisely what I need. Now I know better and once I find an artist I can tell him what I want. The textures are not quite perfect, especially since some of the ones I have picked don't tile perfectly, but still the result is instantly noticeable and a big step forward. Some minor mip mapping issues are present, but they are hardly noticeable.

Still, even though the textures now look great, the game still did not look any better. While you were staring straight at a texture it was great, but as a whole it did not seem to improve that much.

So I implemented smooth transitions between blocks! This sure did the trick! I think it has a greater impact than any texture I could use. And smooth transitions plus good textures made a huuuuuuuuuuge impact. The game no longer looks like from 1999! I had some problems where different texture kinds met with the smoothing, partially because not all textures tile but mostly because of different color schemes, so I ended up gray-scaling all textures and normalizing them.

There was one last piece to the puzzle: the borders. Borders were blurry, had huge mip mapping issues and often when two borders met at an angle of 90 degrees the corner was missing. I set of to fix these issues. Borders are represented by masking patterns, so I needed a ton of such "tiles" in an atlas to represent all possible combinations. Also, to properly render borders 1024 precise floating point coordinates must be used. I did not do this by hand: I created a 92 line little program that created the masks and calculated the coordinates.

The new borders cover all the cases and while there is some variation across GPUs, they are generally clear and crisp, without mip mapping issues and as black as the night! Almost as black as that weird super black poo you find sometimes in the middle of the field!

So many changes and not a single picture? I'm such a tease!

I only applied transitions to top sides. The rest are barely visible in top down mode so you wouldn't notice the transitions. I'll figure out something for first person mode. Even so, these effect caused a dip in performance. I also temporarily removed another optimization I had for ages. It is a great optimization but makes modifying the code very cumbersome. I'll reenable it before a major release. 

Together these changes have dropped my maximum FPS by 40, from 250 to 210. Average FPS seems to take a drop of about 20, but still around 150-170. Needless to say: all these new features were not optimized at all so in the future performance might increase.

I also added a bunch of semi-useless options to the launcher:

Modern GPUs are great at all of these so no real use disabling them. If your GPU can't handle trilinear filtering you have no chance of running this game. Still, it is a PC game so extra options can't hurt.

Snapshot 5 is not approved yet but I already found and fixed a bug. The fix will be in Snapshot 6. Also did that major code cleanup I was talking about last time.

So it is feedback time! Watch the video at 720p if possible and give me your honest opinion. All of you! It is not too late to go back to the drawing board.

75 – Diversity

It's time for Snapshot 5! I was not happy with the way the last video turned out. That makes sense since it was an early preview, but still... I focused the rest of the dev time to make block interaction better. And I love the consequences the floor/wall merger had on the structure of the world and the way you interact with it. I foresee some small problems with shallow streams, but I'll fix those when they arise.

I went ahead and properly implemented this merger, getting rid of all floor interactions in the game and replacing them with wall interactions. You need to do something with a floor? Modify either the block bellow your feet or the surface information for that block. This is done automatically so you don't need to switch current Z levels. Fixing all the bugs that came after this merger took a lot of time and was fairly frustrating.

I went back and redone the border system. You saw the border system a long time ago, but then it was just a prototype secluded to the engine, without interacting with the "real" world of the game. Back then I used a fairly basic hack and had hardcoded variants for the floor textures with baked in borders. So since a face can have up to four borders, I needed 16 different textures. This time I only have a texture and I am blending the borders onto them on the fly as they are needed. With the new system I can blend anything on top of a texture as I am rendering it. So in the future expect to see small icons and other helpers on top of you blocks. The hest part is that the blending occurs at render stage and can be placed on top of any texture without the need to create a new one, so it is very efficient memory wise. I got rid of over 9 MiBs of PNG textures this way, so Snapshot 5 is going to be light and breezy. Going commando can do that to you.

To make the picture complete, I started making the final implementation for multiple level rendering. By default you get your current level and the one bellow rendered. Actually, you get N levels bellow and optionally N above, but for Snapshot 5 I am sticking with N = 1. This is the way I'm doing snapshots, gradually easing both you and the engine into the full set of features as they are developed. You can change levels again with keys ',' and '.'. When looking at your keyboard, don't look for ',' and '.'. Look for '<' and '>'. This way you'll find it easy to remember which keys to use and what they do. Pressing 'L' togles the rendering of the above levels. Default is off. 

The new levels have a higher polygon count so this lowers performance. Blending probably is also slower. Using fewer textures and thus context switches raises the performance. But without taking all these into consideration, terrain rendering should still be 8 times faster now. Terrain is only a small portion of the entire world and items eat up a lot more resources, but this still should gibe you 10-30 extra FPS. The factor of 8 is only for Snapshot 5, it will probably decrease to 2 or 4 as I start doing fancier stuff with the terrain.

I also did major code cleanup. Over the months a lot of crab accumulated in the 3D side of the code. I got rid of about 2000 lines of code and I should be able to get rid of another 2000 lines as soon as I am sure that they are not needed. I isolated terrain into its own class with a pleasant interface.

So putting all this together, we got this:

Snapshot 5 marks the one moth anniversary of Dwarves & Holes! Happy anniversary! It has been a good month.The change from Snapshot 1 to Snapshot 5 is substantial. This would make a proper first tech demo.

Also take a look at this picture:

Nothing special! Just ye good old barrel. But not everybody makes such barrels. Some like to make them more accessible:

Some crafters still like them accessible, but don't want the pets having access to it, so they place a heavy block on top of it:

(The small black rectangle hides a popup from my contact list than accidentally made it into the screenshot)

In some regions of the world, barrels are put on their side. To prevent them rolling away, the quickest solution is to place two heavy blocks on each side:

I you have time, you can create two carved out support pieces for it:

Or even create a barrel stand for optimal balance and stability:

I want such item diversity for everything. Right now it is not used for anything. Maybe I'll make Snapshot 5 pick one at random. But in the future I might assign them to geographic regions, so a new immigrant might come with a new way of making things. Some dwarves might like the new design. Some won't. Crafting blueprints will be exchanged and passional wars will be fought!

These are completely random and premature musings, but if at some point in the distant future Dwarves & Holes becomes a massive financial success, I would like to hire somebody to make all models and textures procedural so you won't get twice in a row an identical looking world. That would be crazy and a first I think!

I'll take a small break so Snapshot 6 will be light on new features. I want to playtest and bugfix. I want to model a basic armor stand and weapon's rack. Actually, I want mannequins, not armor stands, but armor stands are easier to model for now. Maybe I'll make it render a higher number of levels. Oh, and I have a budget right now so I'll be hitting my contact list hard over the head to try to find a (preferably) local payed modeller/texturer. This is going to be hard. Not because of the money. Finding a (semi) freelancer for such tasks who is both skilled and responsible is almost impossible. Such persons usually have a stable job and are not interested/have no time for side projects. And the ones who have time are usually jobless, desperate and lacking both on the skill side and the knowledge and experience side for the very special needs of modeling for a low poly 3D game. Your 30000 polygon bed might look great in Blender/3D Max, but it will blow my game up when I put tens of thousands in the same scene and you probably got some normals wrong. So probably I won't find anybody, but still I'll try.

If everything goes fine, Snapshot 5 should be up on IndieDB tomorrow.

74 – Floorapocalypse

I've been busy working on the floor system in an attempt to fully finish it. Texture wise I am at a crossroad: I have implemented a texture atlas system that would allow me to use a tile system similar to a topdown 2D game, only higher resolution. Creating a texture atlas that has zero texture bleeding is nigh impossible but I think I  found a good and adaptable formula. I can't really design textures and see if my system does not destroy the smooth transition between tiles at the same time, so I need to borrow and existing tileset for testing purposes only. I'll google it, but if you are aware of something I could use feel free to send me a link.

Meanwhile I am stuck with my ugly tileset.

When selecting floors I color coded the selection: red is soil, blue is stone. Sand and water will get their own color.

I enabled floor switching and I found that selections are particularly hard to do in the air because you don't have anything to select. So I created dummy selection markers that only appear when you are trying to select a non-existing surface (i.e. you want a selection where you will create something). I color coded these dummy floors with yellow.

I also found it very hard to navigate and tell on what level you actually are. I extended the yellow dummy surface for the entire level and made it transparent. It is very ugly and I'm not sure it helps. I need to try again dimming the non-current levels. If I can't manage to discover an easy to navigate and interact with solution, I might have to  redesign the game as a single deformable surface: you play only on top of a smooth surface and you can dig and deform it by other actions, but you are never on two separate heights at the same coordinates, except in some building. I hope it does not come to this.

Ignoring the above mention problems, I gave it a go and tasted my solution with color coding, stone and soil floors, building and destruction. And I did not like it! I never liked the floor/wall duality and still think in 3D it makes little sense. It always seemed convoluted and archaic. So I did a major and maybe controversial change: floors are gone!

Now the world geometry is formed out of blocks. A cell has a single block rather than a high black called a wall on top of a thin block called a floor. This might seem at first like a dumbing down, but there are exactly two consequences for this decision. You can still build; dig; smooth and engrave; have different engravings on different sides; fortify, build structures upon; have pressure plates/levers; canal; paintings; etc.

The first negative side effect is that you can't have a floor of one material bellow a wall of another material. This is a minor inconvenience.

The second consequence is that if you dig out two identical zones at the same position, but on different levels, you won't have two livable/walkable rooms. You'll have instead a higher cave-like hole. This means that your tactic for digging out rooms at the start of the game will be different and you'll want to skip some levels. But I can compensate with a few interesting things, like "sky-bridges" and brick multifloor structures. In the first stage you might want to live in a hole with a bed, but latter maybe you'll want to dig out huge areas, reinforce them and build huge cities. Buildings will have "thin" walls so a building wall will not occupy an entire cell.

Selection mechanism has been updated: if you select blocks that are on top of the same surface as your feet (the former walls) all visible sides will be highlighted. If you select block under your feet (the former floors) only their surface is highlighted. I found this to be very intuitive when dealing with interface and a mixed selection.

So what do you think about these changes? Give your honest opinion. As previously said, the depth of the interaction with the world should be maintained with this change and in the future I hope I can even add the possibility to adjust the height of a block.

I've been working on this since snapshot 4 and as soon as I got a half decent version working I posted and created a video. The video is very rough around the edges, more and experiment than anything else, but I hope I can get the systems in a better shape and have fully featured walls for snapshot 5: