76 – Uber graphics

So how do I spend the first day of my short break away from Dwarves & Holes? By implementing the largest and most comprehensive graphics overhaul yet! Even if at the moment of writing this Snapshot 5 is still not approved on IndieDB. I submitted it yesterday.

The first step in this overhaul was replacing the textures. I did another run optimizing the texture atlas system and then used some textures from two distinct landscapes from Ryzom. The assets can be downloaded from here and are available under Creative Commons License Attribution-Share Alike 3.0. They do a fine job of being test textures so I can figure out more precisely what I need. Now I know better and once I find an artist I can tell him what I want. The textures are not quite perfect, especially since some of the ones I have picked don't tile perfectly, but still the result is instantly noticeable and a big step forward. Some minor mip mapping issues are present, but they are hardly noticeable.

Still, even though the textures now look great, the game still did not look any better. While you were staring straight at a texture it was great, but as a whole it did not seem to improve that much.

So I implemented smooth transitions between blocks! This sure did the trick! I think it has a greater impact than any texture I could use. And smooth transitions plus good textures made a huuuuuuuuuuge impact. The game no longer looks like from 1999! I had some problems where different texture kinds met with the smoothing, partially because not all textures tile but mostly because of different color schemes, so I ended up gray-scaling all textures and normalizing them.

There was one last piece to the puzzle: the borders. Borders were blurry, had huge mip mapping issues and often when two borders met at an angle of 90 degrees the corner was missing. I set of to fix these issues. Borders are represented by masking patterns, so I needed a ton of such "tiles" in an atlas to represent all possible combinations. Also, to properly render borders 1024 precise floating point coordinates must be used. I did not do this by hand: I created a 92 line little program that created the masks and calculated the coordinates.

The new borders cover all the cases and while there is some variation across GPUs, they are generally clear and crisp, without mip mapping issues and as black as the night! Almost as black as that weird super black poo you find sometimes in the middle of the field!

So many changes and not a single picture? I'm such a tease!

I only applied transitions to top sides. The rest are barely visible in top down mode so you wouldn't notice the transitions. I'll figure out something for first person mode. Even so, these effect caused a dip in performance. I also temporarily removed another optimization I had for ages. It is a great optimization but makes modifying the code very cumbersome. I'll reenable it before a major release. 

Together these changes have dropped my maximum FPS by 40, from 250 to 210. Average FPS seems to take a drop of about 20, but still around 150-170. Needless to say: all these new features were not optimized at all so in the future performance might increase.

I also added a bunch of semi-useless options to the launcher:

Modern GPUs are great at all of these so no real use disabling them. If your GPU can't handle trilinear filtering you have no chance of running this game. Still, it is a PC game so extra options can't hurt.

Snapshot 5 is not approved yet but I already found and fixed a bug. The fix will be in Snapshot 6. Also did that major code cleanup I was talking about last time.

So it is feedback time! Watch the video at 720p if possible and give me your honest opinion. All of you! It is not too late to go back to the drawing board.

75 – Diversity

It's time for Snapshot 5! I was not happy with the way the last video turned out. That makes sense since it was an early preview, but still... I focused the rest of the dev time to make block interaction better. And I love the consequences the floor/wall merger had on the structure of the world and the way you interact with it. I foresee some small problems with shallow streams, but I'll fix those when they arise.

I went ahead and properly implemented this merger, getting rid of all floor interactions in the game and replacing them with wall interactions. You need to do something with a floor? Modify either the block bellow your feet or the surface information for that block. This is done automatically so you don't need to switch current Z levels. Fixing all the bugs that came after this merger took a lot of time and was fairly frustrating.

I went back and redone the border system. You saw the border system a long time ago, but then it was just a prototype secluded to the engine, without interacting with the "real" world of the game. Back then I used a fairly basic hack and had hardcoded variants for the floor textures with baked in borders. So since a face can have up to four borders, I needed 16 different textures. This time I only have a texture and I am blending the borders onto them on the fly as they are needed. With the new system I can blend anything on top of a texture as I am rendering it. So in the future expect to see small icons and other helpers on top of you blocks. The hest part is that the blending occurs at render stage and can be placed on top of any texture without the need to create a new one, so it is very efficient memory wise. I got rid of over 9 MiBs of PNG textures this way, so Snapshot 5 is going to be light and breezy. Going commando can do that to you.

To make the picture complete, I started making the final implementation for multiple level rendering. By default you get your current level and the one bellow rendered. Actually, you get N levels bellow and optionally N above, but for Snapshot 5 I am sticking with N = 1. This is the way I'm doing snapshots, gradually easing both you and the engine into the full set of features as they are developed. You can change levels again with keys ',' and '.'. When looking at your keyboard, don't look for ',' and '.'. Look for '<' and '>'. This way you'll find it easy to remember which keys to use and what they do. Pressing 'L' togles the rendering of the above levels. Default is off. 

The new levels have a higher polygon count so this lowers performance. Blending probably is also slower. Using fewer textures and thus context switches raises the performance. But without taking all these into consideration, terrain rendering should still be 8 times faster now. Terrain is only a small portion of the entire world and items eat up a lot more resources, but this still should gibe you 10-30 extra FPS. The factor of 8 is only for Snapshot 5, it will probably decrease to 2 or 4 as I start doing fancier stuff with the terrain.

I also did major code cleanup. Over the months a lot of crab accumulated in the 3D side of the code. I got rid of about 2000 lines of code and I should be able to get rid of another 2000 lines as soon as I am sure that they are not needed. I isolated terrain into its own class with a pleasant interface.

So putting all this together, we got this:

Snapshot 5 marks the one moth anniversary of Dwarves & Holes! Happy anniversary! It has been a good month.The change from Snapshot 1 to Snapshot 5 is substantial. This would make a proper first tech demo.

Also take a look at this picture:

Nothing special! Just ye good old barrel. But not everybody makes such barrels. Some like to make them more accessible:

Some crafters still like them accessible, but don't want the pets having access to it, so they place a heavy block on top of it:

(The small black rectangle hides a popup from my contact list than accidentally made it into the screenshot)

In some regions of the world, barrels are put on their side. To prevent them rolling away, the quickest solution is to place two heavy blocks on each side:

I you have time, you can create two carved out support pieces for it:

Or even create a barrel stand for optimal balance and stability:

I want such item diversity for everything. Right now it is not used for anything. Maybe I'll make Snapshot 5 pick one at random. But in the future I might assign them to geographic regions, so a new immigrant might come with a new way of making things. Some dwarves might like the new design. Some won't. Crafting blueprints will be exchanged and passional wars will be fought!

These are completely random and premature musings, but if at some point in the distant future Dwarves & Holes becomes a massive financial success, I would like to hire somebody to make all models and textures procedural so you won't get twice in a row an identical looking world. That would be crazy and a first I think!

I'll take a small break so Snapshot 6 will be light on new features. I want to playtest and bugfix. I want to model a basic armor stand and weapon's rack. Actually, I want mannequins, not armor stands, but armor stands are easier to model for now. Maybe I'll make it render a higher number of levels. Oh, and I have a budget right now so I'll be hitting my contact list hard over the head to try to find a (preferably) local payed modeller/texturer. This is going to be hard. Not because of the money. Finding a (semi) freelancer for such tasks who is both skilled and responsible is almost impossible. Such persons usually have a stable job and are not interested/have no time for side projects. And the ones who have time are usually jobless, desperate and lacking both on the skill side and the knowledge and experience side for the very special needs of modeling for a low poly 3D game. Your 30000 polygon bed might look great in Blender/3D Max, but it will blow my game up when I put tens of thousands in the same scene and you probably got some normals wrong. So probably I won't find anybody, but still I'll try.

If everything goes fine, Snapshot 5 should be up on IndieDB tomorrow.

74 – Floorapocalypse

I've been busy working on the floor system in an attempt to fully finish it. Texture wise I am at a crossroad: I have implemented a texture atlas system that would allow me to use a tile system similar to a topdown 2D game, only higher resolution. Creating a texture atlas that has zero texture bleeding is nigh impossible but I think I  found a good and adaptable formula. I can't really design textures and see if my system does not destroy the smooth transition between tiles at the same time, so I need to borrow and existing tileset for testing purposes only. I'll google it, but if you are aware of something I could use feel free to send me a link.

Meanwhile I am stuck with my ugly tileset.

When selecting floors I color coded the selection: red is soil, blue is stone. Sand and water will get their own color.

I enabled floor switching and I found that selections are particularly hard to do in the air because you don't have anything to select. So I created dummy selection markers that only appear when you are trying to select a non-existing surface (i.e. you want a selection where you will create something). I color coded these dummy floors with yellow.

I also found it very hard to navigate and tell on what level you actually are. I extended the yellow dummy surface for the entire level and made it transparent. It is very ugly and I'm not sure it helps. I need to try again dimming the non-current levels. If I can't manage to discover an easy to navigate and interact with solution, I might have to  redesign the game as a single deformable surface: you play only on top of a smooth surface and you can dig and deform it by other actions, but you are never on two separate heights at the same coordinates, except in some building. I hope it does not come to this.

Ignoring the above mention problems, I gave it a go and tasted my solution with color coding, stone and soil floors, building and destruction. And I did not like it! I never liked the floor/wall duality and still think in 3D it makes little sense. It always seemed convoluted and archaic. So I did a major and maybe controversial change: floors are gone!

Now the world geometry is formed out of blocks. A cell has a single block rather than a high black called a wall on top of a thin block called a floor. This might seem at first like a dumbing down, but there are exactly two consequences for this decision. You can still build; dig; smooth and engrave; have different engravings on different sides; fortify, build structures upon; have pressure plates/levers; canal; paintings; etc.

The first negative side effect is that you can't have a floor of one material bellow a wall of another material. This is a minor inconvenience.

The second consequence is that if you dig out two identical zones at the same position, but on different levels, you won't have two livable/walkable rooms. You'll have instead a higher cave-like hole. This means that your tactic for digging out rooms at the start of the game will be different and you'll want to skip some levels. But I can compensate with a few interesting things, like "sky-bridges" and brick multifloor structures. In the first stage you might want to live in a hole with a bed, but latter maybe you'll want to dig out huge areas, reinforce them and build huge cities. Buildings will have "thin" walls so a building wall will not occupy an entire cell.

Selection mechanism has been updated: if you select blocks that are on top of the same surface as your feet (the former walls) all visible sides will be highlighted. If you select block under your feet (the former floors) only their surface is highlighted. I found this to be very intuitive when dealing with interface and a mixed selection.

So what do you think about these changes? Give your honest opinion. As previously said, the depth of the interaction with the world should be maintained with this change and in the future I hope I can even add the possibility to adjust the height of a block.

I've been working on this since snapshot 4 and as soon as I got a half decent version working I posted and created a video. The video is very rough around the edges, more and experiment than anything else, but I hope I can get the systems in a better shape and have fully featured walls for snapshot 5:

73 – When it comes to coding I am!

Here it is, Snapshot number 4:

http://www.indiedb.com/games/dwarves-holes/downloads/snapshot-4-released

Better than ever :P!

Let's go thought he changelog:

• New LOD system. I talked about it extensively. I get he feeling that this system will serve me well for a long time. But I've been mapping out the weight of the nodes in my mind as I was testing and I think I've discovered a counterproductive pattern. I really need to write a quick real-time LOD level visualizer for the entire map, like a mini-map, so I can prove this theory. If it turns out to be true, I already know what he new weights should be in order to improve visual quality even further, but I am not going to do it now. Untextured objects have quite obvious LOD transitions. Here is a video I recorded earlier this week, so check it out and it's ample annotations:

• The Options dialog now has an extra LOD setting: Ultra.
• Recreated all the low LOD meshes for items to be better suited for the the new LOD system.
• As traditional for a new snapshot, a new GUI. I'll talk about it in detail another time, after I make sure it is stable.
• Keyboard bindings have changed around quite a lot.
• Optimized shallow item population algorithm probably to the max!
• Optimized floor building algorithm.
• Because I am trying to finish floors 100%, this update focuses on floors. Walls are thus temporarily removed so you can interact with floors.
• Stone floors are back. You can change between stone and soil floors with a few new tools.
• Implemented shallow floor building. While shallow building can't always be used, when it can it stacks with the above mention floor optimization, giving an impressive boost in performance. By combining these two techniques the floor system is 90% there at catching up to the item system. I still need to re add smoothing/engraving and add roads. All mouse based selection updates on floors greatly benefit from this stacking.
• Using the above two mentioned optimization, I mad grass growth about 100 times smoother. Updating the entire surface of the map (grass only grows on the surface) when every cell can have grass takes a single digit number of milliseconds, depending on CPU power. If in the long run it still turns out to be too slow, I can always break the update process into two. As said in this video, grass is still a little bit simplistic, but it is a good start:

• BUGFIX: Fixed crash when releasing the right mouse button outside of the main window.
• BUGFIX:  Hopefully fixed most of the weird behavior when releasing the right mouse button outside of the main window by implementing a rudimentary mouse capture system. Only time will tell if this is enough. And testing. Test it!
• Implemented middle mouse button click support (still no mouse wheel support, so no zooming with the mouse wheel).
• Made the window (when not running in full screen mode) resizable. The GUI does not scale or reposition yet.

I also made a quite random video:

I am fairly confident that by snapshot 5/6 I'll have most of the wold interaction tools in place. One more snapshot after that for workshops and I can get started on them dwarves.

Screens of the day 22 - I am a real artist... derp

I just love the new LOD switcher! It works great. It has better visual quality and uses considerably less CPU to update LOD levels. Everything is quite snappy. It even uses slightly less resources, so it even renders faster. Pop-in is not completely gone, but I am happy with it's current levels. So I have an engine which supports tens of thousand of objects without bill boarding and pop-in is (IMHO) minor. Has anyone else created such and engine before? I want to find the best there is at stuff like this. Please link me to such and engine or game if you know about it's existence. The only thing that comes to mind is the Total War series.

I also managed to adjust the LOD system it to work for top down mode. It is not quite as good as the old one for top down, but rather than looking back I'll keep it for all modes.

I also tested fly camera and the LOD switcher works great. While doing this I realized that I was still running on the quarter map, so I redid the experiments on full map. It still runs great, but 90K objects on "ultra" sure gives the good old a run for the money. I am getting between 50 and 60 FPS under these settings. But with only 22.5K items ultra works great! I don't know how many object a normal map will have, but if it's under 15k I'll probably add "Super Duper Ultra" mode too.

I tried to take the finished model of the barrel made by BrewStew and adjust it for the new LOD levels. Near the middle point I made a mess of things and I had to revert back tot the base model and start over. This after an hour of work. Second time it only took me minutes. After you get some practice with Blender you can become quite fast.

Confident in my abilities, I decide it is time to do my first ever texturing job. I chose the chest. Under Blender the results look absolutely horrible and I am ashamed of them. In game it looks a lot better. Still  a poor quality texture, but it is OK for my first try:

Of course I went in with no plan, no measurements and no sense of proportions. I make extremely poor use of the texture's resolution and I managed to screw up the little handle:

How the hell are you supposed to open this chest? Good job, mister texturer.

Then I decide to test out the barrels. They look great, except for the old problem of the seam. I tried for about an hour to fix it and I couldn't. Then I modeled a new barrel. Not quite as good as BrewStew's, but not bad. I am no longer horrible at this. This didn't work either. So I modeled another one. For this one I had the theory that the seams that were marked inside Blender are the cause. And I was right, because after I fixed all the problems caused by the seam marking in Blender on the original barrel (just removing the markings does not fix this), we got this result:

There is still a small seam. Can you see it? But look at the chair to the right to see how bad it used to be. Here is the same picture with the seams marked:

I have a theory that these seams can be fixed/improved by texture work. I'll try and mirror the two corners of the texture where the seam is. This way the GPU should blend the border of a mirrored image, basically creating a tiling effect. If this does not work, still the new seams are barely noticeable.

I also need to export normals from Blender. Without Irrlicht computes it's own normals and quite frankly the results are not that good.

You will also notice that everything is more shiny and bright. I'm not sure about the shininess, but the brightness I like. I needed to stick a ton of lighting on objects to not cause them to be super dark. The change is done by different export options. Now maybe I can try alternative lighting. Probably still need the help of a professional. Somebody who's done lighting for games in the past, who can look at my meshes and give me advice on all the 9000 parameters lights and materials have.

I'm off to retexture the chest again and fix the seam on all objects now that I know how. Anyway, after about 6 hours messing around with Blender I am starting to get comfortable.

72 – WTF LOD case study, part two

Welcome back to part number two!

Let's see what we get if we use LOD switching with my traditional technique from top down view of a quarter map with 22500 chests:

Memory consumption of  85 MiB with 18 FPS. Of course we will get small fluctuation as we scroll around and zooming can have dramatic effect. Still a huge improvement upon no LOD switching. Here is a completely zoomed out view:

Memory use of 29 MiBs, but only 6 FPS. Here is a first person view:

As you can see, there is some gain but not so drastic as with top down.

Now using the new LOD system, we get these results:

Memory consumption has gone down a lot but the FPS barely feels it. With this item density and this weak computer I could work actual miracles and the FPS won't improve that much. I shouldn't give away all my secrets, but here it is:

So after the practical experience of playing around with the new LOD in motion and getting a feel for the distances, I recreate some of the low detail meshes and create four performance profiles. Some of the new meshes are not quite as low as they could be. Rather than getting the best ugly performance available, I instead made sure that things still look good and if performance is not enough, you'll have to use a lower profile. Now, using a good computer, the "high" performance profile and adding back some color we get this:

"Medium" profile:

"Low" profile:

You may be wondering: "These look identical. What gives?". Well as static images there is not a lot of difference. In motion on the other hand, the level of the quality profile controls how much pop-in you will experience. And as object density gets lower, pop-in becomes more and more noticeable because you don't have as many object obstructing you visual path. On "high", pop-in is barely noticeable if you walk at a specific angle relative the bright side of the map.

But what about the fourth quality? Well this one is called "Ultra" and it will burn you up! Well, actually it is still quite tame and far from having zero LOD switching, but it is a huge step up from high both in quality and system requirements. I for one will be using Ultra because my laptop can handle it:

But what about top down mode. Unfortunately, the new LOD switcher does not like top down mode. It is designed for first person. But as you saw in the last post, top down mode is a lot faster, so I'll keep the old LOD switcher for top down and the new one for first person.

The quality profiles are not set in stone yet and I may adjust them. I also need to do intensive CPU profiling to make sure that the new LOD switcher does not over strain the CPU.

But most importantly, I need to get the chest textured and see if it still looks good with the new LOD switcher. UV mapping! This is going to be painful! And ugly!

71 – WTF LOD case study, part one

I need to make sure that slow but steady progress is made towards finishing the models required by the game and that the process of getting the assets in game is perfected. I am not going to bore you again with barrels, so today we'll use chests as an example. 

And be warned: wackiness will ensue. I have absolutely no explanation yet for some of the weird stuff that is going on. And I apologize for the length of the post.

Here is the chest mesh created by BrewStew:

Pretty high quality stuff! Probably too high. Blender reports that it has 184 vertices and 169 faces. But Blender faces are polygons while in the 3D engine we deal with triangles, so this mesh is actually 184 vertices and 1014 indices. How do we get from 169 faces to 1014 indices? Well if we apply a Decimate operator to the mesh and parametrize it so that it does nothing, leaving the mesh intact, it will report 338 faces, and... that leads us nowhere. Or wait! If blender has changed the emaning of faces from polygon to triangle, we get 338 * 3 = 1014 . On the other hand 169 * 6 = 1014. There, that's it! It makes sense. If all polygons have 4 vertices we need two triangles with 3 vertices each to represent that polygon. Each triangle has 3 indices to select the correct vertex, so we have number_of_faces * 6. We'll call this mesh A.

Taking this mesh and applying smoothing to it leads us to this:

Maybe not the best result. What if we only selectively smooth:

Look like this need more thought. I'll just go with flat shading for now and see what happens. Our goal is to try and reduce mesh complexity. First step is to remove that elevation shift in hope that we can compensate with a texture:

A lot of detail is lost, but the idea is to use this only for objects that are further away. We can go one step further and remove some back-facing handle detail:

The difference is minor for the handle. Blender says we have 128 vertices and 107 faces. In the engine we have 128 vertices and 618 indices, so the formula holds. This is mesh B.

The final step is to create a really low detail mesh:

This is just an approximation, but it is meant to be viewed from great distances where the loss of detail is hard to notice:

That speck still looks kind of like a chest. I'm thinking of removing the handles and baking them into the texture for this LOD level. Metrics: Blender 32/19, engine 32/114. This is mesh C.

In the past I used the Decimate operator to simplify meshes for low LOD. The problem with it is that you can only use so much before it starts to become really ugly and no longer resemble the object. This was the mesh produced by decimate and as you can see it had too much detail and I could't lower it more:

So far so good. I am barely starting to reach the part where I actually start to talk about subject of this post.

I tried to create an uniform stress test: a leveled plain where every single cell is occupied by a chest. Because of the memory requirements for this I am only testing on one quarter of a map. So all memory measurements should be multiplied by 4 to get accurate numbers for default sized maps. And we should have 22500 barrels at once in a scene.

And this is where the weirdness starts: exporting mesh A with either smooth shading or flat shading (as *.obj) and without exporting normals has the same look in game. I diff the two exports and see that one has the smooth flag set to off and the other has it set to 1. Going into my importer/exporter and deactivating Gouraud Shading makes everything ugly and setting the normalize normals flags has no effect. Recalculating normals with the CPU has no effect.

Now when exporting normals too things start to become really weird. Doing so with flat shading causes the vertex count to jump up to 652 while index count remains 1014. Exporting as *.3ds makes the vertex count be equal to the index count, as in 1014 and the up directions behave differently from *.obj. Exporting as *.dae has again 652 vertices, yet another up direction mapping that is different and a different scale factor. Awesome.

So I guess I'll continue the experiment with the smoothed look. First step is to fill the quarter map with high LOD meshes. I've updated the little counter to only count items and ignore terrain and I am running on a weak computer so I can really see the difference:

I won't list all the values, suffice to say that mesh A eats up 184 MiBs and has a FPS of 17 from top down view. From first person things are much worse:

While memory consumption and item count and complexity is exactly the same, from this view point frustum culling can't eliminate as much of the scene and we only get 3 FPS.

Let's repeat the experiment with mesh B:

We now only have 124 MiB memory use for item meshes and one extra frame/sec. I won't add another picture with top down view, but we get a FPS of 23. And one final experiment with mesh C:

Memory consumption is at 29 and we have 16 FPS, while from top down we have 46. Using this low detail mesh I had to make sure that I export the mesh smoothed and with normals because otherwise it was too flat (especially from top down view). Strangely, this had absolutely no effect on vertex count, like it did for mesh A.

Oh god, this post is soooooo long. I'm sorry. I am nowhere near done, so I have to break it up into two parts.

So join me next time when we see what can be done with LOD switching. With the help of this technique we can get high visual fidelity while maintaining a reasonable memory consumption. I'll show the benefits of my old LOD technique and see if I can get better results with a new idea I had for it.