100 – Announcing "Project Tworenas" and second round of snapshots

Hello there! Long time no see!

Since last time, I've been up to several things, but most of them not related to this project. One thing that was related to this was a pretty successful port to DirectX 9. For now I won't be porting further because I do not want to replace the physics engine.

In order to pick up the pace on this project, I am starting weekly or sometime bi-weekly snapshots. The goal of each snapshot is to add functionality and stability at a fast pace, while minimizing random rewrites (which I am prone to) and trying not to break anything from snapshot to snapshot.

It is also time to distance myself even further from the "Dwarves & Holes" project, so it is my pleasure to announce my new project code-named "Tworenas". It is not a new project, just a rename. For months actually I had a "C:\Tworenas" folder with the engine install.

Which brings me to the first issue: the engine is installed there because it can't handle yet loading resources from a changing folder. This and a few other things must still be improved before the first snapshot, so this week I'll consider this one Snapshot 0 and I won't release it. Next week will be Snapshot 1, the first public release.

But what is Tworenas about? It is meant to answer two questions: can you take a 3D game level editor, add some classical RPG mechanics to it and make it fun? And can a physics playground be fun? I already know the answer to the second question. Even in its current limited state, skiing around and checking out the physics interactions can be fun, especially when they surprise you. I remember the first time I walked fast towards a barrel rolling down hill and without desiring it, my character climbed on top of the barrel and before you know it I was "riding" a barrel down hill. But I don't know the answer to the first. That we'll find out in a few dozen snapshots.

Each snapshot will have a changelog posted, even Snaphot 0. Below I will detail the changes from the last video you saw to Snapshot 0. Some snapshots will also get an "engine features" video.

Changelog:

• Terrain uses vertex buffers again by default. For larger terrains, this can speed up terrain rendering by a factor of 3-4. Terrain can now change on the fly between using vertex buffers and arrays on a chunk by chunk basis.
• Terrain has customizable anisotropy for both close and distant chunks. Setting these values low, together with the above mentioned vertex buffers use has greatly increased rendering speed, especially on weaker hardware. It is now possible to have 60+ FPS on Intel HD chips, albeit at low to very low quality settings.
• Natural physics reaction of objects to terrain shape change. When you raise or lower the terrain, objects lying on top of it will shift and slide naturally with the curvature change of the terrain.
• Terrain persistence: the entire terrain is streamed live to the disk when changing it. When loading the same map, all changes will be preserved, together with camera position too. This basically gives you a persistent load/save system. Maps in this engine are different from other games, because you can always edit them on the fly. This means two things: baking of map data is not possible, since baking takes a long time and editing should be done in real time. This means that everything about the maps must be dynamic, from creation to rendering with lighting and all effects. The second thing that is related to saving is that maps are big: a 64 square kilometer map takes up 512 MiB with one height point at each meter. If I were to double the resolution, a single map would take 2 GiB on disk. Such large maps can't be compressed. So in the first versions at least, the engine won't use a traditional quicksave/quickload feature, one in which a new quicksave is a new save on disk. The disk would fill up really fast, plus the engine prides itself in very low loading times. So the engine will use a map based quicksave feature. Anytime you do something to the map, from changing the terrain to dropping a barrel, the map will be saved. There is no going back to a previous save state. This is done in the background, so you don't have to wait for an on screen spinning icon to stop spinning like in a lot of modern games. And saving will be disabled during combat, so if you die you will get back to the point a few milliseconds before the combat started. Snapshot 1 will feature only terrain saving, without clutter saving.
• Major RAM optimization. RAM consumption is a lot better now, but it wouldn't hurt to get it lower. I am investigating Vertex Texture Fetch in order to store terrain data more compactly, but I am yet to figure out a way to compute normals, tangents and binormals based on a bitmap heightmap in the vertex shader. This computation must also be fast enough to be done each frame for large scale terrains that also use a heavy pixel shader.
• To preserve memory, terrain texturing has had each channel reduces from floating point precision to a byte. Currently, I couldn't find any visual difference between the two, indicating that 8 bits/channel are enough.
• Terrain is slowly starting to transition from singleton to multiple terrain object support. Once this is done, some interesting things are planned, like small floating islands.
• Individual terrain vertex updates.
• Added mouse event support. Also used this mouse support to try and fix some of the Neoforce activate/deactivate GUI inconsistencies when going from first camera mode to window interaction with mouse mode.
• Editing a single height point has become much more atomic. In the past the entire chunk needed to be recalculated. Now, smart offset computation algorithms allow one update to result only in a few vertex changes. Unfortunately, this doesn't work yet with tangents. If a tangent change is needed for terrain, the entire chunk must be recalculated.

Caveats:

• C001: Terrain in the engine is chunk based. This means that some editing operations need to update more than one chunk for a singular change. This happens on chunk borders. Editing terrain at chunk borders is slower than in the middle of the chunk. The worse case scenario is a chunk border where four chunks must be updated for a single terrain write operation. Possible solution: some optimizations can still be done to improve this, but the corner case will always be slower.
• C002: Edited chunks change from vertex buffers to vertex arrays in anticipation of further changes. Editing a chunk once is rare, the user will probably do multiple edit operations, so it is a lot faster to not initially recreate the vertex buffers. Currently, the vertex buffers are not recreated after an edit unless you walk so far away from the chunk that it is no longer needed and then you come back. This means that if you continuously edit terrain for long periods of time, increasingly more and more will change from vertex buffers to vertex array, slowing rendering somewhat and eating more RAM. Possible solution: Future snapshots will periodically go over the list of chunks and if a chunk hasn't been edited in the last N seconds, it is probably safe to change it back to a vertex buffer. Temporary solution: this isn't a major issue and is not a memory leak, since the maximal wasted extra RAM from arrays is deterministic and based on map size. If you feel that after minutes/hours rendering has slowed down noticeably and you suspect this is the cause of it, just close the game and reload. Reloading gets rid off all arrays.
• C003: Since large terrains require a lot of streaming power, chunk streaming is spaced out, making sure that not a lot of chunks are streamed back to back. With the addition of terrain texturing persistence, the time to stream a chunk has increased a lot. This is only noticeable on slower computers with slow/full disks. Possible solution: As a first fix I'll space out streaming even more, first streaming in the geometry, taking a few milliseconds break, then streaming in terrain texturing. In the future, streaming will be multithreaded.
• C004: Streaming is currently based on the whole region that needs to be brought in. This means that if you were to teleport to a random part of the map, streaming would start somewhere in your vicinity, but not guaranteed near you. It might take up to 3 second for the terrain under your feet to become visible. It is still there and physics works just fine, only you can't see it. This is not noticeable that much since you can't teleport yet. Possible solution: biasing streaming based on camera distance and frustum intersection, so the first chunk that is streamed in is always the one bellow your feet, then the ones in front of you, finally the ones in the back. Also, if a teleportation spell has a short period in which character control is taken away from you, like a mediation or spell cast animation, chunk streaming spacing can be reduced to zero, streaming in at full disk read speed. Animation plus a fade in/fade out effect should be fast enough to stream in most of the new surroundings so that no loading screens are necessary.

Bugixes:

• A ton of smaller bug fixes all around the engine in preparation for snapshot release. From now on, bugs will be given a code so I can track them.

Bugs:

• B001: Turning on physics debug information at he same time with SMAA causes render to fail. Only an one pixel thin screen diagonal gets some color rendered. Temporary fix: turn off SMAA when using physics debug information.
• B002: When skying at high speed, after landing from a high jump the character can rarely get stuck in the terrain. Usually the character will start rising up, slowly getting ejected from the terrain, but sometimes this won't work. Temporary fix: moving the camera around a lot fast while repeatedly pressing jump should free you.