Thursday, September 5, 2013

108 – Summer cleanup!

It's still summer, am I right?

So the plan for this week was to get the container list going and implement inventories, together with persistence with them, but I realized that containers are not the simplest form of statics and I should get the foundation finished first before attempting something more complicated.

I updated the module system to support list, for now a single list: statics! As I mentioned, most gameplay rules can be expressed as lists, the implications items derive from the nature of the list and their properties. The static list is the simplest list and it has the simplest implications: if a mesh is added tot he static list, that mesh will be available to be added to any map/interior on the terrain/floor and it will persist. Items can be added to separate lists at the same time, so if you define an item in a static list, you can have it in another list that let's say implies that the objects is dynamic and you can interact with it, but will expire if left alone for 20 minutes.

But first let me describe the complete structure of game objects:
  • Raw textures. These are stored on disk and they usually have the same name as their path on disk, but without extension, so that the engine can differentiate between multiple versions of the texture used for different things, like when running low on RAM the engine might decide to reduce the texture resolution. It could do this by loading only a lower mip-map, but alternatively it could load a specially prepared texture from disc meant to be used in this scenario. Right now textures can be loaded multiple times form the same path and you will get two independent copies, but if I determine that there is no need for this, the engine could only load one path as one texture in memory and use that for all the "copies". You do not have explicit control over textures as resources in the editor because they are governed by materials.
  • Materials. These are basically a structure describing how material should look and they usually have a diffuse texture and a normal map texture, but also properties relating the the strength of the material channels, specular highlights, etc. This is what you change when you need to change the way an object looks, not the raw texture objects and there is full support in the editor for this.
  • Meshes. These basically describe a 3D mesh as exported by a 3D modeling program, under a neutral stance (position, scale, rotation). Example: an upright barrel.  A mesh has multiple parts, and each can have a material. In the past a mesh object used to have properties related to its scale when introduced in the world and its physics properties, but starting with this version a mesh will be size and physics agnostic, just a raw data, similar to the raw textures. The old properties are moved to a new objects, that shares the same relation with meshes as the relationship between textures and materiel.
  • Object descriptors. These take a mesh and add size and physic properties to it. Let's say your exported mesh is a cylinder with dimensions (200, 450, 200). This size is resulting from the scale that the 3D artist used. It would be great if all meshes would be created with the same scale, but with an army of modelers that I'll surely have soon, this is not very likely. This size of the mesh informs its use in the descriptor, so you could choose that the descriptor would use the exact size given by the mesh. But more likely you'll choose another object. This way you could have a static barrel in an object descriptor which specifies that the barrel mesh will be used with static physics and a size of (0,8, 1.2, 08). This item descriptor will be added to a lists and then used. Or you could specify a static physics model that uniformly scales you input mesh, but to height of maximum 1.5 units. Or some ranges. Or make it dynamic and force it to a convex wrap. Or mobile cylinder with some dimensions.
  • A model instance. This is an actual instance of an object descriptor, inheriting all properties form it, but this will have it location in world and other properties, like content. The entire world is made up of model instances. Their specific properties are saved to disk to enable map loading.
Seem complicated, but here is the short version: a game object is an instance of an object descriptor, which is describes a mesh with ti physics properties and size. A mesh is a 3D model that can use multiple materials, which in turn will load multiple textures. Here is a screenshot with the mesh editor for the skybox, showing multiple materials:

So now we know what resources we can have and our final goal: use this information to populate our world with model instances. But what do you do with those instances? You add them to a slot. Currently there are two slots types:
  • The master model pool. This can handle anything. It can handle static meshes with one or multiple sub-parts and any shape or dynamic objects, but only with one submesh, and currently forced to become a box or a cylinder. As the need arises, I'll add spheres, convex wraps and compound shapes as a valid shape choice. The master pool can handle any number of objects in any mix and added in any order with perfect batching. Or at least how I currently understand how perfect batching should work. It frustum culls and can further cull your objects based on distance, but it does have the downside of having to go though all the objects that you added to it every time it tries to determine what to render. This fact combined with the ease it handles mobile objects makes the master pool an ideal candidate for mobile objects, which you should have less of than static entities.
  • The terrain chunk pool. Actually, there is one such pool per terrain chunk and it can theoretically hold the same things that the master pool can, but I never tested with mobile objects because this is meant for static objects. If you add a static object to the game world and choose the terrain pool as destination, only the chunk that contains the object will be updated. It has perfect batching, but only for the objects inside a chunk, chunk boundaries breaking perfect batching. This pool first frustum culls the entire pool as a single test as an optimization, so if you have a terrain chunk with 10000 objects and the entire chunk is outside your view frustum, with one test all 10000 objects are eliminated. After the chunk test, all objects inside are further tested if needed and there is distance based culling. So not all terrain pools must be fully traversed like in the case of the master pool, but it doesn't handle mobile objects well.

Again, to clarify, let me give you the short story on how objects should be added to the game world. If it is static and should be cull-able, add it to the terrain chunk pool. If it is mobile, add it to master model pool. The engine has default behaviors and since the object descriptor tells it how the object is meant to be used, you generally don't need to worry or know this stuff. Just select a descriptor and do level.Add(descriptor, position) as an example. The engine will select the pools as appropriate. And will do this efficiently, with constant cost and low overhead.

This is why I called this post summer cleanup. Most of these systems existed before, but now I formalized and cleanup everything related to them and added the new object descriptor system. This is what I wanted to say, so you can stop reading now.

but bellow I'll detail parts related to the development of this system and some distance based filtering. So after a first round of cleanup and creating support for object descriptors and adding list support to the module system and adding a first item to the static list, I needed to finish the support for adding such a descriptor. I started with the foundation dropping code, but made it use the new object descriptor system:

Above you can see this work, maybe a little bit too well. The pots are the size of foundations and terrain gets deformed and its texturing changed. I strip out the unneeded parts and I compare a pot added by the old system and the one that uses the object descriptors:

Great! thinks look the same. But they are not! Before the object descriptor system you needed to give some physics properties to a mesh directly, and if you did not, it defaulted to a box. The new descriptors coexist with the old properties, but since we selected an object from the static list, their physics behavior is quite different:

The one on the left is a static pot, that wills stay there until somebody or something removes it, while the one the right is a mobile one mapped to a box which you can just push around by walking. From this screenshot I can draw two conclusions: maybe, just maybe, the pot mesh is too high poly. But for sure, the physics mesh, which can be different from the render mesh is too high poly. I'll model a new physics mesh, that will have at most half as many vertices and use it as physics impostor.

The static mesh is in the terrain chunk pool, the dynamic one is in the master pool. They both have support for different ways of culling as described earlier, but also support for distance based filtering and actions. I created this very approximate grid of pots:

I intentionally set the distance on the pool to something very low, like 20 meters. See what happens as I move back:


And move back more:

Object further away that the 20 meters will get culled as I move about. This partially solves the problem of very ugly pop-in as you move around. Do you remember my old video where I did a 64 square kilometer map with hundreds of thousands of physics enabled objects and I ran diagonally from one corner to another with high speeds? Well, you could hardly tell because of the YouTube blurring that distant chunks would pop in all at once when running around. As in all the hundred of objects inside the boundaries of one chunk would appear as soon as you got close enough to the chunk. This was very immersion breaking and due to the chunk single frustum pre-test. Now, if I were to repeat the same test, internally the same pop-up would happen, but the engine would only render things in a set radius around you, so you would see object pop in one by one instead of the whole chunk. And in practice, I won't set the view distance to 20 meters.

I will set a generous radius and add other methods to enlarge the radius, like omitting very small objects and using meshes with LOD levels. In the above screenshots, instead of hiding the objects, I could switch to a lower LOD mesh. If I had one. I need to model that when I sit don to do the physics impostor.

Switching to a lower LOD will still have some pop in, but it will be minor. Full chunk pop in: VERY VERY BAD. Single object pop-in: PRETTY BAD. LOD transition and/or smart billboard: pretty good. That is what the AAA are using, so it will be good enough for me.

I couldn't test LOD transition, but I could test pixel shader complexity transition. I set the view distance to something very small, like 1 meter. If I am inside the radius, everything looks as normal:

If I take just one step back, the normal mapping effect is omitted:

Seeing this screenshot I realize that I did do a fair job at modeling and texturing this simple object, but I did a horrible job of normal mapping it. The pot looks a lot better without the normal map. I'll have to redo it.

Anyway, the change is very jarring. but as you increase the radius, the difference becomes less apparent, like at 5 meters:

At 10 meters, the change is barely noticeable:

So if I set a larger radius, the change won't be noticeable. But this won't give you any great performance benefit. The object will be only a few pixels wide and you will save a few pixel shaders instructions which in the long run will barely affect your framerate. The way you use this to actually get some benefit is use it to count the uses of the normal map based on the radius. If it is zero for a given raw texture, you can unload the texture and save on VRAM. As an example, if your object is so small that from 200 meters you can't tell the difference between high LOD and normal mapping and low LOD and no normal mapping, you make it so that at 220 meters it unloads the normal map and maybe replaces the full size diffuse texture with only 64x64 mip. The 20 meters extra units gives you a little bit of leeway if the texture streamer is very busy right now. It gives you a better change that by the time you reach the 200 meters mark , it is done streaming in the previously unloaded textures. And if it won't quite make it until 180 meters, you still won't notice the change, only when you are really looking for it.

No comments:

Post a Comment