So I probably need streaming

Hey, this is technically a screens/journal post, but since I managed to break terrain lighting there is no screenshot I currently wish to put out there :P. Still I have some data that I want to record for posterity.

Starting with the mention that sometimes I am stupid! My current terrain is not 2 square kilometers like I said two posts back! It is 2 kilometers wide. So that's 4 square kilometers, with 1-2 kilometers view distance. The speed calculation is still correct though.

So in my quest to reach that 10 square kilometers terrain (or maybe even limitless) I started profiling memory allocation and enhancing terrain load/save.

Now terrain is created by a new component that only has this responsibility and what it creates it always saves on disk, then it does some preparation for an eventual load of said data, then it clears up all resources.

My application starts with consuming 32 MiB of RAM. I don't know what eats up so much, ask XNA. Terrain generation for a heighmap of 2048x2048 takes this up to 65 MiB. After the save and cleanup phase,  I am left again with 32 MiB consumed RAM and a 16 MiB map on the disk. Because the maps are saved pre-chunked and I don't want to deal with non-square chunks while still keeping the game running under the XNA reach profile, a little bit from the map is always lost and the actual saved size is 2032x2032.

The saved data contains the height information, together with other useful information, like global min/max height, grid spacing, chunk bounding box, etc. After a load of the map memory consumption grows by about 16 MiB and before using the data, one can change some properties, like grid spacing, without any actual performance hit or need to process the data again.

So after loading the data, we are now ready to create the vertex data for the map in memory. This is when memory consumption jumps up to 199 MiB. Then we create the vertex and index buffer objects, raising memory to 690. 

Interestingly, BEPU is very smart about heighmap handling, and as long as the data loaded from disk is in a format it likes, it will not use any significant amount of extra RAM besides the actual height data.

Then finally after everything is loaded and prepared I am left with 777 MiB total memory consumption. Sometimes even over 800. It's over 800!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! OK, not funny  unless it is over 9000.

I did some ugly hacks, getting rid of some data and breaking device loss recovery in the process, but I did manage to get the RAM down to 620 MiB.

So this won't work.

I'll try a few things. The first solution will be to keep only basic heighmap in memory. Taking my current 4 square kilometers map and adding 3 more equal sized chunks for a total of 16 square kilometers would only eat up 64 MiB. Then I'll try to only load initially the terrain around the character, up until the fog limit. Then as I move around, I'll dynamically add and remove chunks from memory.

My guess is that this will be far too slow. So I'll probably have to add multithreading.