Friday, June 17, 2011

42 – IPS FTW!!!

I was supposed to do boulder stockpiles, but I finished IPS and I am too excited to not talk about it.

So, problem recap: create huge world, 1000x1000x60 squares and inspect RAM consumption. First it was too big, now with IPS it is just right:

Very nice for a final result: 380 MiB! It used to eat up almost 1.2 GiB RAM and right now I am even storing more properties into this tight space than I used to. It is not realistically possible to reduce RAM consumption any more while still keeping the capability of the engine to inspect any corner of the world with all of its properties. Actually, the compiler won't even allow me to create smaller objects on the current optimization settings. There are some very strict alignment rules and memory boundaries that the compiler will adhere to in order to optimize memory access. But a world of 1000x1000x60 actually consists of 60.000.000 floors and 60.000.000 cells, a total of 120.000.000 full featured objects that fit into 380 MiB RAM, without causing performance problems or memory fragmentation. I am more than happy with the results.

And normal worlds are not so big. I playtested the huge world, and it is not that fun, because every distance is huge and you spend most of your time watching dwarves dart about. Normal worlds used to be 300x300x40, and as such RAM consumption with IPS is very low:

A consumption of 42 MiB RAM is negligible now and so it was 10 years ago. So I decided to bump up normal word sizes to 300x300x100:

I will not increase default word size any longer for now, because 3D pathfinding data is not included yet in these 72 MiB used RAM. And I hope this will be the last Task Manager screenshot for a while.

Another side effect of IPS is that I need to test and often adjust every single result of tasks executed by dwarves. This is a huge bugfixing operation and I am almost done with all objects that have a visible presence on the map. But while doing that, I also created a very comprehensive and high level API for creating objects and getting their properties. The old API was low level and relied a little too much on hardcoding. If my game ever reaches the heights where people try to write extra tools for it, like reading its memory and interpreting the results, maybe I will release the API and a C++ library so those people can get direct support for interpreting DwarvesH maps. It would be the same library I am using (only slightly cleaned up), so the problem of third party tools falling behind with newer versions would not arise that often.

So now that the RAM problem is solved, I can get on with the rest of the work. Here is a screenshot of the new WIP inspection panel, that will show you all the meaningful data for cells that you click on and also will allow you to give context sensitive commands in the near future.


  1. That's a really good improvement in memory usage!!!

  2. congrats for the progress! good improvement indeed.

    what about testing on a computer that is not as smart as yours? :)

  3. To Anonymous #1:
    Thank you!

    I do not expect to get any problems with memory until I do something crazy with the world again, so this chapter should be concluded for now.

    To Anonymous #2:
    Incremental Proxy System/Storage. It is the name I have given to the technology that allowed me to greatly reduce memory consumption while keeping performance high. Some extra details in post 41.

    To simo:
    Thank you!

    I do test every few months on old machines to make sure I do not break backwards compatibility with old hardware.