Wednesday, June 29, 2011

Z2: An introduction

This is going to be a fairly long off topic post about programming languages research and the introduction post for my new series. I understand that people come to this blog to read about DwarvesH, so I will be taking two measures to keep this new series from hogging the spotlight. First, I’ll prefix all posts from the series with “Z2” so people that are not interested can easily skip these posts. This series will be of very little interest to non programmers. Second, development for this research project will have several constrains, the most important one will be related to the length of the programming sessions. When I fell like it, I will enter a heightened mental mode and do a 60/120 minutes long frenzied programming session. On the clock. When the time runs out I’ll stop and I will only do this at most once a week. So plenty of time left for DwarvesH! Damn it, I really need a new name for the game ASAP!

But why programming languages research? The design of programming languages has been my passion for ages, long before I picked up the mantle of the dwarf. I had a previous attempt of fixing programming languages, more precisely C/C++ that I have abandoned for good reasons a few years ago. That attempt was called Z, and I am reviving it as a research project and as a discussion and meditation on the art of programming language design.

Why now? Well it all started a few months ago, when I read that some company has revealed its “Java killer” after two years of development. I don’t remember the name of the company or language and I am not going to Google it since it is better if I don’t name them. Most of the ideas were good. God knows Java is far from perfect. Some ideas were really bad and in my opinion very narrow minded. “Let’s do some drastic change that will probably not affect our work flow but would force this upon other people who might be profoundly affected by this” sums this up very well. But the conclusion is that their language is a nice, but niche step forward. But not after two years of development. And there is no way you can label this attempt as a “Java killer” after two years of development. Under a different label and shorter development time I would welcome their language. But under these conditions, all I can say is: way to completely miss the point guys! Have you ever used Java? Do you understand the principles behind the problematic areas and why they should be fixed? Because it seems like you do not.

That is all I am going to say about that. I did not mean to insult their effort and apologize for any parts of the above that may seem insulting. I just disagree. Probably I could have been more polite in expressing this. And I would find it extremely amusing, yet sad at the same time, if during my efforts I would prove that I do not understand either what is wrong with my language of choice and lack any kind of greater insight. Anyway, after reading their presentations, I started thinking about this series. I did not start it before because I was afraid of taking time away from DwarvesH. But by keeping the discussion largely theoretical and strict development times this won’t happen.

But let us go back in time even more. DOODLY DOODLY DOODLY DOOP! My first programming language was Visual Basic. I started learning Visual Basic (VB from now on) one or two years before my formal training in CS started in high school. BASIC has a very poor reputation and it has been claimed that it “ruined generations”. I can’t say anything about that because I was not there during that period, but I can assure you: VB has nothing to do with that unstructured programming mess that the first versions of BASIC were. VB was a decent language and VB 4 was a great RAD environment, but in retrospect I do find it very verbose and the standard non GUI library lacking. This was way before VB .Net. Back in the day VB and Delphi were IMHO the only true RADs out there, with Microsoft Visual C++ claiming to be that but failing. Sure, it may have been RAD when compared to Windows API, but VB and Delphi were in a league of their own. And MFC has the honor of being the worst API I have ever worked with.

Then my formal training started with Turbo Pascal (TP from now on). A good didactic language, with Borland Pascal actually being a good choice for development with its advanced tools and build modes that were lacking from TP. While they were teaching us the basics of programming, I was experimenting with DOS graphical drivers, VESA modes and GUI. Especially Mode X. There was this great compilation of Pascal sources called SWAG. It contained very advanced stuff that I had troubles understanding at first but it was very fun. I was forced to start learning assembly to truly benefit from all the interesting parts from SWAG. Became obsessed with writing the fastest “putPixel” procedure out there. Everybody should read “The Art of Assembly Language” by Randall Hyde. Even if you are not interested in assembly, there are a few very important and very well handled chapters on data representation. Maybe even give HLA a try, but people experienced with assembly will hate the inverted operand order. I know that it is actually not inverted and what is considered normal can be seen inverted (see binary code representation on x86), but still. OK, side not finished. Now where was I? Ah yes! I even experimented a little with C before they started teaching it at school.

I kept playing around with Pascal and adopted a GUI convention from a Pascal library which I think was called R3. The names are starting to get fuzzy. This library colored all my future GUI API design and was quite compatible in principles with Delphi GUI. I ported R3 to C++ and was doing a lot of STD/STL C++ and a lot more Delphi. Lol, he said STD.

Delphi became my primary development language, especially since it had those very powerful “enterprise” components and also several third party online component repositories. The language is not extremely expressive, but unlike C/C++, it had very few bad design decisions and very little to object about. A lot of nitpicking for sure. And please, do not even try and link me “Why Pascal is Not My Favorite Programming Language” by Brian W. Kernighan. That text is from 1981 and is so outdated that basically every single point he made can be rebutted by Delphi, which is actually not Pascal, but a dialect of Object Pascal, Object Pascal being an OOP dialect of Pascal. So a dialect of a dialect. It may have been true when it was written, but after so many years, that paper is more outdated and irrelevant to the state of Delphi than a paper entitled “Why America will never have a black president” written by the fourth future black president of America. Sure, Delphi is not perfect and it has been having a hard time in past few years, switching owners and all, but still I never understood why it did not become very popular.

On the other hand, I stopped using Delphi! I have gotten bitten by a very nasty bug that caused a very serious sickness: Linux! Delphi was not cross platform! Kylix was a mess with its Wine dependency, difficulty to install and the fact that it was abandoned by Borland, making increasingly harder to install and run as the Linux platform was moving ahead at a brisk pace. C++ was portable, but lacked a viable GUI library and the STL was as pleasant to use as a bear for dental flossing purposes. I could write a book on the “failure” of STL and I will probably write a post sometime. But it is not awful and not really a failure. Just very unpleasant. Please use it and do not roll your own general use container library. I’ll go mental if I see another custom list implementation that is inferior to STL. Every open source library seems to roll its own. And ranges fix most of the problems with iterators (except one that I do care about), so check out Andrei Alexandrescu’s papers on ranges. I did not follow the recent development of ranges, so please please please do not let it be a dead idea. With the underwhelming new C++0x standard, all that is missing is that ranges get abandoned for some nonobjective reason.

So without a main programming language, I was penduling between C++, Delphi, Java and other on a case by case basis.

So in my free time I started to work on Z, symbolically named as the language that would end all other languages. Na├»ve, I know, but still a great programming and design exercise. But not long before it would reach a critical feature set to take off, I discovered something on the Internet: the D language. I was awestruck and spent that day and late into the night to around 5 o’clock reading the design documents and the forums. Most of the time was spent nodding my head in approval, as around 80% of D’s design was identical to Z. Sometimes even the syntax and semantics were 100% the same. Now Walter Bright is a person who truly understand the problems with C++, an understanding derived (I wager) from experiencing its faults first hand and reacting to them, the same way I reacted. This is why our designs were so similar. But D was light years ahead of Z and had a lot more ideas. It did use garbage collection, a feature Z did not have and one that I was not particularly fond off, but still totally worth it. I have this old saying: “The world does not need yet another programming language”. So a few days after the discovery I killed Z and started using D. I did not regret my time spent on Z or its untimely death. This was way before D 1.0. Meanwhile I stopped using D. The long awaited version 1.0 seemed lackluster, version 2.0 took the language on the path of concurrency, a path I am not that interested about personally (and I use concurrency basically every day professionally) and most importantly, getting D to use other C and especially C++ libraries was extremely difficult. The main reason for this is that the object files were not compatible, so you needed to recompile those libraries with Digital Mars C++, probably a good compiler, but one that constantly chocked on sources that GCC and MSC had no problems with. And Digital Mars C++ was not actively developed.

So I was out of a main language again, even before I have gotten to use D enough to call it “main”. So I did the only reasonable thing that would allow me to find a cross platform language with great expressive power, great compatibility with other libraries and a fully featured GUI library: I tried them all. The big ones anyway. I think I went through 20-30 GUI libraries/language combos, researching their pros and cons. Another requirement was the language/library combo needed to have support and a fair user base, so I unfortunately had to discard a few very promising candidates that were apparently used only by a handful of people. I am not going to list all candidates and my conclusions now, maybe someday. In the end I settled for C# and .Net for when a virtual machine is desired and Ruby just to play around and do some light webdev for my secondary languages. C++ with Qt was the strongest candidate (even with the unfortunate use of MOC) for GUI work, but in the end I settled on U++. It is more modular and easier to tweak, while still having immense power. So in the end I still stuck by C++. But C++ in conjunction with U++ is a very clean C++, one that could be considered almost as clean syntax wise as a Java with operator overloading (and slight operator overloading abuse).

When I am saying main programming language (and library), I mean it as the language that I use for all my personal projects and the one I find that has the best principles put into practice. It is all about design, expressive power and most importantly elegance. If the language is not elegant or is not capable of providing an elegant subset it is wasted potential. This is why C++ can cover these requirements. It may be a horrible and ugly language in general, but one can use a subset from it, have a strong and consistent programming style and if you squint your eyes a little, it may even seem the perfect programming language (alcohol helps during the squinting process to facilitate the creation of this illusion).

How egotistical of me, filling three pages about what an experienced and great programmer I am. And I assure I censored myself and only included the most important key events. This is not an autobiography after all. I just felt like giving some background. Sharing is caring. Or scaring.


“The world does not need yet another programming language” 

So why am I doing a programming language? Is this not hypocritical? Well, no! I am not doing a programming language as much as researching and having theoretical discussions not as much about the failings of some specific construct in some languages, but more about the process of finding such failings, determining why they are failings and solving them while keeping the solution general. Any coding and actual language implementation is a simple exercise in grounding the theory a little in reality. And while the principles of this theoretical language, Z reborn, called Z2 (I know, I am not very good at picking names) may not always express this, the central idea is that C is a horrible language not fit off any practical purpose and its seed has corrupted C++. There are two good reasons to use C: you need very high performance low impact real time code very close to the machine or you are maintaining a C code base.

So the first principle I have already named above.

The second is also the second constrain on development. The code base will have limits put upon its maximum size. For starters it is going to be maximum 3000 lines of code. I will not allow it to go over only when some huge new feature comes. Then I’ll increase it to 5000. Some future increases after these are possible, but not too many. The source will always be short, very lightweight, easy to understand, didactic and very hackable. The Squirrel scripting language was at one point a fully featured language with only about 9000 lines of code. I don’t know how many lines a more recent version has, but I doubt it has blown up exponentially. Needless to say, any standard library code for the language will not be counted for this purpose.

All code will also be open sourced eventually. Every few months when I have time I’ll drop the code somewhere. And maybe I can do some binary releases more regularly, but DwarvesH has the top priority.

The language will be influenced by scripting languages. These languages have made a deep impact in the programming languages ecosystem and ignoring their influence would be silly. So even thought at its heart Z2 will be a very C++ styled language, its syntax will be more Ruby/Python/others inspired. But this does not matter. A lot of near religious buzz is generated by programmers around some conventions that are not important. One of the goals of Z2 will be to point a specific finger from its hand in the general direction of this nonsense. So no “indentation wars” or other minor yet blown up controversies will be given any attention. Indentation wars are particularly silly and a pet peeve of mine. Are we programmers or mice? No self respecting and professional programmers should ever have troubles following any kind of indentation from his favorite style to a style where everything is on the same line and never should he or she become profoundly disturbed by such styles or any kind of conventions. Especially not at the levels some people stick to their styles, turning them into crusades against everybody else. I can read any kind of indentation. I am not equally comfortable with all styles, but after a day or two I get over it and reach full performance. And I think no programmer with enough experience in the field has any excuse not to be fluent with all styles, or at least not become angry about them. My personal pet peeve it prefixing member variable with “m_”. I hate this convention and I think that proponents of it are completely missing the point of OOP. They are in one universe and the point is in a parallel universe, with the only thing making these two universes parallel is the fact they exist at the same time, not that they have even the slightest thing in common. Yet I do not get angry when seeing this convention. I avoid it though like the plague J.

So when there are two takes on the same idea, and one cannot objectively be declared a winner, Z2 will intentionally do both or neither, depending on case, in order to make o a point.

I will talk about one last topic and end this extremely long post right here, even though I did not systematically list all the principles of Z2. The compiler will take Z2 code and output C/C++ code. This will make it very portable and keep the source code small. This solution is not optimal, but perfect for a toy language. And yes, Z2 is a toy language. I will not be using it for real development and will not port DwarvesH to Z2. It is just the practical result of the theoretic talks related to programming languages and my two cents related to fixing C. An ideal solution would be to use LLVM. I have very high hopes for LLVM and I hope that one day it will be the messiah that will unite all languages the way JVM or CLI could never do. Muad'dib! Muad'dib! Muad'dib! But we are not there yet.

Tuesday, June 28, 2011

47 – Stocking some piles, part 3

I delayed the stone stockpiles video a little because it was not that interesting, especially after two posts about stockpiles. Post 43 gave the general final design and rules for stockpiles, using wood as example, while post 44 detailed food stockpiles and optimizations for small or light objects. Stone stockpiles are the same as wood stockpiles.

With one important caveat. Wood, food and other resources are what I call rare resources. For rare resources automatic hauling is desired. If a new resource is lying about and there is a free stockpile for it, a low priority hauling task will be added. But this would not work for stone. Stone is abundant. Virtually the entire map is made from stone cells. Every time you dig you get new stone resources. If stone hauling is automatic as in the case of rare resources, every time you dig you would add a hauling task if a stockpile is available. Dwarves would spend most of their time hauling stone and it would be impossible to maintain a free stone stockpile for some particular purpose.

To compensate the abundance of stone, stone hauling is not automatic. Digging through stone will leave stone boulders that are not scheduled for hauling. I have defined a new action that will allow you to manually designate stones for hauling. This option is called "Haul Boulders" and it is an option under floor/channel actions:

Stones are probably the only exception for automatic hauling. Ore may be the second exception, but I need to decide first the exact rarity of ore.

Another consequence of their special status is that boulders lying on the floor that do not have a manual hauling task enabled will not be considered for building operations that use stone. For such task only marked stones or stones that are already in stockpiles will be considered. This actually turns out to be a nice feature, since you will be able to easily choose specific stone. Maybe you want your building to use the same stone or form a pattern out of different colors. In the future, a menu based system will be introduced to fine tune your selection.

Monday, June 27, 2011

Screens of the day 10 - Procedural grassy floors

Like wow! Two posts in the same day. I am too excited to keep this under wraps until I try something meaningful in game with it. But please remember, two posts today. Keep them views rolling!

I went on to apply the same algorithm to grassy floors. These images having a different content, the parameters of the algorithm should be adjusted. But I could play around all day with them and not necessarily find something I consider close to optimal. So I just went with random parameters. Isn't it ironic that the choice of parameters that will define random transitions is left up to fate?

Here are four screenshots with one of the floors:

This is my favorite soil floor, visually speaking. Darker floors don't seem to get along that well with these parameters:

Grey floors seem foggy, almost surrealistic like this:

And finally, my least favorite soil, again not flattered by the parameter selection. I have a hard time deciding if this or the dark brown one is the worst looking:


46 – Crazy oak overload

I did not get to do any work during this extremely lazy weekend. And I am cut off from the single computer I have FRAPS on so I can't bring you the stone stockpiles video today. But I hope I can make it up with today's topic.




The following post contains a lot of oak. If you had any previous negative experiences ranging from mild nausea to seizures when confronted with bright flashing oaks or just oaks in general, please stop reading.


I have rather complex trees. Even without their produce, they still have leaves that can fall of or change color. In the current system a tree has 28 unique tiles. But these tiles are fixed, and for a single "leaf fullness level" I have only four distinct tiles for the transition from fully green leaves to fully brown (or other colors). This means that at 3 distinct dates, the tile for the tree will be changed.

But what if I wanted more? Making these tree tiles can be fairly tiresome. Or expensive. Or both, depending on the volume of tile you need. So let us try procedurally generating the transition. We start with the green tree and end with the brown tree and the rest is randomly filled in. I have an algorithm which has a few parameters that can tweak the outcome. Here are two random transition with basic settings:

The two images may seem identical, but there are small differences in detail. Of course, since the algorithm is random, there is no guarantee concerning the number of tiles you get:

The problem is that the transition is too rough, with bright pixels just popping up randomly. Let's use some blending to increase the smoothness:


Even more smoothness:

For the final two pictures I am increasing the smoothness to maximum. Even thought it seem to finish once it reaches 255 tiles, this is intentional. With these parameters, the algorithm can produce hundreds of thousand of unique tiles before it finishes:

I like the last two, but there are some problems still visible. The biggest one is that the yellow tint starts to dominate the image too soon. Unfortunately, I can't take a "I want the transition to feel like this" thought and figure out the right parameters. A lot of trial and error is needed, maybe even a smarter algorithm.

So what do you think? How much is too much? Are the four unique transition tiles enough? Maybe use the algorithm to smoothen the transition to a reasonable frame number, like 16 ore 32? Or use the whole range from the final two images? Or make it so that each new game has a new set of randomly generated transition tiles? Maybe even with random colors?

Ahhh, procedurally generated content. So many possibilities! Maybe I should create a video with the transition animated to better show case it when I get back to my FRAPS.

Friday, June 24, 2011

45 – Irrlicht GUI? More like Irrless GUI!

Am I right?

Stone stockpiles are too boring so ?I'll leave it for tomorrow.

A quick history lesson: in the beginning, when DwarvesH had just a few lines of code and was top down, I was using a GUI with U++ controls, which since I was compiling on Windows had Windows look & feel. As complexity and map sizes increased, I had to abandon "native" bliting and turned to Irrlicht for hardware acceleration. I couldn't get the Irrlicht and U++ event loop to play nice together and Irrlicht refused to go fullscreen if it was running in a host window, so this lead to the separation of the game from the editor into two different executables. The game was using Irrlicht widgets and the editor the far richer U++ widgets. But I grew tired of the Irrlicht GUI API. While I sincerely dislike Irrlicht API design, it is a very fine piece of software with indisputable advantages, so I thought that using the same API for GUI would not bother me that much. I was wrong. I quickly wrote a very hackish wrapper/bridge around Irrlicht GUI, having a U++ interface but using Irrlicht widgets behind the scenes. In time this solution turned out to be too much to maintain, so I replaced it with a very thin wrapper over Irrlicht, using both the widgets and the API style. Way to go around in circles, I know. So I started wrapping stuff to escape Irrlicht API design, and ended up with a wrapper system which was easy to maintain but had a similar API design.

Fast forward to a while ago, when I started correcting this mistake. I wrote a very light weight GUI system with U++ style API that was not using Irrlicht widgets but was using its drawing API. This solution was less than ideal, but with the emphasis I am putting on user interface, working with an API that I disliked would have certainly influenced your user experience in a negative way. The new GUI library is extremely lightweight, but also not very rich. This is not a problem, since my GUI uses simple widgets in complex ways, rather than using complex widgets.

The system has stood the test of time and right now it only has the following widgets: TopWindow, StaticRect, Button, Label and Option. For the rest, I am still using Irrlicht widgets, but they will be phased out.

What is the purpose of this post? To tell that the above mentioned custom 5 widgets have fully replaced their Irrlicht wrapper counterparts. I even deleted the wrapper classes. Button has a new default skin, which probably won't be the final skin used in the game, and the rest of widgets will get skined soon.

I leave you with a gallery of screenshots showing the new GUI. Keep in mind that I did not touch some of the windows in quite a while and their content is outdated. The menus used to set up a new world and expedition are particularly out of date since I did not work on them since my first video presentation.

See if you can tell Irrlicht and custom widgets apart!

Wednesday, June 22, 2011

44 – Stocking some piles, part 2


Scheduling is hard. This is why sometimes such systems turn out to have small caveats and strange behaviors. And in my project I try to compromise as little as possible.

Generally speaking, moving stuff to stockpiles is not that hard. It can be hard to make it fast, but the general idea is simple: select an item from the pool of hauling candidates, select an appropriate dwarf for the task, plot a path from the dwarf to the item, add a pick up task, plot a path to the destination stockpile, add a place item/update stockpile task and pray that while these actions are executed, a different dwarf does not decide to obstruct your path. This is the most basic way to do it, and after you get a good implementation for this you will most likely be fed up with scheduling and pathfinding. With this approach, if a source stockpile has multiple items a ping-ponging effect will appear: the dwarf will go from source to destination and back several times, until the source stockpile is depleted. For a lot of items this is what you want. Logs, boulders and other large or heavy items should be carried one at a time. Even if the object is uncumbersome enough to carry two, I would prefer carrying one without reducing your movement speed.

But for harvested plants, this ping-ponging effect is not desirable and just seems lazy. Here is a video showing it in action, with me pausing after every action to show how stockpiles update:

A better system is needed only for food and other light items. But a smarter system will be exponentially more complicated. Since I started the post saying that I will not compromise, I will embark on the path of non-compromising. A path that starts with smaller compromises! The absolute system will allow a dwarf to fill up his inventory and then place all items in appropriate stockpiles. Without wasting a lot of time and keeping your dwarf locked on this task. But an intermediary system will allow the dwarf to pick up as many items as he can from the same cell and attempt to place these before moving on to other cells. Here is another short video showing drastically reduced ping-ponging levels:

This is why food stockpiles were a lot more complicated to get right.

I still need to decide if it is worth it to visually differentiate harvested food sacks based on content.

And of course, the item browser will show food stockpiles the same way it works with wood, but since I am experimenting with short videos instead of images, I won't spoil this post with a screenshot.

Next post will be about stone stockpiles. After I have another stockpile post scheduled, but I will take a break showing some buildings, probably the kennels, in order to not overly bore people with so much stockpile talk.

Monday, June 20, 2011

43 – Stocking some piles, part 1

Let me start again by congratulating Irrlicht on being random. I was loading some textures, and I accidentally deleted a few of them. So what does a responsible little toolkit do? Drop FPS by about 70 of course. I can understand not showing an error or something, but dropping performance like this is very silly. *whine* *whine* *whine*

So in this short series I will be demonstrating the final implementation for stockpiles, followed by another short series for buildings. And after I will post an updated feature set for version 0.1, complete with color coding and progress report so I can see how close I am to the release feature wise. Some of the content in this series will be familiar, some new.

The new inspection panel plays a big part here, so let me show it first. Here we see it in action showing the information for a single floor tile:

We see the current coordinates, a preview of the floor, a description, the type of the material and the current grass coverage. Now I'll cut down a tree and see what the panel shows:

Nothing ground breaking here: we are informed that we have 5 pine logs and that there is a trunk in the cell. The interesting part comes when we involve stockpiles.

But before, I would like to talk about a change in the stockpile system that might turn out to be quite controversial. I've decided to make stockpiles nontraversable. A dwarf will no longer be able to enter the cell occupied by a stockpile, the same as if they try to enter a cell with a solid obstacle, like a wall. There is no technical limitation for this decision and in the past stockpiles were traversable. I just grew tired of the way stockpiles are set up usually: a huge rectangular sea of stuff. This change should lead to more interesting stockpile shapes and change the way you structure your fortress.

Back to wood stockpiles:

First the panel is informing us that we have a wood stockpile with 5 logs inside. Then, it tells us exactly what kind of logs: 5 pine logs. Let us cut down an oak:

As expected, we now have 5 pine logs and 5 oak logs in the stockpile. One more tree, this time a willow:

Just as expected! A wood stockpile can only hold 12 logs. The remaining 3 logs are still in their original cell:

Of course, you can create larger sections of wood stockpiles, but if you are not careful, since stockpiles are nontraversable, you can waste area if you create rectangular ones:

The to stockpiles in the middle are not reachable, so we wasted two squares there. Another thing to notice is that the panel grows to accommodate all content.

So now we can easily tell what logs are in a single pile. And no, I will not differentiate logs from different trees visually. Maybe sometime in the far future. But how can we get an overview of all our resources? Surely clicking on each stockpile is not that efficient. This is where the item browser comes in:

By browsing the appropriate tab, we see our total stockpiles of logs, together with small icons and color coding.

That's it for wood stockpiles. Join me next time for food stockpiles, which are surprisingly a lot more complicated.

So now that you read that, what do you think? I tried to make the system as easy to use as possible, especially for people who are not that happy with the way stockpiles work in other games and how involved the process can be.

Friday, June 17, 2011

42 – IPS FTW!!!

I was supposed to do boulder stockpiles, but I finished IPS and I am too excited to not talk about it.

So, problem recap: create huge world, 1000x1000x60 squares and inspect RAM consumption. First it was too big, now with IPS it is just right:

Very nice for a final result: 380 MiB! It used to eat up almost 1.2 GiB RAM and right now I am even storing more properties into this tight space than I used to. It is not realistically possible to reduce RAM consumption any more while still keeping the capability of the engine to inspect any corner of the world with all of its properties. Actually, the compiler won't even allow me to create smaller objects on the current optimization settings. There are some very strict alignment rules and memory boundaries that the compiler will adhere to in order to optimize memory access. But a world of 1000x1000x60 actually consists of 60.000.000 floors and 60.000.000 cells, a total of 120.000.000 full featured objects that fit into 380 MiB RAM, without causing performance problems or memory fragmentation. I am more than happy with the results.

And normal worlds are not so big. I playtested the huge world, and it is not that fun, because every distance is huge and you spend most of your time watching dwarves dart about. Normal worlds used to be 300x300x40, and as such RAM consumption with IPS is very low:

A consumption of 42 MiB RAM is negligible now and so it was 10 years ago. So I decided to bump up normal word sizes to 300x300x100:

I will not increase default word size any longer for now, because 3D pathfinding data is not included yet in these 72 MiB used RAM. And I hope this will be the last Task Manager screenshot for a while.

Another side effect of IPS is that I need to test and often adjust every single result of tasks executed by dwarves. This is a huge bugfixing operation and I am almost done with all objects that have a visible presence on the map. But while doing that, I also created a very comprehensive and high level API for creating objects and getting their properties. The old API was low level and relied a little too much on hardcoding. If my game ever reaches the heights where people try to write extra tools for it, like reading its memory and interpreting the results, maybe I will release the API and a C++ library so those people can get direct support for interpreting DwarvesH maps. It would be the same library I am using (only slightly cleaned up), so the problem of third party tools falling behind with newer versions would not arise that often.

So now that the RAM problem is solved, I can get on with the rest of the work. Here is a screenshot of the new WIP inspection panel, that will show you all the meaningful data for cells that you click on and also will allow you to give context sensitive commands in the near future.

Thursday, June 16, 2011

41 – RAMy Stony Sand

Oh my gosh! A real post! And the first one for this month. I am so excited! But I can hide it. Because I am over the Internet and you can't see me.

And I can't see you. Which can be a problem. While my blog is in no way popular, I am slowly getting a steady stream of comments. With YouTube messages, forum messages and direct e-mails, there is quite the communication going on. And Blogger is the weak link in this communication chain. It is very hard to communicate if I keep getting anonymous comments. So please, if it is not too much to ask, post while signed in. If you do not want to reveal you real sign in ID (for some strange reason; there is nothing I can do with it that would warrant such extreme security and privacy measures), make a fake one or something. It would certainly help not having several comments labeled as anonymous on the same post or on different posts.

So on with the show!

New RAM efficient storage system

As I said in one of the "Screens of the day" posts, I tried to increase map size to some ridiculous heights, like 1000x1000x60. The game is not designed to work with such huge maps because of the way task durations are balanced right now. But I found that with these dimensions, the game was eating up almost 1.2 GiB RAM. Completely unacceptable, considering both my future plans with it and that for 3D pathfinding it will eat up even more RAM. I set up to create a new system, which I have cleverly named IPS (Incremental Proxy System). Because it is clever. So am I. I only finished and tested the first iteration of IPS, which is not so efficient as the second one will be, but it is still an improvement:

That is roughly 610 MiB RAM. Well that is about half of the number that I found unacceptable. The second iteration is actually done, but unfortunately it is too buggy, so for now I'll only use this first version. The second should slice of at least another 200 RAM if everything goes according to plan. 

And here is the RAM consumption for a normal 300x300x40 map:

Negligible to say the least. It will use up even less RAM in the next version. After I finish this task, I will increase standard map size to 300x300x100 and add the fields required for 3D pathfinding and record the final RAM measurements.

Solution to stone problem

Another oldie here. I had a problem with stone digging. A problem of realism. When you set out to dig a cell made out of stone, you dig out a volume of 1.5 cubic meters. This stone needs to be usable afterwards for building stuff. But what to do with it. The obvious solution is to go with stockpiles. As walls are destroyed, the resulting boulders are hauled to stockpiles. This is not acceptable, since you would wind up with a system where digging takes a lot of time, most of it spent on hauling. The actual task of digging would be trivial when compared to the time and energy expediture of hauling. Not to mention that you would wind up moving the mountain from inside to outside or wherever your stockpile is.

I solved this problem at first in a rather poor way. Stone, after digging, would disappear, but you would still have access to it. So basically stone would move to a virtual storage, not findable anywhere in the world.

My new solution should be familiar to DF players, because it is basically the same thing. Not quite happy with it because it is a huge compromise, but I can't find a better system. When digging a wall, the dwarf hits the wall with his pick, until a boulder breaks off. The process continues until the cell is traversable, leaving the boulders on the floor. A little suspension of disbelief is required, because the 1.5 cubic meter volume is reduced to a smaller volume. So let's consider that you only deposit the usable boulders on the floor, with the unusable ones and other debris disappearing magically. Maybe in the future I'll have some refuse stockpiles filling up automatically as you continue to dig with ruble. And you will be able to pay trade caravans to relieve you of your surplus refuse. If I implement this this will happen automatically, without a dwarf having to walk from the freshly dug wall to the refuse stockpile. Here is the result of a huge dig operation on stone:

As you can see, digging though soil does not result in stone. The tiles are placeholder and too high for my tastes. I'll replace these with new ones that are also properly tinted. In this image, the floor and the resulting stone has different colors because it is not tinted, and it can be even more obvious when the stone has a darker color:

The resulting stone is fixed. Objects in the world are either fixed or not. Fixed objects are not automatically hauled, while the rest are automatically hauled if possible. Like logs resulting from cut trees: if there is stockpile, the logs will be moved there. Stone is fixed, so it will not be moved even if there is free place in a stone stockpile. You will need to manually move them. I will implement this for the next post, mimicking to log hauling system. Why not move them automatically? Because hauling is not fun. Dwarves do not suffer any movement penalty when walking on boulders. It has something to do with the fact that they spend their frikin' lives doing just that! But you can not place any other object on a floor that is covered by boulders. So you will only need to clear out small sections of the fortress, like living and industrial quarters. Stone will be very abundant anyway, so keeping stone hauling at minimum is best. When stone is needed for a task, a dwarf will pick first from the stones lying around on floors. If none is available, it will be taken from stockpiles. Or maybe the other way around? Doesn't really matter. No stone hauling! Geez!

A little more suspension of disbelief is needed when that small volume of resulting boulders is used to create a full volume artificial wall, but if you can't wrap you mind around that, just consider that the built walls are hollow.


You may have noticed a few tiles without boulders. These were sand walls. Sand is not randomly placed at world generation, even if it does not make any sense geologically. Here are a few cells with sand:

There is a small bug: sand floors are described as soil. I'll fix that by the time I introduce the new cell inspection panel you can see in the screenshots.