42 – IPS FTW!!!

I was supposed to do boulder stockpiles, but I finished IPS and I am too excited to not talk about it.

So, problem recap: create huge world, 1000x1000x60 squares and inspect RAM consumption. First it was too big, now with IPS it is just right:

Very nice for a final result: 380 MiB! It used to eat up almost 1.2 GiB RAM and right now I am even storing more properties into this tight space than I used to. It is not realistically possible to reduce RAM consumption any more while still keeping the capability of the engine to inspect any corner of the world with all of its properties. Actually, the compiler won't even allow me to create smaller objects on the current optimization settings. There are some very strict alignment rules and memory boundaries that the compiler will adhere to in order to optimize memory access. But a world of 1000x1000x60 actually consists of 60.000.000 floors and 60.000.000 cells, a total of 120.000.000 full featured objects that fit into 380 MiB RAM, without causing performance problems or memory fragmentation. I am more than happy with the results.

And normal worlds are not so big. I playtested the huge world, and it is not that fun, because every distance is huge and you spend most of your time watching dwarves dart about. Normal worlds used to be 300x300x40, and as such RAM consumption with IPS is very low:

A consumption of 42 MiB RAM is negligible now and so it was 10 years ago. So I decided to bump up normal word sizes to 300x300x100:

I will not increase default word size any longer for now, because 3D pathfinding data is not included yet in these 72 MiB used RAM. And I hope this will be the last Task Manager screenshot for a while.

Another side effect of IPS is that I need to test and often adjust every single result of tasks executed by dwarves. This is a huge bugfixing operation and I am almost done with all objects that have a visible presence on the map. But while doing that, I also created a very comprehensive and high level API for creating objects and getting their properties. The old API was low level and relied a little too much on hardcoding. If my game ever reaches the heights where people try to write extra tools for it, like reading its memory and interpreting the results, maybe I will release the API and a C++ library so those people can get direct support for interpreting DwarvesH maps. It would be the same library I am using (only slightly cleaned up), so the problem of third party tools falling behind with newer versions would not arise that often.

So now that the RAM problem is solved, I can get on with the rest of the work. Here is a screenshot of the new WIP inspection panel, that will show you all the meaningful data for cells that you click on and also will allow you to give context sensitive commands in the near future.

41 – RAMy Stony Sand

Oh my gosh! A real post! And the first one for this month. I am so excited! But I can hide it. Because I am over the Internet and you can't see me.

And I can't see you. Which can be a problem. While my blog is in no way popular, I am slowly getting a steady stream of comments. With YouTube messages, forum messages and direct e-mails, there is quite the communication going on. And Blogger is the weak link in this communication chain. It is very hard to communicate if I keep getting anonymous comments. So please, if it is not too much to ask, post while signed in. If you do not want to reveal you real sign in ID (for some strange reason; there is nothing I can do with it that would warrant such extreme security and privacy measures), make a fake one or something. It would certainly help not having several comments labeled as anonymous on the same post or on different posts.

So on with the show!

New RAM efficient storage system

As I said in one of the "Screens of the day" posts, I tried to increase map size to some ridiculous heights, like 1000x1000x60. The game is not designed to work with such huge maps because of the way task durations are balanced right now. But I found that with these dimensions, the game was eating up almost 1.2 GiB RAM. Completely unacceptable, considering both my future plans with it and that for 3D pathfinding it will eat up even more RAM. I set up to create a new system, which I have cleverly named IPS (Incremental Proxy System). Because it is clever. So am I. I only finished and tested the first iteration of IPS, which is not so efficient as the second one will be, but it is still an improvement:

That is roughly 610 MiB RAM. Well that is about half of the number that I found unacceptable. The second iteration is actually done, but unfortunately it is too buggy, so for now I'll only use this first version. The second should slice of at least another 200 RAM if everything goes according to plan. 

And here is the RAM consumption for a normal 300x300x40 map:

Negligible to say the least. It will use up even less RAM in the next version. After I finish this task, I will increase standard map size to 300x300x100 and add the fields required for 3D pathfinding and record the final RAM measurements.

Solution to stone problem

Another oldie here. I had a problem with stone digging. A problem of realism. When you set out to dig a cell made out of stone, you dig out a volume of 1.5 cubic meters. This stone needs to be usable afterwards for building stuff. But what to do with it. The obvious solution is to go with stockpiles. As walls are destroyed, the resulting boulders are hauled to stockpiles. This is not acceptable, since you would wind up with a system where digging takes a lot of time, most of it spent on hauling. The actual task of digging would be trivial when compared to the time and energy expediture of hauling. Not to mention that you would wind up moving the mountain from inside to outside or wherever your stockpile is.

I solved this problem at first in a rather poor way. Stone, after digging, would disappear, but you would still have access to it. So basically stone would move to a virtual storage, not findable anywhere in the world.

My new solution should be familiar to DF players, because it is basically the same thing. Not quite happy with it because it is a huge compromise, but I can't find a better system. When digging a wall, the dwarf hits the wall with his pick, until a boulder breaks off. The process continues until the cell is traversable, leaving the boulders on the floor. A little suspension of disbelief is required, because the 1.5 cubic meter volume is reduced to a smaller volume. So let's consider that you only deposit the usable boulders on the floor, with the unusable ones and other debris disappearing magically. Maybe in the future I'll have some refuse stockpiles filling up automatically as you continue to dig with ruble. And you will be able to pay trade caravans to relieve you of your surplus refuse. If I implement this this will happen automatically, without a dwarf having to walk from the freshly dug wall to the refuse stockpile. Here is the result of a huge dig operation on stone:

As you can see, digging though soil does not result in stone. The tiles are placeholder and too high for my tastes. I'll replace these with new ones that are also properly tinted. In this image, the floor and the resulting stone has different colors because it is not tinted, and it can be even more obvious when the stone has a darker color:

The resulting stone is fixed. Objects in the world are either fixed or not. Fixed objects are not automatically hauled, while the rest are automatically hauled if possible. Like logs resulting from cut trees: if there is stockpile, the logs will be moved there. Stone is fixed, so it will not be moved even if there is free place in a stone stockpile. You will need to manually move them. I will implement this for the next post, mimicking to log hauling system. Why not move them automatically? Because hauling is not fun. Dwarves do not suffer any movement penalty when walking on boulders. It has something to do with the fact that they spend their frikin' lives doing just that! But you can not place any other object on a floor that is covered by boulders. So you will only need to clear out small sections of the fortress, like living and industrial quarters. Stone will be very abundant anyway, so keeping stone hauling at minimum is best. When stone is needed for a task, a dwarf will pick first from the stones lying around on floors. If none is available, it will be taken from stockpiles. Or maybe the other way around? Doesn't really matter. No stone hauling! Geez!

A little more suspension of disbelief is needed when that small volume of resulting boulders is used to create a full volume artificial wall, but if you can't wrap you mind around that, just consider that the built walls are hollow.

Sand

You may have noticed a few tiles without boulders. These were sand walls. Sand is not randomly placed at world generation, even if it does not make any sense geologically. Here are a few cells with sand:

There is a small bug: sand floors are described as soil. I'll fix that by the time I introduce the new cell inspection panel you can see in the screenshots.

Screens of the day 09 - Stones, Take Two

I am working on two things right now: overhauling the item system in order to achieve less memory consumption and working on the dual interface system.

The way my interface works should obvious by now. Operations are grouped by category. You select the category and the the operation. Selecting a category gives you all the possible operations, with the inactive ones disabled. This way I can facilitate the process of memorization by having constant position for the different possible actions. You can use the mouse to select operations, but the final objective is to memorize shortcuts. I will always remember that 1,1 is wall digging for example.

You may have noticed the secondary interface taking shape: a new panel, that has sections for floor, cell content and will even show dwarf information or stockpile information depending on the content of the cell. Every section will display some buttons for operations. Last time you saw a screenshot with this I was totally not seeing the forest because of the trees and I actually added all buttons instead of only the relevant ones like in the old interface. This will change, and you will get context sensitive buttons with only the possible operations.

I am also reworking the item system, but I need to tackle every category of items separately. I am almost done with the wall system. The problem is that the new system is easy to create but difficult to change. So I am taking my sweet time designing all things on paper first and also making sure I include some space for properties I may need in the future. As a side effect, in the next post you should see sand floors and walls, not just soil and stone.

So I decided that an engraved stone will have a pattern and a quality. The quality is random right now, but it will depend on the skill of the dwarf who is making the engraving. Those skills are not only for show, you know! I also decided to limit the number of engraving patterns to 255 and five quality levels. Floors will not have quality, since the focus will be on sturdy work that can survive under the feet of busy dwarves rather than finesse, so all floor engravings will have normal quality independent on skill levels, while wall engravings, where the expectations are higher, can be bellow normal if the dwarf is not skilled. For the quality levels I picked color coding:

These colors are not random and should be familiar to a large percentage of the gaming crowd, ever since back in the day Diablo made the both popular and instantly recognizable. We all want purplez, don't we? The poor contrast the colors have with the background is only temporary. Actually, the whole new panel is still very rough around the ages.

I know that the last few post were not particularly exciting with huge features announced, but the small maintenance and detail work must also be done by someone.

Screens of the day 08 - Gallery

I managed to break the compiler. It worked fine in debug mode, but when compiling in optimal mode, it kept refusing to compile one of my files. It would just hang, eating 100% of the processor and using huge amounts of RAM. Sure, there was a huge function with a lot of template using code inside that was used to set up the word list, but still. So long story short, I saved the resulting data structures on disk and now I have about 2000 less lines of code. This way I will never have 20K! It is going to be pretty painful to produce a new word  list, but I hope this one will last me for quite a while.

I have gotten a second submission with engraving tiles. Thank you! So let me show you how this looks with the tiles randomly placed on floors:

Now this is the raw result of the submissions that I have gotten. The only thing I did is drop the two submissions into the content folder and load up the game. I know this is not what a lot of you were expecting when thinking about engravings. I will fine tune a little the images before the end of this post. As I said, I intentionally left all details as vague as possible to see what interesting results I could get. And interesting it is! See if you can spot my awesome turtle in this or a few more samples:

Just for fun, I went on to add engravings to grassy areas too:

I feel like this gives it a little bit more context. But let's see what we can do if we adjust the values a little based on the sizes and properties of the submissions:

A lot more detail is visible like this. Through trial and error I got even better results:

The final image was pretty clear and had recognizable patterns. Now it is time to see what can be done with the system with properly aligned patterns, or maybe even with a common theme. Here is an outline of the floor tile:

The content of the red outline is the visible surface of the floor. A pattern will be fully visible if it is that area, including the border. The yellow outline defines the sides of the floor. It makes no sense to have a pattern here. Of course, in the final version of the tile the outlines should not be visible. So in order to apply the awesome turtle pattern to the this outline, I would first center it in the red area:

Then I would remove the outline, getting the final version:

Also, keep in mind that while the background color is not important, it will be normalized, so I should declare one single taboo color. Working with isometric tiles is a little more difficult. I'll investigate if using some form of automatic conversion of top down tiles to isometric gives any interesting results.

I'll try to come up with some simple patterns during weekend and continue my investigation of RAM consumption. I have some good ideas that should bring down the 1000x1000x60 map memory consumption to 344 MiB. If it turns out to be as "simple" to implement as it seems it might be a worthwhile investment of my time. With this new system the traditional map of 300x300x40 should only take up 20 MiB.

Screens of the day 07 - RAM

Incredible! I have actually gotten a submission already! Thank you very much (you know who you are; I am not sure if you wish to break your anonymity, so I did not name you yet)! The experiment is already a huge success. I knew that random submissions would break my system and give me valuable input.

The biggest conclusion is that 1920 words are not enough and there is no way for me to match a preexisting word list with image names. In this first submission, most words were very common, yet there were plenty of them that I did not find in the database. An the submission also includes "neosesquecentiennialism.floor.png":

And a mighty fine visual representation of this concept it is! OK, I have absolutely no idea what this word means. Is it a real word? So the preset word list matching will be abandoned. Rather, I will create the word list on the fly containing only the words that were encountered in the image files. This change should take me a couple of days to implement and then I'll be able to post the first screenshot of the engravings.

There are further conclusions to be drawn from the sizes of the submitted images. I'll talk about this latter, once I decide how I can get the engraving tile both on floors and walls. Maybe applying a skew effect on the fly to create three different tiles?

So engravings are delayed a little while I adapt my system to the conclusion that I managed to draw from the submission. I still accept future submissions. Go wild!

So what else can I talk about? I finished implementing and cleaning up the final version of the structure that the "core" content pack will take. Nice and clean:

Here you can see the XML content descriptor and a few image files. There is still some work left on the matching of tile indexes to tile sheets, but the system is coming along nicely.

I said in my last post that the new floor system eats up a lot of RAM relatively speaking. And a few posts ago I said I would like to create bigger maps. So I thought I'd do this and see how much memory it would end up eating. With 300x300x40 maps, the RAM consumption is low enough and not a problem at all. But I increased it to 1000x1000x60 and this is the result:

Ouch! Almost 1.2 GiB RAM. Not cool, dude. Sure, 1000x1000x60 is way to big. I tried to play like this and scrolling the map from corner to corner is tedious and you can easily get lost. Zooming out to the maximum helps a lot in finding you way around the landscape, but playing like this can quickly turn into an extended game of zoom-a-weasel. On the other hand, the game runs exactly as smooth as it did with the smaller size, barring the map load time.

So what to do now? I do not know if going all out and implementing a very complex and space efficient map storage is worth the effort. I have an idea that would save data in two steps, that would greatly reduce memory consumption on such huge maps, but the reduction would only apply if the map is not explored. If you explore 100% of the map, memory consumption could get even bigger. Or maybe a less extreme implementation that is more of an optimization and reduces memory consumption to let's say 600 MiB, which is more reasonable. Or try to find a better solution that suffers from none of the weaknesses above? Or just go with what works fine for common map sizes?

What do you think?

Screens of the day 06 - Floor saga

As promised, while I keep working on the larger features that prevent me from posting regularly, I'll do a few Screens of the day posts.

At the end of the post I'll talk about the first opportunity for you to mod the game.

I am making progress on updating the floor system. It is way behind with its design when compared to cells and item management. The new system is shaping up, but it is a little bit heavy on the memory consumption side. Let's look at the following screenshot:

The first thing to notice is that there are a lot of places without grass. This is of course for testing purposes only, since natural habitats will generally have an abundance of grass unless near some kind of special geographical feature. The second thing to notice is the new panel on the upper right corner dedicated to floors only. Here you can see an image with your floor tile, its name and in this case it will also inform you that it is a soil type. Then we have some small buttons that will be used to only give floor specific commands.

Bellow we have to grass percentage, informing you how much of the surface of the tile is covered with grass. Pretty high in our case. Lets see another one:

This floor has almost no grass on it. I only have two tiles right now, one for full of grass and one for none. I would like to have a lot of different tiles corresponding to different percentages, especially since I only have a handful of soils. You can't have grass on sand or rock. In a future update, the grass will continuously grow until you reach 100% on all surface areas, quickly covering cells where the grass has been removed. Right now the world is generated with random grass percentages for testing purposes.

Here you can see what information is displayed when selecting a ramp You get the floor panel, informing you that it is covered by something, so probably you won't be able to interact with the floor until you remove the blocking item. Then we get the old panel, informing us that we clicked on a ramp. This panel will be updated to match the new design of the floor panel.

This time the floor is covered by a wall. And here we have the same scenario, but the wall is stone, not soil:

Now, that I have a word list, I would like to implement random engravings. I probably need to cut down the list, since right now I have 1920 words that can be used as a possible subject of the engraved floor. But how to cover the graphical representation of a large number of engravings? Here is where modding comes in. I have added a new tab to the editor:

The editor will autodetect any images that are usable as engravings, and populate the appropriate tile field for a word used for the engravings. Right now all field are empty.

And here is where a first opportunity to mod comes in. I have not released the service and editor yet, but they are not needed. My blog and project are probably not popular enough to even get a single submission, but I would like to do the experiment even as such.

Let me clarify one thing. I am not asking you to do my job, create content for the game that I will use or anything similar. I would like to see what shape the game world takes with random submission from people around the Internet.

It is very easy to do. First you need to decide the subject of the engraving. It is based on dictionary form of singular nouns, like "cat", the noun form of an adjective, live "bravery" or the -ing form of a verb, like "eating". Then you take this word, add ".floor.png" to it to create a file name under which you store a random PNG image, 32x20 pixels dimension, with a single random background color. So you will get "cat.floor.png", "bravery.floor.png" and "eating.floor.png". I will not give you a word list, since the 1900+ common word list should have you covered.

Let me show you how the tile for turtle looks like, more precisely the file called "turtle.floor.png":

This is the most beautiful engraving of a turtle that anyone has ever seen! As you can see, the future of DwarvesH (no progress on the new name yet) is in good hands from an aesthetics point of view.

So if by some miracle someone wants to submit a sample of an engraving icon that will be applied to floors, I will use it to populate the engraving pool and feature the content in a new post. All submissions are welcome. You can create ore or more, just ZIP them or something on my mail address. Any content will do (but please nothing illegal). While some may think "My, if I invest this tile with the very essence of the beauty of my soul and marinate it in its light for three months, I will get the create the most perfect piece of ART THIS UNIVERSE HAS EVER SEEN!!!!", things thrown together in 10 seconds in Microsoft Paint (like the lovely turtle from above) are perfectly valid too. Yo can even get the name wrong and I'll fix it manually (but only this time; I don't want you forming bad habits :)).

I intentionally did not go into too much detail to test the robustness of the tile sheet creator. It should work with anything that you could possibly send me, as long it is a 32x20 PNG picture with a background color. This is why I can't try this myself, because I know what the system expects.

If this experiment is a success, I would like to extend the engravings system to allow for complex multi tile patterns, like a 6x6 tile pattern for throne rooms and other decorative patterns. But enough wishful thinking for now.

Status update

Yes, I know that I haven't posted in quite a while. It is my fault actually. I have been posting too frequently, creating this false expectation that I will continue to do so. That I can continue like this... forever. But I can't! But no more! The lies stop now...

I am only kidding. Of course updates will continue as usual, but I can only post when I have some content. Sorry for the lack of updates. I'll try and make up for it by posting a not too detailed overview of what I have been up to. It is not that I have been not working, it is more about spreading myself to thin.

First, I have been working on one new feature for the engine which I have been foreshadowing for a while. It is a fairly minor and secondary feature, but I think it will contribute great value and also add a new perspective for when it is needed. I am 95% done, but the last 5% are very tedious. And there is also the part regarding rams. My god, I do hate working on those ramps. I am actually considering trowing out ramps and replacing them with half height cubes, but I do not want to get into Minecraft territory.

Another thing I have been toying around is procedurally generated content. This subject tends to rub both geeks and DF fans the right way (are there non geek DF fans out there? *troll*.*troll*) and I my qualify as both, I find it also very interesting. So I decided to go this route. As a symbolic gesture, further cementing my commitment to procedurally generated content, I will be throwing out all my hard work on the hand crafted plant list (only plants, trees and all other data remains) and replacing them with procedurally generated content. Now this is a very risky decision, replacing hand crafted content with the interpretation of the output of a random number generator, so for the first version I'll compromise. I'll use random plants, but I'll generate more than needed, hand pick from this list and also adjust balance if needed. So the first version will have a static plant list, but this list a subset of a randomly generated list.

So how am I generating this content? First, check out Sphalerite's Procedural content creation: Random plant creation script forum thread. Here is a direct link tot the script. This little script generates random plants and trees in a very clever way: it randomly generates a name and then deduces the properties, so when generating the same name twice, it will have the same properties. Of course I am not using the script to generate my own plants, because that script is for DF. But mainly because it is Python. This is a snake free zone right here. Please check in your snake at the customs desk. I am doing my own implementation of the algorithm. Sure, the basic idea is the same, but the implementation is fresh. I do not know if I will be able to make my algorithm generate DF content too, since the required data is so different in content (shape it is not important, I could eaily output RAW files instead of XML). 

This little piece of code will be released to the public when it is ready as open source.

Just one more little side note regarding content. Both the Python script and my implementation need a word list. The script loades DF data. I have all data included in the code so no DF versions needs to be installed. The form the data takes is very different. In DF, we have RAW files with entries like this:

[WORD:ACE]

[NOUN:ace:aces]

[FRONT_COMPOUND_NOUN_SING]

[REAR_COMPOUND_NOUN_SING]

[THE_NOUN_SING]

[REAR_COMPOUND_NOUN_PLUR]

[OF_NOUN_PLUR]

[ADJ:ace]

[ADJ_DIST:1]

[FRONT_COMPOUND_ADJ]

[THE_COMPOUND_ADJ]

This describes the word "ACE", telling me it is a noun with singular and plural form, but also an adjective and also gives a few properties for the word. I describe this word by the following C++ code snippet:

sGlobalWords.Add("ACE", Word(665767).Noun("ace", "aces").Adj("ace", 1));

It is obvious that the form the data adopts it different, but the content is the same. There is no way around this. This is not random data, this is the English language. The word "cat" will always be a noun, the plural will always be "cats" and it will always have the same word composition properties. As long as a word appear in my list and also in DF, it will represent the same logical data under a different physical representation. Just wanted to be clear about this from the beginning.

The random plant generator is about 40% done and it is not integrated into the editor yet.

Another thing I am working on is a system to give large scale generic orders, like "we need more food", "we need a new bedroom" or "we need an artificial lake". Dwarves will the proceed to break this order down into smaller ones, and without user intervention figure out the best way to execute this. This will not be an option for the player. It will be a behind the curtains system. Beside the obvious uses for such a system, I would also like to release a semi-interactive screensaver version of DwarvesH. Sure, seeing the Windows™ logo slide across the screen never gets old, but wouldn't you rather see a fortress evolve from scratch to a massive complex in a non interactive way while your CRT screen is protected from burn in?

And speaking of DwarvesH, the time has come to pick a new official name. I said from the first day that this name is placeholder. Unfortunately, I am very undecided and it will take me probably weeks (if not decades) before I can pick a name I like.

I have also been slowly investigating the legal ways one could get out some money out of an endeavor similar to the one yours truly is undertaking. Domain names, hosting, security certificates, design and other services all cost money on a monthly or yearly basis. While I can easily bear the burden of these costs for shorted periods of time, on the long run it does not seem as a good idea. I would like to offset this virtually endless stream of small expenses with a means of covering them. Without a method of sustainability, the only thing that I will achieve in the long run is bankruptcy.

This has a few interesting implications. Let me tackle the one that springs imediatelly to mind first: no, this not necessarily mean that you will need to spend money to get your hands on The Game Formerly Known as DwarvesH. I said that I am investigating financial stability as opposed to guaranteed deficit. There are multiple ways to achieve this, from optional donations to advertising to premium services to outright selling game licenses, and a lot of options in between.

Another implication is that it can only happen only after all third party assets have been eliminated, like Stonesense tiles and a few left over recognizable terms from DF. I am sure you will appreciate a fresh DwarvesH, bathing in the refreshing stream of original content.

Until I have enough features for a new post, I'll try and fill the place with a few "Screens of the day" posts and a pseudo-challenge.