29 – Captain! We need to bridge the wooooüoorlds!

I would like to thank all the people who gave suggestions in the comments section. I have looked over your suggestions and also investigated additional techniques for optimization and I am fairly sure that I can make the 3D engine work fine on strong computers. I am not sure yet if I can do it on middle or low-end computers. So the 3D engine stays. If I can’t manage to get it to run on a wide range of computers, you will still have it as a 3D integrated “visualizer”, where performance is not that important. The problem is that most of these techniques are quite complex and will take a while for me to implement the changes, especially when I am at a moment where I would rather like to add new features. I need something that works right now.

So I brought back the isometric engine. And it immediately felt familiar and like a good fit. I felt like a dwarf after regrowing his long lost beard in a freak gasoline fight accident. I just need to add a few new features to the isometric engine to solve some of its problems. But it is basically done and I will never remove it again. All new features can easily be done by simply adding tiles, so there is no good reason to not support this display mode for the entire lifetime of this project.

On with the changes!

Isometric and 3D coexist

I have fine tuned both engines and they basically coexist peacefully in the same code base. The problem is that the caching system was not designed to handle both 2D and 3D objects, so I need to decide which rendering mode will be used before map load. But once I extend the cacher, you will be able to flip between isometric and 3D at will at the press of a button without any loading time, just something under a second while the cache is rebuilt.

Isometric engine support borders and short walls

Let us look at a picture that illustrates the problems with isometric engine. There is not enough contrast with some kinds of soil and stones and walls can cover up things behind them:

First thing I did is add some borders. Stonesense has this, but I added more borders. I still need to fine-tune which borders are drawn. I wanted to add these borders for a long time, and finally they are here:

The display of borders can be turned on or off at the press of a key. Another new feature is short walls. You press a button and walls are not drawn at full height, but instead are drawn very short. I still need to decide what to do with ramps:

Here the contrast is very poor. But, if I enable borders:

28 - Failing in three dimensions

So for the first time I am going to show the 3D engine. Grass isn't textured yet, there is no lighting and shadows and I don't have enough models and the ones I do have have been hastily created and not that pretty:

The above view was somewhat of a default, where you have a good overview because you are supposed to interact with the map using a mouse or keyboard. But the advantage of a 3D engine is that you can change your view point. You can zoom in and tilt the camera:

Or you can zoom out a lot and even see the cliff side in the distance. Rock are not textured, but color coded for my convenience:

Here is another example, when I scroll back:

And, of course, you can scroll and tilt in such a way that you can see the entire map up to the horizon. The number of objects is staggering (and there is no sky box yet):

Here is what happens when you cut out a complex shape from the wall. When something gets covered up, you can always scroll a little to free up the view, but thing around the middle tend not to get covered up:

So if you are seeing this pictures and think that they are ugly, you are probably right. I think that it is both ugly and fixable so it won't be ugly forever.

The good news is that I am very happy with the way the engine works technically.

The bad news is that it is extremely slow. There were a few performance problems from the beginning, but the real problems started when I replaced the placeholder cubes with real models. As can be seen in pictures, there can be thousands of object on screen at once. And while in those screenshots I have around 60 FPS, there are two things to take into consideration:

• those screenshots were taken after the static scene has lingered on for a while, after all the tricks and hacks to make it faster have kicked in. When actually playing and interacting with the map, I have more like 20-30 FPS.
• I have a very good graphics card. I can imagine it becoming unplayable on a “normal” graphics card.

The main culprit is the number of objects. There are limits on the number of objects you can have and those limits are quite harsh. Ever wondered why maps in games tend to be a small number of large and relatively uniform meshes put in a matrix? There are a few things that I can still try. I'll try using a mesh merger, converting let's say 1000 trees or bushes into 10 meta-trees and bushes. But if this does not give any good results, I am afraid I can't continue the 3D engine. If it fails I'll re-enable the isometric one and focus on that a bit until I read more about 3D engines.

Another concern is the complexity of objects. Since everything is so small, it ts very hard to create something that looks good. You have to have a number of polygons as small as possible and textures made specially to highlight the detail in small objects, with broad features. It is a lot simpler to make something like this in 2D, where you can add borders.

I really don't know which way to go right now. 3D still offers advantages, but I am starting to get the feeling that 3D is not that well suited for this kind of game, where you need to be zoomed out and show a huge number of small items that still need to be instantly recognizable at a glance form a tactical view. 2D is well suited for these goals, but you need to add a few techniques to combat obstruction caused by high items and you can not change around the camera.

The good part is that I still have time to experiment around. There is no pressure from a studio or publisher or anything, so I can make a few mistakes or have a few failed attempts before I create something successful.

27 – Story and trees? Or a story about trees?

I have encountered two little obstacles in the development that I have to deal with. While describing those, changes and new features, this post is going to be more old-school free form, not so structured like the last ones.

First obstacle is that I am severely lacking graphical assets. Even where I do have meshes, they are not textured and generally I do not have adequate meshes yet. The sheer number of the meshes makes it impossible to finish the job in a useful timeline if I don’t contract some for them. It is surprisingly difficult to find a good freelance artist but I’ll continue my search.

The second obstacle was created by people discovering my project and overreacting in typical Internet fashion before getting informed about the subject. There are plenty of patient and rational individuals out there, but a large number of people took one look at the videos and they decided that DwarvesH is just a DF clone or that I stole the code. It is hard to explain the phases of software development and why DF data is the perfect test case for my engine, but now the situation is under control and there are polite discussions going on.

So since people have no patience for my pragmatic plan that involved coding the engine technically first and then starting to create the content, I’ll change the way I am doing things around here. While I could go on the way I did, even if by the time my project got finished I had a unique take on this genre, with original content and gameplay mechanics, it would be of no use if I have alienated the entire online community. Negative views spread faster than positive ones and they can be quite contagious.

From now on I’ll pick from all the features that I have planned the original and “flashy” ones first. Bear with me, this is not as superficial as it may sound to some of you when first reading. I would argue that it is not superficial at all, since I am just reordering tasks so I implement elements that I know that are original gameplay wise first, while at the same time giving me time to write new content, which is a time consuming task. I will also pick from these features the ones that are easier to code or built upon starting from the current engine, so you don’t have to wait while I create a new foundation before I can post something about new features.

I have a lot of planes that I did not talk about before since I knew that I would not be implementing them yet and I didn’t want to do a bait and switch.

One of the big differences that require a lot of work but barely any coding is that I am going to have a story. Or more like a setting and a framing device. Generally such city simulations allow you to play from the perspective and with the invisible hand of an all-knowing god. You see the full map and are able to take decisions and influence everything. This will stay the same, but it is implied that these changes happen and these tasks are designated because your expedition leader says so. You will have a “player character” and you will be role-playing this expedition leader. There is quite a bit of back-story. The mighty nation of dwarves has been at war with a named but unseen (by the player) foe for decades. There were periods of peace following major victories, the last one being almost a decade ago, when the dwarven nation scored a crushing victory and your character was declared a war hero. Since then, major conflict has broken out again and this time the dwarves are having a hard time facing their enemies and resources have become scarce. Your character is no longer fighting. In the period of semi-peace between the two major war efforts, there was a major “falling-out” between your character and the nobility of the dwarves, getting striped of rank and possession and jailed for life. Only the fresh memory of the heroics done in the past and fear of former allies you had in your days of glory have kept you from getting executed.

But recently, a very prosperous but dangerous area has been discovered, that not only seems to be rich in adamantium, but the much coveted Tears of Tara’Zul. Two expeditions have been butchered there and there are no resources to spare for further failed attempts. This is where you chance comes in. You are offered a royal pardon from your imprisonment and offered freedom and full administrative privileges over the new region. But basically you are exiled there and are expected to defeat what foes have butchered the last couple of expeditions  there and create both a powerful fortress and a gateway into these new rich lands. And it is the secret wish of the nobility that you will die trying!

OK, not award winning writing here and this is a fairly cliché story, but here is the thing: you don’t need to. This is a sandbox game after all and too much narrative structure would get in the way of that. And it leaves just enough holes so you can role play. Your character is not given a name or skill set. Why was he a war hero? A feared warrior? A brilliant tactician? Master of creating new settlements in hostile territory? The only given is that you are supposed to be skill wise a good choice to head this new settlement, especially since everyone else who could fill your shoes is off to war. What was the cause of falling-out? Did your character try to defend someone against the wishes of the nobility? Did he get fed up with the horrors of war and desert? Did he turn against his former mentors? Did she take advantage of the chaos of war and do some terrible and cruel deeds while leading the army to victory? It is up to you.

So your character is a dwarf on mission. A personal mission of regaining lost honor or revenge against his foes. And on a royal mission. Your primary mission is to survive together with your expedition. And build a successful fortress in a tactical position. And supply the dwarven nation with resources. There is not going to be an envoy that periodically comes to trade with you peacefully. No, this a royal task force that will inspect the progress you are making and make sure you are not planning a revolution,  and while they will trade with you, they expect you to supply them and do your part in the war effort with resources and finished goods. This requirement won’t be too harsh, allowing you to expand the fortress whichever way you please as long as you can send a few uncommon goods back.

Your character will also get more skill points and be able to raise starting skill higher. This is already implemented. You can also customize his or hers inventory. This would be easy to do using the means to customize your starting expedition in a few lines of code. The only reason why I did not do it yet is that I have no items that would be worthwhile to give him. I want you to give him clothes, armor and weapons, maybe customize his look, and not give him a sack of mushrooms. On the other hand, maybe a sack of special mushrooms would make planning the fortress a lot easier.

This also means that your character will be more important to your success than the other dwarves. You will have to weigh the risk of using his advanced skills vs. exposing him to danger and losing him. If he dies, a new dwarf will be sent to the fortress and take over as expedition leader.  You, as a player, will still remain in control though. This new dwarf will have random skills and offer no perks or additional bonuses as your character did.

There is lot to still talk about this subject, but this post is getting long so let me continue with other changes. I have removed the tree regrowing code. No, when first embarking, you will still have trees, but if you cut them down they won’t grow back. Actually, I am making trees take longer to regrow as now, but for now I’ll just remove the code. When I have enough gameplay to last you years of in game time, the time it takes for trees to grow back, I will re-enable this feature. On the other hand, trees now have fruit. Not all of them, but most. Fruit can only be eaten for now. Since I did not have time to come up with clever names, all fruit is named after the tree it is from followed by fruit.

What will you do with this excess food? Well I am making food requirements a lot harsher and realistic. Dwarves will need 3 meals a day. And cutting down an elephant won’t be a frikin’ jackpot of meat that lasts you 10 years. The yield that you get from trees may not be that much, and you may only get one from a tree species every year, but it is there and is reliable. It will help you keep your dwarves well fed, while making the decision to cut down a tree or not more interesting. You may gain wood and fuel for now, but you lose a long term food source. Extra tabs for fruit have been added to all adequate windows.

26 – Bye Bye Beautiful

I am talking of course about the isometric engine. It has served me very well. It was fast and slick. But now it is gone. The 3D engine is ready to be included in production code, so in order to symbolically confirm my commitment to 3D, I deleted all isometric code. Don’t worry, I still have backups with it. I doubt I’ll ever need it, but it is there.

But before I continue, let me tell you a short story. It began with me working late into the night and applying the finishing touches to the 3D engine. In the prototype state, it was a total mess that maybe had the potential to one day be called a proper engine. By Monday, when I last posted, it was a chaotic blob of hacks that worked surprisingly well. And by 2-3 AM, it was pretty much production ready. Code was in very good shape and technically there was very little to object (I don’t have yet enough meshes and did not implement lighting, but that is a different story).

In the morning I selected all folders and wanted to create a backup archive. I noticed that the backups are getting quite numerous, so I selected a few and hit delete. Without deselecting my code! I noticed that it was deleting to many files and realized my mistake, but I decided not to hit cancel and revert the delete after it finished, since most of the files were deleted already. I would like to stop here and congratulate Window 7 Recycle Bin for being a total piece of crap. In the past it was a useful tool. Now it greats you with a list of all your files, and after restoring, it screwed up and restored different versions of my files. And most of them old. Oh well, shit happens. I managed to rewrite 99% of the changes and merged with a different backup that had all the new features that are not related to the 3D engine.

I really need to start using a repository. But a local repository wouldn’t have helped, since it would have gotten deleted. I don’t want to set up an online repository server on my machines. Most free hosting is for open-source, and anyway I don’t trust such services not to steal my code. Sigh…

Let us get back to the change log.

New 3D engine now in production code

And let me tell you, I am really starting to like it. There are still a few issues, like tilt and zoom are very restrictive because I can’t figure out how to make them more flexible without killing performance. Rotate is not implemented yet. But still, combining zoom, tilt and old fashioned scrolling makes for a pleasant interaction with the map and obstructed items can be easily brought into view. And speaking about performance, the engine is a little bit slow. Of course, when I am claiming that something is slow and don’t give explicit numbers, you should always take that with a grain of salt. I am somewhat of an elitist and perfectionist. The engine runs at around 150 minimal FPS and it can get up  to over 600 when rendering a barren landscape. But it is not enough!

The new tweaks to the engine have made it more “center” centric. I don’t want to get into technical details. Suffice to say the math is simpler now and I am happy. There is a problem though. Since the camera is now centered on the viewport it did not look that good with the interface taking up half of the screen. Anyway, such huge panels are a thing of the past. I’ll talk about this later.

I think I managed to recode everything that was lost in the little delete incident.

Interface takes up less space

Because of the above mentioned centering, I’ve made the interface smaller. Now it is harder to use at first, but no worries. In the future I’ll add proper graphics for these buttons and when mouseing over them a little panel will open up telling you what the button does. In the next screenshot you can see the interface in its new minimalistic look. I won’t show you the new engine though. Even if it is technically very sound, I haven’t gotten around to making it less ugly. So enjoy a brown screen and the interface (better than a blue screen, am I right?):

Moving the cursor with keyboard

This feature was requested on YouTube. Some people are more comfortable with a keyboard than a mouse. DwarvesH allows you to do everything with a keyboard except move the cursor and do selections. I implemented the moving of the cursor by keyboard, but this change was lost in the incident. I’ll silently patch this back in. It is only a few lines of code.

Stress testing items

I have begun both adding items and really stress testing the system. I added 2000-3000 items and by the end of the month I’ll try to get as close as possible to 20000 unique items. Creating so many items out of scratch is time consuming, so I’ll use again Dwarf Fortress data as a test bed. There are three parts to this. First, the items must be created and represented in the game. Here is a terribly outdated tree made yesterday with the item available:

Items in red are the general root categories, entities in their raw form as they appear in nature, like plants, animals, minerals… Grey items are forms in which these entities may appear which you do not own, but interact with them. So you may have a cow and 7 axes, but you do not have 10 walls and 5 floors. Items I bold are final products, in blue are the process though which they were obtained and normal text items are categories.

The second part of this process is the actual stress test. Adding these items in large numbers and seeing if everything works fine and consumes few resources.

The third part is creating and interface for managing these stockpiles. This interface must be able to handle huge quantities of items but no feel barren or obtuse when you only have a few. Here is the first iteration of the item manager. It is not that smart yet, but it will get better:

pre-alpha-1 – 2 – I can has update?

Yes!

Back from a break and I sure have updates.

But first let us talk a little bit about the evolution of DwarvesH and the idea behind it. Long story short, I wanted to see if I can take the formula of such an involved city builder and make more accessible without taking away the depth. I have been getting quite a few negative comments on YouTube. People seem to be split up relatively evenly between people who like the new interface and graphical perspective and people who are calling this a Dwarf Fortress clone. I don’t want to debate that right now, but DF’s take on the city builder is severely underrepresented. Maybe we need more there.

But I am getting side tracked. In order to make everything more accessible, I went from top down 2D engine to isometric engine. But isometric has a few disadvantages.

-          Walls and other high tiles tend to obstruct your view. There are numerous workarounds, like making tiles transparent if the cover something, making walls half-height, drawing extra borders,  allowing 90 degrees rotate on map and others. But the end result is the same: you will have some level of obstruction that is unavoidable.

-          Isometric graphics are hard to produce. Because of that angles you have in such an engine, getting smooth transitions between tiles is harder. Not a problem for an experienced graphical artist (which I am not).

-          Modern graphic cards are not that good at old-fashioned 2D bliting. So you need to be careful with the number of draw operations that you make. You can’t just say “Hey, this card pushes out 100000 polygons, I’m sure it can handle 1000 blits”. The numbers are arbitrary, but the truths is that you need a lot of precomposed tiles, by either composing them on the fly and caching the results, or just have them precomposed on disk and load them as such.

-          And others that I forget.

I’m sure you know where this is leading, especially since I talked about my prototype 3D engine. So after finishing a few important tweaks, I started working on the 3D engine and preparing it for full time service.

At first I had a lot of difficulty, but then remembering my future plans, I greatly simplified the math and made the camera less flexible. You can tilt, rotate and zoom, but there are constrains on these operations. I am aiming for a Warcraft III type engine, with a good default camera angle for overview and some rotation capabilities, but no absolutely freeform camera where you can zoom in and see one of your dwarves’ toes from underneath.

This is not the best time to introduce a new engine. You know, the alpha consolidation and polish phase. But I can’t really delay it, since further efforts in isometric engine are going to be wasted effort. I do not have the resources to keep both engines running.

This also raises some accessibility concerns. Event thought the engine offers better view point, with almost top-down default perspective, but 3D aspect ratio and camera management, it does mean that I am cutting away people without 3D capabilities. I’ll do my best to make the engine as scalable as possible. And I feel bad for retiring the isometric engine, which was extremely polished. Oh well…

So let us review the changes.

New 3D engine

Read the rant above to find out what this is about. The new engine is in no way optimized, but as the weeks pass and I learn more, it should even be faster that the old one. The problem is that I am lacking the assets right now. All the 2D tiles are represented as cubes with the tile applied as texture. I am working on inserting real meshes here, but it is going to take a period of time. Also, a lot of things are not textured yet; they just have a bland color. These two elements combined make the new engine look uglier than the old one, and this is the reason why I won’t post a screenshot yet. Maybe next time!

Optimized A*

It is very hard to benchmark the A* algorithm, because it depend a lot on the start and end position, map structure and obstacle distribution. But while keeping these parameters as constant as possible on a random map, I managed to get a good test bed for various implementations. The new A* is between 5 and 9.75 faster than the old one, depending on scenario.

Other stability tweaks

There are tons of small fixes that just go under the good-old bug-fixing umbrella-ella-ella-ella-ella.

And with that, I’ll declare pre-alpha-1 done. The period got cut short a little, but I am wary of making very specific niche case optimizations now with the new engine in fear of them becoming obsolete. I’ll continue according to plan, adding the wood and stone products with their respective industries. I’ll also patch up the new engine as time goes by, but not as a concentrated effort. As soon as I have enough assets for one of the biomes I’ll start posting visual again.

There are a lot of questions that 3D raises that have no correct answer. What should be the default view angle? How far is the camera placed? What FOV should it have? How big should the landscape chunks be? What format to export the 3D graphics? I’m hoping these answers will come to me slowly.

pre-alpha-1 – 1 – Yogi Bear

Since I am in a feature freeze until I finish this pre-alpha, I am not going to be able to post new and exciting stuff about bright-and-shiny-bing-bong hot new features. So this obviously means that the Internet will declare this project dead. We don’t want that, do we? So I’ll continue posting I my usual style even if I don’t have any new features, only refinements. It will be clearly visible from the title that I’m talking about the pre-alpha.

Optimized rendering engine

I have added a new optimization to the rendering engine. It is very weird and has had some even weirder consequences on mouse interaction with the world (of all things), so I’m not sure if it good or not. Only time will tell. But, it did raise my FPS at maximal strain from 72 to 107-109. That is 35 extra FPS on a stationary screen, so it might be just worth it only for this bonus. As always, when I’m talking about FPS and don’t give any additional details, I always read the FPS on the same machine, with software rendering and an 800x600 window. On my gamming rig I get over 400 FPS with hardware acceleration, but this is not that representative, since my holly goal is to make DwarvesH run on anything with a processor that is still used today by non-dinosaurs. I may not be at my goal yet, but I’m damn close. I hope. All the dwarves have their fingers crossed.

Height map added

A persistent height map has been long since overdue. And now it is here. It has made world generation faster. It will make grass and snow easier to implement and fix in a satisfying manner. And other benefits.

Making the world a little less dull

Partially due to the height map, I can now do some small tweaks in world generation so that the world is not that uniform. I’ll add small patches of sand in the future, but for now we have small de-elevations in floors. It will look a lot better once we have grass in these holes. Also notice that it is possible for plants to grow there and that the mouse cursor selection box is displayed deeper, taking into consideration your elevation:

A lot less hardcoding

While most objects are handled by the editor, there will always be a few entities that have place nowhere but are essential. The tiles for such entities have been hardcoded for ages. This was a huge nuisance every time the tile set changed. Now I can handle these items from the editor. It will be easy to tell if they need adjustments.  This is the only tab from the editor in which you are not supposed to make modifications randomly or risk breaking your game because and everything has fixed meaning.

Missing skills

I removed all skills that are not used, but I would have liked to keep the combat skills in. They have been streamlined. Now we have “Melee” and “Ranged” governing their respective combat style. “Dodge” is useful for everyone. “Armor Use” allows both the better mitigation of damage when wearing armor and the reduction of the penalties resulting from the use of said armor. “Shield Use” is used both for damage mitigation and chance to block. These are the only skills you have for fighting, but I might add back “Wrestling” and others in the future. All weapon specific skills are no longer skills, but proficiencies, which stack with your skills. An expert melee fighter will be able to use all weapons effectively, but will have a bonus if he is fighting with a weapon with which he is proficient. You will not be able to put points into proficiencies, but they grow on use. Dwarves can acquire proficiencies with multiple weapon types, but you do not need to see them all as with skills. In order to activate the adequate proficiency, the dwarf only need to pick up a weapon.

Video Tour and November-January in review

Yes, I know that we are still in January. But in December I did not get to do too much, so it counts as half a month and this is an appropriate time for a review.

But first: check out my new video!

It is a small presentation and tutorial video, trying to recreate some possible steps one would take in a new fortress, like cutting out a few storage rooms, gathering some plants and logs, building a grand dinning/meeting room and getting started on personal lodgings for your dwarves. The video is annotated with explanations.

I am not 100% happy with it, but this is my best take in over 10. It is not as easy to make a good video as it seems. Coding is a lot easier. gEEEEEEEEEEEEEEEEEEEEEEk.

So, back to the review. November and January went great. I managed to get in most of the features I planned in October. I got sidetracked with stones and layering, but it is for the better.

So what is the status of the project? Pre-alpha-one! I’m calling it!

So for the immediate future I am not going to add any new features, but bring the final polish to the existing ones and once I’m done I’ll archive and call it pre-alpha-one. I’ll post a series of short-posts about what feature got the final polish.

Then I’ll start work on pre-alpha-two. This version is going to be all about wood and stone working, bringing you the associated workshops and all the products that result from these materials. Dwarves will have selectable appearances so they don’t all look the same. And as some of you know, I have an experimental 3D engine that I’m playing around with, but I don’t think we’ll see it in pre-alpha-two.