Saturday, July 27, 2013

Tworenas Snapshot 003

So, here it is, again a couple of days too early. Next week I'll be going to the seaside to get my tan on, eat a lot of junk food, pick of women and have in depth conversations with them about shaders. So I won't manage to put out a very substantial snapshot. Instead it is going to be more about optimizing what I have.

And it sure needs optimizing! The new grass system is here and it works surprisingly well, but it is far from done. I just can't create a top notch grass system with extreme streaming capabilities and good view distance in a week. When I'm talking not optimized, I'm not kidding: grass is not frustum culled and it uses no hardware vertex buffers. Also memory leaks, snags from the GC shitting itself over the amount of data and other bugs are uncommon, but possible.

More than half of the development time was me fighting C#, the .Net runtime and the .Net GC. C# might be good for general use programming, but for streaming in tons of data fast and reliable you need to bring out every dirty hack in the book, and when that fails, invent some new hacks.

I uploaded a video showing the new grass.

So, a few things to keep in mind:
  • As said, grass is unoptimized and buggy. Don't expect stellar performance.
  • World generation takes a while. At maximum map size it will take you 30-45 minutes. I will optimize this someday, but for now avoid such worlds when possible or run the generator while doing something else. Also, world sizes on disk can be up to 4 GiB each at maximum size. If you run out of free disk space: crash! Loading such worlds is fast though, so once you have one, you can keep using it.
  • Since I did that video, I also added a few more grass distance options. It is not recommended going over 6 grass distance for now. It will kill your framerate.
  • No detailed changelog this time. Here's the short version: grass.
  • The new rendering profile is on by default to give it some more serious testing. You can switch back to the old one by pressing "l'.


Link: Snapshot 03

Thursday, July 25, 2013

104 – Obligatory weed joke

So for Snapshot 3 I'm trying to get rid of the old, buggy and perfectly unscalable grass implementation. Using some simple polygons with grass textures was one of the first thing I ever did in XNA and unfortunately the current grass system is not a lot more advanced than just that. I need a system that can handle potentially infinitely large maps.

But today I'm not going to focus on the core of grass implementations, which is mostly just screwing around with buffers. I'm going to focus more on the Construction Set and pipeline side of things, making sure they are up to par.

So first, let's open up the spartan model manager and add the grass model:

Well, that worked. There is a new model in the system and... it won't do us any good. Let's update both the manager and the pipeline!

One of the main problems that I'm having with my engine is the reliance on the XNA content pipeline. I phased it out before XNA died, and that sure turned out to be the right choice. But even though I'm not relying on the content pipeline, somebody needs to import the model at least once. And that somebody is the content pipeline. I am using a specially compiled version of the engine, which loads all models thought the content pipeline, converts them and after you don't need the original models or the content pipeline anymore. Not this is a major engine pipeline problem. You want to add a new model to the system? You need to open up the engine project in Visual Studio, add the model to a project in a special place, compile the special version of the engine, run it once to get the conversion going, then start using the normal version of the code. For now there is no avoiding adding the model to the project, but the rest of the phases must be removed. 

Let's try to fix this...

So after some coding, I delete the new model and use the manager to add it again:

After adding the model, I can now input the path. Pressing the small "R" button will try to look for the content pipeline model and do the conversion for you and then load the model. If it can't find it, it will just load the already converted model from disk. Much cleaner, and you don't have to worry about special versions of the engine or what not.

Next we use the material manager to add a new texture for the grass:

Here there are no pipeline issues. All I need to do is drop the textures into the game install folder, and the editor can load them.

There is one final problem. How does a model know which material to use. In the past this was hardcoded, but I update the module code to handle this, including loading and saving of these relations. No more hardcoding, but the model editor has no GUI for these tasks yet, so you need to do it the XML file. I edit the XML to include the relations for each model, from the barrel to the skybox, making the module completely self-contained.

Finally, I add two lines of code to the engine which are literally if the key '9' has been pressed, add the grass model:

Awesome! We now have a grass object created in the world! It is fully physics enabled though. My engine has no support for non-physics enabled objects, so I hack together support for this and I also write a shader that handles transparency:

Rendering transparent objects is not that easy, especially if you want them to look antialiased. XNA unfortunately has no support for alpha to coverage, so we need something else. There is a method to improve rendering quality based on dual pass and alpha blending, but the problem is that this method needs individual parameters for each texture. You need to experimentally fine tune these parameters for each texture and also the artist developing these textures should take care to create textures which play along nicely with this method. In order to make debugging this easier, I create keyboard shortcuts that allow you to modify parameters on the fly to see the result, together with a debug mode that highlights the edges of the grass that will be smoothed out so you can figure out the optimal parameters in real time, one texture at a time:

In the above screenshot, the white areas of the grass will be the ones affected by the improved grass rendering algorithm, while the rest won't be.

Next we update the shader to include some subtle and simple grass animation. I can't show animations with pictures, but I can show you the funny effect of applying these shaders to normal geometry:

Finally, we write a very simple and ugly algorithm to populate the world with grass. Here are 3 shots, one during nighttime,showing a landscape with grass:

This looks very ugly. A considerably smarter placement algorithm will be needed that populates grass without such obvious repeating patterns. 

The next step is going to be to create s system that merges all the grass bushes into large vertex buffers because in the above screenshots I am rending 47000 grass bushes individually without any frustum culling or optimization and the framerate is non interactive. After I need to write a streaming system that pulls in only as much grass as needed and update the world generator to write grass to the saved map.

  • Module system updated to handle more properties for models.
  • Model manager slightly improved.
  • Improved model import pipeline.
  • Experimental: the options dialog can change MSAA without triggering a device reset.
  • The options dialog is in live sync with all other methods of changing device settings.
  • Changing some setting in the option dialog no longer centers the mouse and the game window is not longer shifted a couple of pixels in position.
  • The engine reports framerates when in system hub mode properly.

  • B006: Changing terrain quality in the options dialog when no terrain is loaded no longer crashes the engine. Fixed.

  • B007: Playing around with window frame and fullscreen can easily crash the engine.

Sunday, July 21, 2013

Tworenas Snapshot 002

Yup, it's out one day sooner. I was late a few hours with the Snapshot 001, so this makes us even. No, actually the main reason to push it out sooner is so that I can get in a few hours of work today on Snapshot 003. Snapshot 3 will be slightly less ambitious. I am so tired of avoiding the buggy grass code so I'm thinking that Snapshot 3 should bring back grass, together with user controlled placement, like everything else in the engine. And I'll probably have time for a large scale smoothing tool, so we can get that "Foundation" button enabled. It works even now, but without smart terrain smoothing, I avoided showing it of.

So here is a video with some of the new features:

With fewer announcements to make,  I can give you a few controls. I won't talk about most. Half the keyboard does something. If you manage to load up a map, there are a few important controls. The first thing you should do is press "C" and put as many points as you can into the skill tree. This should take care of the painfully slow starting walk speed. Then, the GUI is mouse driven. Pressing left or right mouse button does things based on what tool is selected. Ctrl is a modifier, so each tool has 4 possible actions. Try it out!


Link: Snapshot 002

  • Throwing is back. Throw small cubic chairs with 4 different strengths! 
  • The engine now support alternate rendering profiles. A single new profile has been added that has the potential to make it as the new default rendering profile. It is turned off by default, but you can swap between them with the lower-case 'L' key: 'l'. Not to be confused with '1', the digit one. The new profile has stronger bump mapping effect because light caused bumps and ambient bump mapping are no longer destructively overlapping. Properties of light circleing objects and spherical harmonics are also different.
  • Module system has been updated to support models. Full support is yet not available.
  • The system hub has a new incomplete model manager.
  • The engine now support running in a frame-less window.
  • The options dialog allows you to control the frame of the window. Might exhibit some rare bugs. It can also be used to change the terrain quality.
  • More system keys are available while in system hub mode.
  • The engine now uses less considerably less array indexing when rendering models.
  • I managed to render to floating point surfaces. Might enable HDR or other rendering methods in the future.
  • Temporary hacky fix for the far-plane clip artifacts on terrain.


  • Fixed a few more bugs related to mouse state management (yet again).
  • B005: Falling off the world corrupts the save file. When loading it, the engine will crash. Fixed. If you have a save file affected by this bug, you will have to delete it and create a new one. New saves are not affected by this bug. Can't fix it retroactively because the save format changes from snapshot to snapshot at these early stages.
  • Fixed a bug with reading floats from XML. This could fail based on your system's internationalization settings (1.1 vs. 1,1).

Friday, July 19, 2013

103 – I suck at shaders

After so much time, I still haven't mastered shaders! Not even something simple, like bump mapping with ambient bump mapping and spherical harmonics. I've been always suspecting that there is something wrong with my bump mapping and this comes down to three things:
  • My bump maps suck. I have done them myself and while things like the stool seems to work OK, the barrel from some angles always seems to have the metal rings protrude instead of extruding.
  • I think that the equations I am using are not exactly correct. The point light one was clearly wrong and I fixed it, but the fix makes no sense mathematically, at least for me.
  • The weights of each component are not well enough tuned.

In order to better locate my errors and increase quality, I am going to do three things:
  • As a background task, I am going to study in depth (again) normal mapping and spherical harmonics. All the theory that I can get my hands on, making sure that my equations are correct and I understand why.
  • Get my hands on some sample normal mapped meshes that I can use as reference and try to improve my own maps.
  • Introduce a secondary rendering mode to the engine.

For now I only managed to introduce the new rendering mode. The engine still renders things as it used to, but with the press of a button ('l' in my case) you can switch to a second rendering profile on the fly. Using this I managed to compare parameters more easily and came with an all new rendering profile. Be prepared, because I am going to spam you with before/after screenshots.

So here is a before scene with a  lot of barrels and stools:

And the after shot:

Notice the added depth illusion on the barrels and stools, but also it is pretty clear that the rings on the barrel are not extruding the right way. I am going to go around this scene and post before/after comparisons form different angles:

Getting those images aligned in Blogger was harder than needed :). I hope they are in the correct order. The effect is customizable. Here is another profile that should look exactly the same to you, unless you know what to look for, and even then it will probably looks the same. Well, it is subtly different and alters the properties of light circling objects.

You can come up with a ton of settings and the parameters have a wide range of valid values. Here is what happens if you go outside the valid values:

I kind of like the new profile. I can't wait to get my hand on some proper normal mapped meshes done by professionals and use them to fine tune my new profile. Also, spherical harmonics produce some unnatural results for quite a few input values with this new profile (carefully avoided these values for the screenshots), so that should be also put on the list of things to fix.

But in my engine I don't have only one light source. Let's try out the personal torch, which in the new skill tree will actually be a spell in which you need to invest ability points:

From the front it makes a big difference, because there are two lights meeting heads on. From behind, there is less of a difference between the two profiles. The personal light is best used at night though:

This was from behind, with the moon illuminating the front of the objects. Let's take a look at a shot from the front, where we can see first only the moon light in the two profiles in dark, then with the personal light on:

I grow tired of these orderly scenes:

So, what do you think? I think this is a massive improvement. I still need to try it out with some proper normal maps. I hope I can fix spherical harmonics and get my hands on multiple sky-boxes, so we can see the harmonics in action, with the skybox contributing in real time to the illumination of objects (the contribution is minor). If I somehow am not able to fix the harmonics with this new profile, I might get rid of the atmospheric illumination element of shading. It is cool, but not really needed.

Screens of the day 34 - Fogged

Oh boy, a new screens post! The blog is back on track!

With the latest changes to world scale and the new skybox, I complained about some very ugly artifacts near the far plane when moving around the camera. Let's take this scene as a sample:

You can see the fogged terrain in the background. It looks OK, I won't complain that much about it. The problem is that even moving the camera a few pixels changes the shape of the terrain that intersects the far plane:

As you can see, looking a bit to the right has caused most of the distant landscape to disappear.

Moving yet again causes new terrain to appear.

What is the solution for this? Well probably I am going to need to implement a proper terrain some day, with LOD. Probably geo-mipmapping. Then you can have a lot of low quality terrain in the distance, move the far plane farther away and scale all fog values so that any popping, if it exists, is barely visible because it is so distant. But my terrain implementation is crap and it has a very shitty draw distance.

Like I said in the last post, I can't fix this for now. But I can hack something together just for Snapshot 002 to get rid of the moving far plane artifacts:

OK, view distance is a lot lower, but at least you can't see the artifacts. Let's fine tune the values and see a few more shots:

The only problem is that it does not behave good when looking from high above down:

Oh well, good enough.

I also noticed that the skybox should be rotated, because the light shines from opposite direction as my terrain is illuminated. Wrong :):


Finally, a few posts ago I mentioned that with the freed up resources from simpler terrain rendering, I would either add more materials or implement triplanar mapping. I did some tests with triplanar mapping. I wanted to show some screenshots, but they don't look that impressive. Yes, things look slightly sharper at steep slopes. The effect is expensive, so until I either increase its quality or performance, I won't be using it.

Thursday, July 18, 2013

102 – Snapshot 2 is going to be great

The theme for snapshot 2 was "RPG", so let's leave that for last. Take a look at this screenshot from the engine:

I should really start cutting out the window border for dramatic effect! Anyway, in this screenshot you can see the new "hub" approach I have taken. The engine has now two centralized states, one meant for exploring the worlds and one for using the tools available in several windows with a traditional mouse  driven interface. You can swap anytime form game mode to system mode by hitting "`" on your keyboard.

When the little menu in the top left corner is visible, you are in system mode. Here you will have access to normal tools, like the world manager which is basically a much more advanced save manager and the game options. Normal things for a game. But you also have access to the engine's constructions set, which is build in seamlessly into the experience. You don't need a separate executable, learn a different tool, wait for loading, push updates or fight a sluggish interface. The tools from the construction set work instantly at any point and the changes are reflected live in the running game. The price to pay for this is minimal, except for the development time :). One note though: when hiding the system hub you will revert back to your loaded game. If no game has been loaded, you are left with an empty screen. Take a look at the screen that greats you when you boot up the engine:

OK, you probably won't have any saved maps. After creating a few you will get this. The purple background signals that no map is loaded. In the future there will be a clearer indication for this. If you load a map you will be placed normally into game and the system hub will disappears. Press "`" and it will be shown again, with all windows in the same state you left them (except for the world manager, which closes by default when loading a map; but if you show it again it will be in the same state as you left it before the close).

The world manager now lets you also give the height in percentage for the newly created maps. By snapshot 002 it will be updated to support entering the world seed too.

Another new comer is the options window. Currently it only supports just the main graphical options and it does not read or save them from disk, so every launch of the engine you have to change the options if you don't like the defaults, but it is a good start. It will be updated soon.

You may also have noticed the skybox. Adding this was done through the material editor, which received a healthy upgrade in functionality.

But before this I added primordial support for modules. Modules describe all the rules and content of the game and are designed with mod-ability in mind. Once the support is rich enough, the era of hardcoding is done for. For the engine to work and look the way it does, you need tot know what materials and meshes to load, what materials to assign to which part of meshes, what meshes to assign to the abstract in-game objects, what physic behavior to assign to them, etc. This was done in the past by hardcoding these relations, but a fully moddable experience needs to add support for creating these relations visually in the construction set and save them as data in a format designed to be easily modded. So now the material editor can add new materials and it also saves them to the main module, which for now is in XML format.

After the editor and and core engine skybox support was added, I was left with a white skybox. Using the material editor, I assigned the textures one by one, seeing the changes reflected live on screen:

I also added night support for the skybox:

I know believe the night is not dark enough.

But the skybox raises a new set of problems. In the past with the sky being a single color and fog having the same color at its maximum intensity, the far clip plane of the terrain was perfectly hidden. Now you can see it and it is rather ugly! My solutions for this: don't look at it. This is very hard to fix and impossible with my current view distance that is very low. So for now I'll ignore it, especially since it will take me months to find a solution for it.

There has been another major change: the engine now uses logical coordinates. In the past I have had many issues with the world scale. I had to change it around several times, a lengthy and error prone process. Now the engine uses logical coordinates and every dimension is expressed in abstract "meters". When going from logical coordinates to real ones, meters are converted to unit-less vertex coordinates at a fixed scale. Every world can have a unique meter size, but once I find the perfect one, all worlds will use the same scale. The scale is a hidden value anyway.

Adding logical coordinates support took me about a day. After adding support, I reduced the scale of the world several orders of magnitude. This had the unfortunate effect of changing subtly the look and feel of the world. Physics behaves differently and fog had its look changed.

So finally let's talk about the RPG elements. I'll write a post about my design for character progression, but for now I added a stamina bar with fairly complex rules to it. You have different rates of gaining and loosing stamina, together with an exhausted state again with its own rates. Governing this stamina bar and your overall mobility is the new skill tree:

Currently it is only a single tree, but the skill points can have drastic effects. 8 of the 9 points are implemented, with the last one being scheduled somewhere in the next few weeks. Instead I'll add further trees.

The progression in the trees is horizontal. The tree has mostly passive abilities, but the passive ones can have such a drastic effect that they will change your active gameplay style. Green outline means a new ability, yellow outline is an upgrade to a previously unlocked green ability and blue outlined abilities are mutually exclusive with another blue outlined ability near by:

This is the build I use during testing. The abilities are 100% implemented and even slightly balanced, but the tooltips are not complete or correct. Since balancing changes every week based on my experience with the ever evolving trees, the tooltips won't be 100% accurate for months. This is a common problem in games, with world of Warcraft being a prime example :P.

The skill tree works closely with a new class added to the engine: the Player class, which manages all stats and attributes and is responsible of spending and gathering resources like stamina. 

Currently the player character is reset to default at each game launch so you'll have to reassign you ability points.

So that's about it for the main features that I manged to squeeze into Snapshot 002. After some more polish and testing, it should be uploaded on Monday.

A fair warning regarding snapshots. At this early stage, save game compatibility is basically 0. Most new snapshots will render your previous saves useless, so a new set of generated worlds for each snapshot is called for.

  • Implemented to system hub.
  • Implemented primordial module system.
  • The world manager can now control the height of generated maps.
  • Enhanced material manager: you can add materials and it saves the material list. 
  • Basic options menu added.
  • Player character class has been added to keep track of the player's status.
  • Ability trees have been added. Currently only about half of the mobility tree has been added and implemented.
  • Skybox support.

  • Fixed most of the bugs with changing states in the system hub.
  • You can now copy & paste in the engine as you can in any windows application.
  • Material editor no longer crashes when loading a non-existing texture.