Greetings Ghouls and Ghosts! We got another devblog examining some of the behind the scenes of the new event coming to foxhole. Hop on the back of my nimbus 2000 and lets go for an adventure.
Disclaimer: All of the content below is heavily work in progress and is subject to change.
The Dead Harvest Game Jam
Making games is one of the most rewarding and challenging things to do. It's highly gratifying to toil with your co-workers for a long time to work on a product that people enjoy. That being said, it can be exhausting work. You find yourself working at a pace and an intensity that sometimes cannot be sustained by anything less than pure passion. Foxhole certainly is a passion project for us, but at times it can wear us down.
We always have fun ideas at the office, but there is such a backlog of work to be completed that there usually never is time. The only way in which we would actually make some of these ideas into reality is to schedule time for it. Coming off the substantial amount of work we did for patch 0.17, we decided to allot a single Friday to spec out and create something fun for the community, that would also be entertaining for us to work on. We treated this endeavor like a “game jam”. Essentially, game jams are events in which developers can prototype an experimental idea into playable game in a very short period of time. The Dead Harvest Halloween event was the output of our game jam.
Like most things on Foxhole we start with what is awesome, what would be so cool that we would love to see in the game. We then start removing the things that are a) impossible or b) not in line with what Foxhole is. We boiled down the infinite ideas of what a “zombie” mode could be, and distilled it to a list, something that we could conceivably complete within 1-2 work days. We really hope that you guys like the event, It was certainly a lot of fun to work on. If this is successful we plan to iterate on this event and make it that happens a few times a year.
The Joy of Animation
For me, that was one of the most interesting parts about doing a new game mode with new characters in the shortest amount of time possible. In order to bring the dead back to life, new animations had to be made. Making brand new animation is a very time-consuming task; so I had to find ways to re-utilize whatever was there already. Thankfully, I knew that the zombies would only have simpler movement patterns in contrast to a regular Foxhole soldier. Most of the complexity of the animation system emerges from the need for the characters to perform many different movements in several different stances while utilizing a myriad of armaments.
Fortunately, zombies don’t carry weapons. With that in mind, what is the recipe for making zombies on a budget?
The first ingredient is identifying which animations have to be unique and which can be re-used from the regular soldier. In this case, I knew that the majority of the movements could be reused. That included: strafes, crouching, climbs, prone movements and more.
Despite reusing several animations, the zombies required a whole set of idle walks and run cycles. I started preparing new animations for the forward movement at various speeds. And since walk cycles are the bread and butter of any animator worth a dime, simple forward cycles don’t take too much time.
Those new moves, when mixed with a brand new hunched-over idle animation, were enough to sell the physicality of this new undead character. The last ingredient to be added to the mix was a new melee attack animation. The regular soldier punch simply wouldn’t sell the illusion of ferocity. No way around this one. No re-uses or modifications of pre-existing animations. In this case, it’s good old straightforward animation work, and it took me a couple hours to get it to a stage I could use in the game.
After all that, the last thing is merging all the new animations together with the old system. In order to save time, what I did was merge the already existing animations to the new ones. For that, I used a little trick where only the front runs in the lower body blendspace 2D get substituted and combined with a single blendspace 1D for the upper body. This upper body blendspace actually remaps the speed so that the arm raises to its maximum to be at 800 speed rather than 400. I can only do that because zombies will always raise their arms forward, towards their prey, rather than having to perfectly blend arm movements with leg movements. That last paragraph probably meant nothing to most people; but suffice it to say, the animation system looks much less complex.
Add some blood. Some moans. And ding! You got yourself a freshly baked zombie character, hot and ready to rip your face off.
Sometimes there are crashes that happen so rarely that they are really hard to track down. Often the hardest part about solving a bug is getting it to happen on our own machine, also known as "reproducing" the bug. Once we can get it to happen for us, then we can attach a debugger and see what's going on. Even if you think you know the cause of the bug and make a fix for it, you can't really be sure you fixed it unless you have a set of steps that reproduce the bug, so you can do a before/after comparison.
In this case, we had an intermittent crash that had existed for a long time, but we were unable to get a reliable repro case for. Up until a few weeks ago there didn't seem to be any pattern to when it happened. However, we began to see elevated numbers of this particular crash after the last release. After following up with several players, it seemed to occur more when they were around bushes.
Before I get into the specifics of the crash, some quick background on how some of the involved UE4 systems work. UE4 has a game thread, where all the gameplay logic happens, and a rendering thread where the engine talks to the GPU and does the actual rendering. Because the game thread could still be modifying things that are being rendered, it uses "render proxies" to capture the info it needs to render a given mesh. This allows both threads to proceed forward without waiting on each other.
The crash itself was an invalid memory access inside a dynamic material instance on the rendering thread. Looking at the crash dumps sent in by players, I was able to determine a few key pieces of information:
1. The dynamic material in question is the hammer material
2. The material was attached to the hammer mesh on the player's character
3. The character was in the middle of the hammering state
Additionally, it looked like the rendering thread was trying to use a dynamic material that had already been cleaned up. The garbage collector (periodic system that cleans up unused objects) had run and freed the memory for this material before the mesh that it was attached to had finished rendering.
With this information in hand, I started narrowing down what could actually cause this. I knew only a few parts of the game used dynamic materials. These are materials that inherit from a parent material, but have some parameters changed for that particular instance. One of the systems that uses this is the character visibility system; when a player fades in, all the meshes on that player have a dynamic material created with an opacity parameter that we change over time to fade them in. A dynamic material is used here or else fading that material would fade in all the things that used it, instead of the just the specific player we want to fade.
I had also recently made a change to the visibility system, where your own character becomes semi-transparent while standing still inside a bush. This makes it more clear to the player that they are invisible to others. This change seemed correlated with the increase in reports of this crash.
Working from the assumption that this crash is somehow caused by hammering while inside a bush, I started testing locally with a debugger attached. I tried various permutations of moving slowly, stutter stepping (to trigger the fade in / fade out rapidly, and building structures inside a bush while invisible. I also had enabled an option to run the garbage collector every frame, instead of as-needed. Additionally (as with most bug investigations) I was simulating some network lag to replicate real conditions. After some time doing this I was finally able to hit the crash locally! This was great, and I was able to confirm that it was indeed the same crash that players were seeing. However, the reason for the crash was still not clear; I knew it was using a cleaned up texture, but why was it cleaning it up?
I worked back from this and set a breakpoint in the code that creates the material in question. I knew that it should be creating a dynamic material for the equipped item when it was equipped and you were in a bush (or it became equipped while you were already in a bush), but it must have been making more than one instance because (when it didn't crash), the hammer still rendered faded out while inside the bush in all cases.
This finally revealed the root cause of the bug. The material is cleared when a new item is equipped, or when you receive a correction from the server for what item is actually equipped. In this case, the player was getting a correction when they tried to hammer something with exactly the right timing and amount of lag that said their hammering action didn't succeed on the server. This correction would then re-equip their hammer, clearing the set of dynamic materials, and creating a new one right before the rendering thread tried to render it. The render proxy was still pointing to the old material, which had already been cleaned up.
The exact set of steps was as follows:
1. Be inside a bush and still (so dynamic materials are created on your own equipped mesh)
2. Rendering proxy for the hammer is created referencing the dynamic material
3. Client receives a correction from the server in response to trying to hammer
4. Client re-equips the hammer (clearing dynamic materials)
5. Garbage collector runs and cleans up the dynamic material
6. Render thread continues rendering and tries to render using the hammer's render proxy and cleaned up dynamic material
Several things had to go wrong for this crash to happen. You had to be inside a bush, get an activity state correction (quite rare unless under high lag), and the garbage collector had to run at the exactly wrong time. This explained how rare the crash was, and the relative spike in crash cases after the visibility change. Even if someone had hit the exact same set of steps, their garbage collector may not have run and it wouldn't have crashed - perhaps just had a slight rendering glitch instead.
Once the root cause was found, the fix was simply to clear the material in a different way that flagged to the render thread that it needed to make a new render proxy with the correct material. Oftentimes a full day of bug hunting results in a one line fix!
I wouldn't have been able to fix this bug without all the players who sent in crash reports,
so please post any crashes you have in the #bugreport channel on the Foxhole Discord so we can take a look!
Q&A with Clapfoot
[RCC] LT_Loser: How far do you think the Unreal 4 engine can go from the point (player count and other features) and do you think the team made the right choice?
Matt: It is possible to push things a lot further, but would require some amount of custom optimizations. We’re always trying to squeeze out everything we can from the engine and to be honest, we push it pretty hard. We absolutely made the right choice. If we had used our own tech, you likely wouldn’t be playing the game yet.
Even with some of its limitations, UE4’s network code gave us a good starting point, allowing us to get off the ground and running. That wouldn’t have been quite as easy with some of the competition. Not to mention, the art pipeline is very strong and only improving. For a small team like ours, it’s an invaluable tool. No game engine is flawless.
We love the Unreal Engine. (GAS) Heimdall the Gimp QM: Will there be more statements on the lore/world at some point?
Matt: To be honest with you, I’m not sure what you mean. But I liked the gusto of the question, so I decided to include it.
You won’t get any statements from myself or any of the team in any direct way about the lore. There will be plenty more to read and discover as we release new features or interact-able lore items get added to the game. Am I just going to answer your questions outright? Or tell you who started the war? Or which faction is better? Or what Callahan’s midi-chlorian count is? I’m sorry, but no.
I think that part of the fun is in the discovery process and using context in the world and the uniforms and vehicles and town names etc, to learn about the world. If we gave you some spoon fed lore, what fun is that? You can read a book or watch a movie for that. Games offer the opportunity for organic discovery and that’s something we believe in.
[PvFOD] barracks bp: High ground has a big problem with visibility, as the camera always flies a fixed position above your head, so you can't see the top of the hill when you aim while standing at the bottom. Will the camera ever be "dynamic" and will fix itself above the position of the cursor instead?
Matt: This is an interesting and ever-present issue. With the Town Revamp, Max and I are working very hard to ensure more predictability w/regards to height. We’re also receiving some more continued support with mechanics to make the process more smooth. Especially when it comes to being on high ground, behind cover.
I doubt the camera will be dynamic as we tie a lot of the aiming and processing of visibility to the position of the camera. Not to mention we don’t allow things to be relevant on the network outside of the player’s camera. This helps with server optimization.
We want to make sure that we have engaging, fun maps with surprising environments and cool set pieces, but we also want players to understand how everything interacts with the world. That’s been a hard thing to balance. We’re getting better at it though, and I hope with upcoming changes, it starts just feeling like a fun part of the combat game, instead of something frustrating that you have to put up with.
PickoYesterday: What are some of the thought processes you take when designing a new map? Is there a theme that you go with or is a lot of it just on the fly? Is balance something you think about early on into the map creation or is it tweaked once you have got the bulk of the map finished?
Matt: To answer simply, balance is considered long before it even sees the engine.
Here's a more detailed breakdown of the process,
First: Mark has a plan. He’ll come to me and say “We want a map for x,y,z purpose.” Let’s use Oarbreaker as an example. Originally Mark approached me and asked me to start thinking about a map that would help to allow for more simultaneous open fronts during a war. At this time, his only real thought was, “maybe it could be an island.”
Second: I establish a theme w/ Mark. We talk about what kinds of themes might support the goal. In the case of Oarbreaker, an archipelago made a lot of sense. It would mean there was a lot of work to do for players, we’d have two Port Bases, so it could always be spawned into, and allow passage into the west coast for landings late game. One other requirement was that at least a couple of the islands had to be the size of, or bigger than, what was Fisherman’s Row at the time.
Third: I draw a paper map, or rather a series of different maps for discussion. Here are some of the thumbnails. We then discuss what’s been presented and talk pros and cons, then decide on an overall direction. A lot of balancing is considered at this stage. To that point, we changed things pretty dramatically even after we decided on a direction, for the sake of balance.
Once I had the islands laid out in 3D, we moved everything around. See if you can figure out which of these thumbnails we chose. It is in this batch.
During this phase, I’m also gathering references. Here’s a couple I used to guide the design.
[RCC]Autopilot: how much of a factor is terrain in the mapmaking process compared to all other elements considered? (Spade's surrounding terrain was just pure gold and I want to see more of that stuff)
Matt: Terrain has been and continues to be a huge part of the map-making process. The terrain is the skeleton on which all else is built. We consider everything from heights to curves to sizes to distances. Everything is highly calculated. If a map is bigger or smaller, that’s intentional. If a road bordering a cliff, that’s intentional. Sometimes, it doesn’t always work the way we want.
Foxhole is the first game at Clapfoot that we’ve had to use a terrain as a significant gameplay element so, over the course of development, it’s been a huge learning process, and we continue to learn more about the strengths and weaknesses with each map we make.
Max and I want to incorporate a lot more variable across the board—the world is not flat—but height in Foxhole is a tricky subject, as mentioned above.
A really hot guy: what kind of shampoo does HB use?
HB: I have to rotate often since my hair gets used to the same shampoo and does not look good if I keep the same one for too long. I tend to cycle between Garnier for dyed hair and Dove for dry hair when the tips are a bit frailer due to the discoloration.
But the main thing is that I treat the tips with conditioner every other day; once a week at least, I apply some argan oil to the dyed portions of my hair (I also use a bit of conditioner on my beard every shower so my girlfriend doesn't get prickled too much).
That wraps up another Dev Blog. Be sure to check out our Foxhole Dev Stream for more information about upcoming features. If you have any burning questions you want answered be sure to tweet them to @Matt directly on Discord or Twitter, if your question gets selected it will be answered Live on stream!
A top his trusty broomstick, Adam swoops across the internet, cackling.