Finally a REAL developer journal.
In game development, one of the most important polish points is making sure that the game is always responsive to the user. It’s better to take 1 second longer than to stutter or hang for frames.
Unfortunately, it often gets very expensive in engineering hours to meet that objective.
In Elemental, we have two areas that are particularly problematic in that area before you even get into the game:
#1 When you first launch the game.
and
#2 Waiting for the game to start (i.e. the time after you start the game and you actually get to the map).
In both cases, the delay is due to parsing in XML. It’s completely a CPU bound issue.
Elemental is designed to support multi-core but we also support single core processors too. And when you first launch the game, that’s where the guy with the quad core is going to notice the biggest difference versus the guy with a single core CPU.
THIS…
BECOMES…
Which has to be parsed…
…
Anyway…
The “stuff” that has to be loaded before you even see the title screen is in the \english directory. All the other XML gets loaded after. The problem is, there’s a lot of stuff in that english directory.
Thankfully, tile designs don’t have to go in there. But UNITS do and “GoodieHuts” and anything you would expect to encounter while setting up the game.
As of now (1:43am on Wednesday, August 11) there is 3.31 MEGABYTES of XML (that’s just text mind you) that has to be read in and parsed. Heck, there's 65K of hats that have to be parsed (I mean literally, hats as in things to put on your guy’s head).
But by far, it’s the units that are the biggest chunk of it. And we’re adding more creatures every day. So this is something we’ll have to find a way to deal with between now and release otherwise, the first impression will be crummy for people with slower machines.
Either we have to find a way to move the unit parsing into a background thread (not as easy as it sounds) or we need to find a way so that we don’t have so many units being loaded up front.
Now, the other issue, which there really is no way around, is the time it takes after you hit “Start game” and you get to the map. All those wonderful tiles have to be loaded and parsed. There’s 11 megabytes of XML there. It’s the nature of the beast unfortunately and one that I think most would agree is worth it. But you can bet, many of us are looking forward to going through all 770 tiles we have as of tonight and seeing which ones we could get rid of.
I have never really tried working with XML cached into a binary but that does seem to be an ideal way to work with a static DOM.
EDIT: To summarize the following - use XSLT to generate multiple indexes that can be used by the loading/creation code to select elements that are then retrieved in full from either the original source files OR from a structured cache also created using XSLT.
One thought that came to my mind, to deal with the size and element count issues, would be to pre-process the /english directory into a single, transformed, "cachehooks.xml" file with two sections. The first section would be for map initialization elements while the second would be for game spawning elements.
For this file figure out what the bare-bones xml element requirements are for selecting an object as needing to spawn and only load the entire XML element from the associated "english" file location when the first spawn occurs. When building the "cachehooks.xml" you would simply include both the source document path that the hook points too as well as the unique identifier found in the souce file.
It would probably be worthwhile to also create a cache directory for all the core assets in which each element is placed into its own file (for play-time spawn items). For map initilization items you'd want to create a single XML file with all the required map items since you know you will be loading most or all of them initially. This would be a good file to binary cache if possible.
For modded elements you can provide hooks so that mods could do similar caching; though by default the size aspect of most mods is likely small and thus simply loading them in up-front would be acceptable; especially since it is likely that the people using mods are already going to be familiar with the basic load times and performance and if you indicate that you are loading mod info they are probably going to understand the additional delay.
One of my questions/concerns is how the end-user of a mod will know if any of the core assets are being overridden by the mod; since an asset in the mod XML files can have the same unique identifier as an asset in the core XML. Having some kind of overview screen in-game would be helpful to alert the user too how assets they are used to from the official game have been altered.
To be thorough, the way I would attempt to create such a cache setup would be to incorporate XSLT applications that can be run through the game (when necessary) but that can also be run in stand-alone mode to create the cache files/directories and bundle them with the distribution. The cache should look to see if the cache is empty and rebuild it immediately. Additionally, the "cachehooks.xml" file should contain a "hash" of the contents of "/english" and the game should evaulate a new hash at launch and, if they differ, rebuild the cache as well.
EDIT: You could take the caching somewhat further, if needed, by creating separate files based upon factions and then only load those factions that are relevant. This would work well for in-game spawn stuff since instead of one-file-per-asset you could do one-file-per-faction (with sub-files where desirable).
@polobo: maybe I'm misinterpreting your idea, please correct me if I'm wrong, but even after xslt translation the amount of xml to be parsed would be pretty much the same, just differently ordered and maybe in less files than originally? with additional cost on ease of development for mods because they'd have to contain more logic than without (effectively removing the beautiful simplicity of directly deserializable xml).
I've pretty much given up on c/c++ so I'm not familiar with the complexity of serialization there, but for c#, the choice of (de)serializing into xml or binary is just a few lines of simple code (using a binary formatter or the xml serializer), the classes need few to no adjustments (probably a few attributes) to allow both methods. a quick google search revealed, what seems to be about the same possibilities with ISerializables etc, but I haven't tried it out (because I'm just to lazy *g*).
one of the projects I'm working on actually uses serialization for xml, json and binary at the same time for different uses. binary as a fast local storage in offline mode, json as the language the REST serverapp uses for communication and xml for exporting to the android app. the different DAO implementations by definition all deliver the exact same objects so the application doesn't have to know where the data is coming from. I really love the flexibility I have with this implementation as I was able to exchange the first prototype direct database DAO with the final json dao without having to change the code using the objects at all. thats why I immediately thought of parsing the xml and caching the data binary until the xml changes. the code within the game doesn't have to change, and it gives a great performance boost without much cost. there surely are even faster methods, but those probably would require a major overhaul which at this late stage of development probably wouldn't make much sense.
I'm really curious how the devs will solve this ultimately and what influences their decision, maybe they find an even better method of getting speed without losing flexibility.
Yeah, that's what I'm thinking too. The game is ultimately going to *use* some kind of binary format in memory when it wants to use the data, the XML parsing is just to read and load the data files into whatever data structure it's using after loading. You can serialize that thing, and next time skip all the parsing by reading and deserializing the binary object(s). (I do this at work for offline storage when someone is in the field and doesn't have access to the web service (SOAP instead of REST for us, but same deal). It's blazing fast compared to parsing a lot of XML.)
I'm not sure what using XSLT would bring to this when you could add a simple "if XML is unchanged, load the cached binary" check to the loading process instead.
I agree on the value of the binary cache.
I guess the indexing of the XML "solves" a related issue in that you do not have to load everything into memory but divide your data into chucks that can be loaded separately and only to the (broad) level of detail required.
It would likely require a larger effort at this point to retrofit something like this in as opposed to simply loading all the XML into one massive DOM (from raw XML or from a cached binary format), load in MODs on top of that, and then generate the game objects in a push manner.
But to clarify; there wouldn't be any additional work for developers putting together the XML. Aside from the apparent limitation that each file may only have one kind of game asset (based upon a quick perusal of the /english directory) within it - as denoted by the root element. There would simply be intermediate XML structures that the game would be looking for and that the XSLT middle-tier would transform the developer XML files into.
Just some thoughts, and probably a solution that is still looking for a problem to solve...I don't have enough insight into Elemental to know whether this would even be worthwhile nor, for that, matter, whether it is something I would even design into my own games. The area intrigues me but I've yet to really find a use for the tools and techniques - to that level of complexity - at my real job.
One advantage to having an index and putting it into chunks like you say is that if something changes (eg: installing a new mod), you don't need to rebuild the whole thing. You can know from the index and a checksum where the changes are and only parse those files. That would make installing a mod significantly faster then if you have to reparse everything. (The most straightforward way is probably to have a chunk for the core game files, then each mod gets its own chunk and you combine them in memory after loading all the chunks.)
So, lots of good ideas here.
...<ObjectLOD>4</ObjectLOD><IgnoreTerrain>0</IgnoreTerrain><BaseScale>1.0000</BaseScale>...
I see a lot of redudancy here. And the schema is simple, there's just a high iteration of elements. It would not be hard to define an XSD against a lot of these with appropriate defaults defined and rebuild these files with up to 40% less content. Parsing would be sped up and your I/O would be reduced as well.
I have to say binary serialization was the first thing that sprang to my mind too. But then your objects become these big blobs. I'd think it'd be to their advantage to have this nice queryable format to refer to later so that they can optimize for lazy loading of resources. So I'm not sure if they'd want to continue to deserialize these large binary objects in to memory everytime the application launched, especially once these components start to grow to hundreds of megabytes in size. (Maybe not now, but with enough modding, patches, content releases, etc...)
Perhaps they could still find a way to chunk out the binaries, as was suggested previously, so that at least the data would only need to be loaded in logical sections that are still granular enough to be read from disk in an on-demand fashion, but XML abstracts a lot of this for them. Sure it's more expensive but that's the trade off for abstraction.
Either way, I'd still look at constructing those XSDs and providing some default values. There's no reason to have IgnoreTerrain = false stored 10,000 times over in your persistence. There's some normalization that can occur here. Just my thoughts.
- Zesban
Some info on monsters, tech, faction or lore on map creation would be good enough.
People with single core CPUs have themselves to blame.
There are many great features available to you once you register, including:
Sign in or Create Account
