Finally a REAL developer journal.
In game development, one of the most important polish points is making sure that the game is always responsive to the user. It’s better to take 1 second longer than to stutter or hang for frames.
Unfortunately, it often gets very expensive in engineering hours to meet that objective.
In Elemental, we have two areas that are particularly problematic in that area before you even get into the game:
#1 When you first launch the game.
and
#2 Waiting for the game to start (i.e. the time after you start the game and you actually get to the map).
In both cases, the delay is due to parsing in XML. It’s completely a CPU bound issue.
Elemental is designed to support multi-core but we also support single core processors too. And when you first launch the game, that’s where the guy with the quad core is going to notice the biggest difference versus the guy with a single core CPU.
THIS…
BECOMES…
Which has to be parsed…
…
Anyway…
The “stuff” that has to be loaded before you even see the title screen is in the \english directory. All the other XML gets loaded after. The problem is, there’s a lot of stuff in that english directory.
Thankfully, tile designs don’t have to go in there. But UNITS do and “GoodieHuts” and anything you would expect to encounter while setting up the game.
As of now (1:43am on Wednesday, August 11) there is 3.31 MEGABYTES of XML (that’s just text mind you) that has to be read in and parsed. Heck, there's 65K of hats that have to be parsed (I mean literally, hats as in things to put on your guy’s head).
But by far, it’s the units that are the biggest chunk of it. And we’re adding more creatures every day. So this is something we’ll have to find a way to deal with between now and release otherwise, the first impression will be crummy for people with slower machines.
Either we have to find a way to move the unit parsing into a background thread (not as easy as it sounds) or we need to find a way so that we don’t have so many units being loaded up front.
Now, the other issue, which there really is no way around, is the time it takes after you hit “Start game” and you get to the map. All those wonderful tiles have to be loaded and parsed. There’s 11 megabytes of XML there. It’s the nature of the beast unfortunately and one that I think most would agree is worth it. But you can bet, many of us are looking forward to going through all 770 tiles we have as of tonight and seeing which ones we could get rid of.
WooooHoooo I Win
P.S. You're up WAY past your bed time Frogster. Get some sleep, man. I'm the only one allowed to be up this late.
P.S.S. NooooOooooooOooo don't get rid of any tiles!!!!!
Heh and I'm the sucker still on a single core.
Oh well, I'm used to:
a. my machine getting reminded it's getting old
b. games not running super fast because of a.
Plus, if the game is good, I'll wait. I can always go get a sandwich or hit the toilet (I mean, I'm not going to want to get up for something as lame as eating or peeing when I have Elemental to be playing or modding)
So bring on the MB of XML to parse! I know Beta 4 doesn't have nearly all the stuff, but it flies on load up, so I'm hoping it won't be all that bad anyway.
In regards to the parsing issue and the amount of time it takes. One possible solution could be to parse all of the xml and produce one large binary file upon starting the game (and loading assets from the binary file). Then for every subsequent re-launch simply scan the entire XML directory for changes (time stamp changes) since the game was last run, if no changes were made use the binary file.
This way you can reduce what ever I/O overhead you're having on loading every single XML file to a degree as well as hopefully parse the binary file at a far more quicker rate. Of course you'd need to work out an entire format for the binary file and work on how to build it from the XML files etc... Would be a lot of a work.
Of course that would mean that the very first time the game was run, it would take a longer amount of time rather than a shorter amount of time. Some sort of "compressing resources" splash up + loading bar would work well to relieve the "augh, this game is slow" emotions.
As for the processor performance: does Elemental only exploit multiple cores while loading the XML files? Because I've got a dual core CPU, but in-game, according to the Task Manager, Elemental is using only 1 core.
Would it help if you combined more of the XML files into bigger XML files only have less of them? Not sure if reading individual smaller files takes longer than reading less but larger files. Maybe be an option though?
Another sucker with a single core here as well . I don't mind waiting at all for a new game, the only thing that would make me sad is that I won't be able to test my own XML modifications as easily, so I guess I'll have to do various XML changes at a time . Or maybe I could do a really small map just for testing purposes, or loading games with modified XML files would update the loaded game (idk if this is even possible).
Regardless of anything, looking forward to the 24th
I know very little of this side of development, I'll never be a code monkey. However I am often accused of asking the right questions. So why does all the data need to be front loaded? Are all 65k worth of hats available to the player from start to finish? Is there no distinction between hats likely to be used early game to those late game. If hats ( the most simple example to use for the discussion ) have different values and uses, can they not be loaded at the time of their use? Likewise, are the lesser hats required to be loaded later in the game when clearly better 'higher tier' hats are available?
On the other hand, if hats are all descriptive and decorative, and the player is in the process of selecting a hat for their unit, why load all the hats when a few wire frame hats would do just as well and the player then design 'their hats' and after only need to load the hats they have selected to use until they decide to design new hats?
I am very much looking forward to this title, and my anticipation is ratcheting up my expectations.. I have a Core Duo, and as long as it runs smoothly I won't care what early sacrifices are necessary for achieve stability and performance. Your core game looks awesome. If all your mem leaks are nailed down, I would be very disappointed if E:WoM failed to achieve a AAA status among the Turn Based Strategy game market. I am a hard core Masters of Orion player, long have I yearned for a title to achieve as much as it did.
yeah, binary serialization/deserialization would be way faster (and use only a fraction of the space the blown up xml uses *g*). If you're set on optimizing even the reloads when xml has changed, use one binary file per directory.. so when someone is modding tiles (s)he won't have to generate the hats on every start, but the tiles will be serialized again. of course a 'clear cache' button in the settings would be necessary to solve the probable problems with the 'hasChanged' detection
the only other solution I could imagine would be to load only the absolutely necessary stuff on startup, then, wile waiting for the user to click something (in terms of cpu thats quite some time) you could already prefetch units, tiles and stuff - in a background thread, of course this would have the nice effect that one can start the game go away and upon returning (with food and drinks for at least a few hours) the game would have loaded everything necessary, starting a new game would be a matter of seconds. right now one has to start the game, wait for it to load, then click new game (chose all the settings) and then wait again for the game to really start while loading the rest.
Any chance of leaving cached, pre-parsed XML attached to save games? That would speed up their loading, at least.
Alternately, begin caching map data while the user is still fiddling around creating the scenario, etc.
was wondering if it would be possible to to trim what the parser interpets.
for example
<!-- Equipment --> <Equipment>K_Female_Head_02</Equipment>
when the interperter reads the the file it becomes
<Eq p n > K_K_Female_Head_02</Eq p n >
that way the inerperter is only reading a percentage of the file instead of the whole thing .
it may even be possible if the xml schema in this game is not too varied, to maybe even improve on this even more by guessin whats next let say that the Equipment catergory is only ever popultaed with <Equipment> tag then instead of reading equipment over and over again the interperter could just read the catergory title and then setup the entries in the system based on how many x equipments they are .
<!-- Equipment --> <Equipment>K_Female_Head_02</Equipment> <Equipment>Spouse_FemaleBasicHair</Equipment> <Equipment>TightBoots</Equipment> <Equipment>Female_Basic_Pants_ID2</Equipment> <Equipment>Short_Sleave_Merchant_Shirt_ID2</Equipment> <Equipment>Female_Warrior_Vest_ID1</Equipment>
is read as
<!-- Eq p n --> <E> K_Female_Head_02</E> <E>Spouse_FemaleBasicHair</E> <E>TightBoots</E>
i dont know if this is would save time or not .
another thing maybe worth considering is allowing the user to compile the xml into game instructions
the game comes with all the xml files but the fefault is already pre parsed and compiled in to the game so there is no nead to re read all the xml.
now when the gamer adds modified game files they need to press a button somewhere to update and configurethis data set . that would probably be a big time saver .
and not to hard to implement .
just some thoughts .
That's very similar to what we're thinking of doing. What we're looking at is loading the XML the first time and then saving it as a binary blob where it only changes if the checksum changes.
Whatever you do, don't reduce content or cut things out of the game! I'd much rather have more content that loads slower then less content that loads faster. Someone can always "mod away" all the extra crap if he/she wants to streamline the game because it's loading slowly.
And, for the record, I like what you and TheProgress are discussing. Sounds like a solid plan and I, for one, would be willing to have that added wait in the beginning for overall faster loading.
I do not find this quite so problematical; I think it is reasonable to wait for loading, as long as you know when and where. You might even consider making a tiny cut-scene frame while the game parses the tiles, with a little text-reminder to the new player: "Loading Map; this might take a couple of minutes", and the player can go make a cup of coffee etc. I'd rather have it this way than poor frame rates and interim loading during the game.
Generate the binary blob during game installation? Then run the checksum when you load the game to make changes or additions as necessary?
I have to agree with onomastikon, an interesting loading screen or one with some helpful information can make up for longer loading times. The play will forgive a developer if they plan ahead a little and do this. APB, for example, has some LONG loading times but I find the loading "movie" pretty interesting so I can forgive the loadtime.
You guys loadtimes are much less then APB, probably not something to lose too much sleep over.
With Civ 4, it's the same issue with the XML. On my Core 2 Duo, every time I load up FFH2 it takes about 2 minutes to start civ 4 and like 3-4 minutes to generate a game. Loading games in the late game also take eons. It doesn't bother me though.... and load times won't bother me in Elemental... just wait until the mods start coming in...
So please don't remove any good content... unless it's extraneous or something...
Finally some good news, yay for me with my I-7. Make combat great and i'll gladly eat my words. One other thing, will 64 bit OS being officially supported and will Huge maps make it by the first patch?
This sounds similar to how some Diablo 2 modding would work, where running Diablo 2 with -direct -txt would load all the txt files and convert them into bins. It seemed to work very well from what I remember of tooling around with it.
you should enable ways to symlink into different hard drives. I have to symlink it onto my solid state to get decent performance
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"? >
Please unify ' encoding="utf-8" ' at xml source code and parse processing. I thought these are need to display MultiByte Character in localization , i.e Japanases , Chinese and so on ...
Agreed with Onomastikon too.
There's no problem with a game taking time to load if it's justified by a lot of content.
(Of course it bugs Frogboy because he's the guy who bought an Ipad not to wait for his notebook to start )
To play several hours of game, one can wait 1-2 minutes at start.
Why not just create one big pre-parsed xml for non modded versions for the game. Which is bundled with the game and tie it in with 'blob' of xml? Or get the installer to do it during install, iv noticed people dont really mind too long install times.
+1000
Waiting isn't an issue if I have something to do.
This is a CPU bound process, using the CPU for something else is going to make it take even longer (and will hit people with single core CPUs the most as there won't be *any* parsing going on while the CPU is busy animating pong).
Ultimately caching the parsed XML as a binary blob is the best bet. The majority of the time the XML doesn't change from one load to the next. Civ 4 did something similar and the load time gains are substantial.
There are many great features available to you once you register, including:
Sign in or Create Account
