Level Editors and Game Engines, how do they store information?

mdeforge

Hey guys,

I've started to create a small game engine with XNA and was wondering if any of you had any knowledge of how information from a level editor is stored for a game engine to read. I figure since a lot of you are in the industry somewhere and work with various engines, you could offer me some insight into how that carries over into something packaged for the game.

I've been playing a lot of FFXIII lately for inspiration on a watered down (really watered down) version of the Crystal Tools engine (Previously known as the White Engine). Square Enix must really guard their info, because I can't find any details on it. Regardless, I'm curious to know how any game engine stores the data that will then be loaded on command inside the engine.

For example:



Here we have a modular map system similar to how I want to handle location in my game. The engine loads in the map. The map contains:

Heightmap
All sorts of different Mapping/Texturing
Object Positions and Rotations
Sounds
Scripted Events
etc.

Obviously games reuse items, so it wouldn't make sense to package everything the level editor uses into something exclusive. That leads me to think that information about the level is simply saved in some sort of proprietary text format that the engine knows how to reads. Am I on the right track here?

SpeCter

I would say you are right on track there. Most things would be stored in a "text" file, where things like position,rotation is stored and the engine parses this things.Something like this:

"model.(add your format here)" "0 0 0" "0 0 0"
and some other things like external scripts for example waypoints(position, which model uses them and so an)

Eric Chadwick

Yeah, often a level editor will store all that data in XML files.

These can be edited by hand if need be, but the level editor needs to be robust if people do that, because people often make mistakes or change the data around in ways that the level editor doesn't understand (but the game itself does). Hand-editing is very useful for tasks like global search/replace.

If a less-robust level editor tries to load a malformed XML file, often it will choke on the file. Or it will toss out the hand-edited data, so it has to be done all over again.

So it's best to either edit everything in the level editor itself (which can be more tedious sometimes than hand-editing), or it's best to make a more robust XML loader for the editor to accomodate hand-editing.

Anyhow... usually these text-format XML files are too slow to run in the actual game. So they're preprocessed into binary format whenever someone kicks off a "build" of the game code (a playable version of the game). The text XML files are an intermediary format, only used for editing, or at least they should be.

mdeforge

EricChadwick wrote: »

Yeah, often a level editor will store all that data in XML files.

These can be edited by hand if need be, but the level editor needs to be robust if people do that, because people often make mistakes or change the data around in ways that the level editor doesn't understand (but the game itself does). Hand-editing is very useful for tasks like global search/replace.

If a less-robust level editor tries to load a malformed XML file, often it will choke on the file. Or it will toss out the hand-edited data, so it has to be done all over again.

So it's best to either edit everything in the level editor itself (which can be more tedious sometimes than hand-editing), or it's best to make a more robust XML loader for the editor to accomodate hand-editing.

Anyhow... usually these text-format XML files are too slow to run in the actual game. So they're preprocessed into binary format whenever someone kicks off a "build" of the game code (a playable version of the game). The text XML files are an intermediary format, only used for editing, or at least they should be.

Excellent information. Thank you!

Eric Chadwick

You're welcome.

Also there's the issue of multiple people editing the same file at the same time.

Like a designer adjusting object placement in the level, while an artist is adjusting collision mesh attributes, and a programmer is implementing a new camera feature. They all need to touch the main level file to do their jobs.

Source control helps solve this problem (Perforce, Alienbrain, CVS, etc.), because each person has to submit the file when they're done. At that point the software warns you that someone else has since edited the file, so you have to merge your changes with theirs.

With text files, it can easily figure out the differences, so you can insert just the things YOU changed but not overwrite anyone else's changes. Binary files make this kind of comparison impossible (or very nearly so).

Kind of a weird concept, but super-important when working on a team.

renderhjs

yup more and more tools these days are going XML for obvious reasons. In the past though and still many engines and tools with a long history have usually a binary storage based architecture - and there are valid reasons for that too.

Pro XML/ ASCI

1. Lets you merge / share/ create revisions of the levels / data using SVN and alike tools. It's the reason why the new Office from MS went XML and the new Adobe Flash CS5 has a open XML structure because of this SVN and revision stuff.
2. Easy to manually edit and write additional batch scripts or tools to fix or convert files. For example say you'd wanted to swap texture paths or ID's of certain emitters it would be fairly easy to write a small script for that. Productivity is higher and it enables others to get into the pipeline as well.
3. Enables more platform independent development and by platform I don't mean just a OS, hardware and alike but also engine, editor or other tool. I mean thats what Autodesk is targeting for with their FBX format - if you want maximum compatibility you save the FBX as a XML version and not the binary version.

Pro Binary

1. Night and day difference when it comes to loading and parsing speeds compared to the XML or ASCI based formats.
2. If your engine and editor are rather close source wise its easier to save a snapshot of a class into a binary dump and use that as a Map format. It means that the current array and variable states of a class in the engine are stored the way they are at that moment. You can load that snapshot back later and restore that state.
   It's kinda like many emulators that allow you to save at any state and create a snapshot.
3. Smaller filesize, just like reading and parsing speed this is often a essential factor when you develop for platforms with a limited storage space or limited drive speed / processing power,...

It usually means some extra work but if your project is big enough (involvement of several people, time wise) I would go for a XML route and a Binary one combined. Whenever you create and edit levels stuff gets stored as XML, and as soon as you create the target build or nightly build generate binary files for a faster streaming, parsing loading etc.

Oh and if you are developing for online games or games that connect online there are some nifty server XML compressions available (unity, flash, php,...) that let you speed up the loading times - obviously with a hard drive that is not needed but with limited bandwidth that could be a interesting thing to look out for.

mdeforge

Thank you for the incredible information! Interesting note about FBX as well. I didn't know that before. A hybrid approach definitely seems best for me. Thanks again for the great info. This should keep me busy for a while

throttlekitty

If you're curious what some portions look like 'behind the scenes', some editors display more information than others.
http://niftools.sourceforge.net/wiki/NifTools
NifSkope (NetImmerse/Gamebryo .nif) for example, was developed to be a sort of 3d hacking tool for the format, it may give you some insight to certain types of structures.