In previous journals we’ve discussed the intention to create a modding layer that would use Python as its scripting language. Python is slower than Lua in terms of interpreted languages but there’s nothing stopping us from having C++ source that people could use to in addition or instead of a scripting language for performance sensitive elements of a game.
With that said, let’s hear from those familiar with both on what they think of either one and why.
I do this stuff for a living... In general the argument that python is slower is False. The question is which python are you using.. Cpython, Stackless, IronPython, jython etc...
The main issue to consider is the use of threads. Cpython does not do this well, for historical reason. Lua and the other Python form are better at this.
C++ is fine as well, as for windows the c++ compilers are free to use and download. However C++ requires a little more skill to use, depending on how well you define you API.
I'd say Python speed concerns can be mitigated by choosing the fastest implementation for each problem domain but currently even PyPy is only close to equal to Lua. That might well change in the future and then it has to catch up to LuaJIT.
While there are C++ compilers for Windows it represents a possible failure point unless the game makers either bundles or anoints a blessed distribution due to quirks and implementation decisions made in the compilers.
As Stardock uses Visual Studio and its compiler, any C or C++ compiler that target Windows should do. They would have to build programs in such a way that they can interact with the WNIAPI correctly (or provide a way to make it do so; in which case the user would need to look it up), which means they will not differ from Microsoft's compiler in anything needed for that.
This means that for compatible compilers virtual function calls (used by COM), functions using the same calling convention as the WINAPI (virtual functions will also need to use this), and primative types should be fine. Accessing an object's data directly may or may not work; I think the WINAPI headers use special commands to ensure that they use the correct layout in memory. There is no portable way of doing this yet (or at least not until C++0x is supported; I think it provides a way).
(for that matter Visual Studio Express is free, though the IDE* stops working after a while unless you register it [requiring an account but otherwise free]; however, the command line tools will still work)
* The only thing the IDE is good for is debugging. Other than that you are probably better off with Notepad if you do not have a problem with using command line tools.
Lets be clear about this: Lua does not support multithreading. It only has coroutines. "Lua does not support true multithreading, that is, preemptive threads sharing memory" p.281 Programming in Lua, 2nd Ed, Roberto Ierusalimschy.
You can however create multiple interpreters. Then you just need one for each thread (as previously discussed, it is unlike the scripts themselves will deal with multithreading), though depending on how that works out with sharing the script-code it may have too big of a memory footprint.
The scripting language does not necessarily have to support multiple threads. In general, you probably would not even need multiple interpreters, though I am certain using only one would have significant speed penalties to your systems. For a multithreaded system like Elemental, one could create a single Lua Interpreter and wrap in a semaphore or some other control code. Thus, each thread attempting to access the interpreter would be blocked if the interpreter was already being accessed and would become unblocked when the interpreter was freed. In a larger scale application like Elemental, you would probably want to use multiple interpreters, which use the same principle as the single interpreter system, to allow the various threads to access the scripting sections they require without constantly blocking other threads from accessing completely different script sections. This would also keep memory requirements low as each section could be encoded in such a way as to only load the necessary chunks for its section with only minimal overlap in cases where modders themselves have created significant overlaps, which depending on the separation would probably be unlikely.
When it comes to multi threading in Lua there are more than a few options. The LuaLanes site offers a comparison of some different implementations. The design space is from green threads to native threads, from 1:1 over N:1 to N:M mapping to native threads, 'occasional' mulithreading to pervasive and message passing.
One plus for scripting in Lua and Python contra C++(can be done but not a walk in the park) is the possibility of interactive development, testing and debugging by having a scripting console open while running a script. For one example of a Lua implementation see ToME T-Engine 4.
That style of development has always been a favourite since way back when I discovered Lisp and Forth. Many modern development and testing techniques are a subset of this.
Actually, that's wrong. Lua has a much smaller footprint than Python, which is why its faster. Anangara points out the PyPy is the only implementation of Python that comes close to Lua, which is correct.
If we were arguing which language to use on a non-embedded project, I'd vote Python. But Lua wins in the embedded scripting game.
There is also one elephant in the room of scripting languages that would place in my top three right between Lua and Python. JavaScript has three major implementations with major support and performance as well as a host of other implementations. JaegerMonkey, v8 and JavaScriptCore. It could quite possibly mean that you could use the browser as an IDE.
Lua over python for 2 reasons:
Performance. Lua is way faster and manages memory better.
Ease of integration. Lua is designed to be embedded. Python is designed to be extended. That is, lua is expected to be called from C (or C++) whereas python is not. Python is expected to call C++, not the other way round, and I don't think using boost:: python for instance is as simple as integrating lua.
Such tools exist for C++, but are hard to find. I know of one but it was a demo that was with a tutorial on using a specific library to make such a tool. I recall it having issues when trying to use it with the Standard Library that goes with said library.
http://www.codeproject.com/KB/cpp/Personal_C___Compiler.aspx
I'd love to be able to use Python. It's an easy to use language and it would really motivate me to get my hands dirty with modding (don't know much about lua, but I suppose the same argument applies).
Having to learn C++ in order to start modding would likely discourage me... I don't think it should be the primary language for modding. At least if you're trying to actually get as many ideas as possible flowing.
Python dictates (ie: forces you to use) a specific object-oriented model, whereas Lua doesn't. Some projects are better written procedurally, rather than using objects - and vice-versa. Lua gives you the power to do both.
I have to agree that Object oriented structure is very hard to grasp, especially when you are not the designer of that structure. So you first need to learn the language and then the structure. So it doubles the learning curve. I would only add the object structure when making 3rd level modding in c++.
In a perfect world you would put ole Jon to work developing an in house scripting language for kumquat games thats designed with the idea of being as simple as possible for non programmers. Designed with mid level functionality but maximum simplicity.
It reminds me of Hexen which had it's own scripting language which was similar to C. You could call all game effects like a function and pass parameters. That was very cool to use. Still I think creating your own language could simply make you waste time reinventing the wheel.
PS: I hate the quoting system - and the forum needs a preview
Well, at least your quotes appear in a box, mine does not
As Stardock uses Visual Studio and its compiler, any C or C++ compiler that target Windows should do.
When I was programming Wizardry, I was using Gcc for windows. I think it's called dev-cpp. The only thing I hope is that Elemental is not compiler dependent.
Lets be clear about this: Lua does not support multithreading.
Why do we need multi-threading? If it's multi-threading within a script, forget it. People does not need to make script that creates thread. If 2 different script cannot be run at the same time, then maybe it could e useful to get multi-threading. But again, it depends where the scripts is hooked in the game. Besides for AI, I do not see why the game would need to run another script while the game is waiting for the player in the first script. Because most of these scripts purpose is to change the rules of the game, not an animation in the background for example.
So after reading the various comments, I think my vote goes to Angel Script first and then Lua.
Ive done extensive work with LUA
- Did the construction AI for Hearts of Iron 3 Sempter FI- Wrote all of the LUA in retrospect for the Hearts of Iron 3 Sempter FI expansion in regards to AI abilities
One side is yes LUA did provide allot of apportunities for me to enhance the engine and overall I did enjoy it over the old Paradox script/event system they had in place from Hearts of Iron 2 (their Europa Engine). But overall LUA is not capable of handling heavy AI loading algorythms and all of that code was never farmed out due to performance. What you could is simply have a seperate DLL that can be compiled in any language which expects certain methods to be located in it and have class variables passed to it that contain data from the main EXE that can be processed. The code for the initial work on that DLL can be made open source but since it is a DLL in itself as long as there is documentation on the method being called and what is expected in return the DLL itself can be written in any language (even Visual Basic). You could even go as far as having a central DLL premade that can pass these parms to a text based language such as LUA. This way you can get the best of both worlds out of the modding community and yourselves.
When I was programming Wizardry, I was using Gcc for windows. I think it's called dev-cpp. The only thing I hope is that Elemental is not compiler dependent.*snip*
Why do we need multi-threading? If it's multi-threading within a script, forget it. People does not need to make script that creates thread. If 2 different script cannot be run at the same time, then maybe it could e useful to get multi-threading. But again, it depends where the scripts is hooked in the game. Besides for AI, I do not see why the game would need to run another script while the game is waiting for the player in the first script. Because most of these scripts purpose is to change the rules of the game, not an animation in the background for example. So after reading the various comments, I think my vote goes to Angel Script first and then Lua.
The langauges listed so far at most support OOP, but none of them require it.
The only things that would be compiler dependant would be those that are not not spec-ed by the standard. And as I said most of those things will be eliminated if they stick to things that must be a certain way for the WINAPI to work.
The multi-threading support in scripts does not matter, only the ability of the intepreter to not be a proformance bottle-neck or memory hog when used in a multi-threaded enviorment. As has been pointed out, some implementations do better than others.
Good.
-------------
Since some of these langauges have diffrent implementation, prehaps those that are interested is having certain langauges should look for which one(s)* are the best and then we can compare. Then we can find out which is best in various areas on a techinical level and let Stardock figure out what is most important.
* If there is no absolute best, then we can see how the better implementation stack up against the other interperters.
Some things that can be compared (if anything is missing, please mention it):
- How much extra space does the interpreter take?
- How fast does it execute scripts?
- How well does it interace with native code? (How much overhead does it add to calling or being called a native function?)
- Memory Management (Does it have ref-counting or pure garabage collection? Can it deal with circular references? Can custom memory allocation/frreeing functions be set (may or may not be important, some games have special allocation schemes to make it more effecient)?)
- JIT Compiler Support (Does it include one? Can JIT support be added without changing the source code?)
- Multi-threading (Does it allow multiple threads to invoke scripts at the same time and without having heavy a overhead cost to memory or proformance? The ability to let scripts create and/or manage threads is a non-issue. People who are using the scripts as an easier way to mod things should not ever deal with this.)
IMO an 'objective' look of scripting languages would show AngelScript, JavaScript and Lua as being far ahead of the alternatives on most measures, excepting C++ as not strictly being a scripting language. All these are made to be called and call native code and themselves. One other important factor is who Stardock can afford/get to do the actual integration of the codebase.
If someone like Mike Pall creator of LuaJIT (both an efficient extremely fast Tracing JIT and the fastest Lua interpreter) could be bothered to help with Lua integration in some capacity or David Mandelin doing the same for JaegerMonkey then that language is probably the best option. Someone similar probably exists for AngelScript too.
I'm for C++, and/or a scripting language with similar syntax (angelscript, javascript)
I despise codestructures that base their interpretation on the number of tabs at the start of each line
Select a C-style scripting language
You'll pull more of the professional and semiprofessional modders, and they're what make the modding community. We make the base packages everyone uses, write the tutorials that introduce new players, answer the questions they have, document the quirks of the toolset and form the foundations of the scripting community.
Provide a robust API using triggering events
onspawn, onopen, onclose, onplayerjoin, onturnend, onplayerchat
This style is familiar to many hobbyist modders and website developers. They're familiar with the idea from working on websites where you have these triggering events, and is easy to manage with small projects.
Document the API before release date
Robust documentation gives both Stardock developers and players the ability to get their vision working faster. The quicker modders can produce professional results, the quicker they will draw in other modders, before the buzz on the game dies down. The more modders, the more players will play mods. More players meaning more mods means the game will continue to sell post release date, after the buzz has died down; and if they are still playing by the time you release an expansion pack, the more of them will be willing to by it.
Provide an IDE built-into the toolset with
The learning process for new scripters looks like this:1) Try 2) Fail3) Understand4) Try again
If they are getting stuck on Step 3, they won't learn. Being shown why they failed (script tried to use an int on this string operation! string not closed! etc) and being shown how to do it correctly in the same toolset will accelerate their learning process.
However knowing the best implementations of the other, I can see about comparing them to AngelScript. The only comparison I could find without doing an extreme amount of searching was done several versions back. It tested version 2.16.0, and the current version is 2.20.1. Link: http://codeplea.com/game-scripting-languages
A few notes about it:
- In the current version, the application defines its own 'native' string type for AngelScript to use, so the preformance on that will vary.
- The size of the interperter when I build it as a DLL is less than half the size it was for that test. The file size is 512 KB, though that is including padding that is not part of the code. (Static linking will make it even smaller, as some redundant code will be eliminated -- e.g. the C Runtime)
Who Stardock can get to integration is not something that can be tested for the sake of moving this discussion along, at least when it comes to non-subjective things.
GCC for Windows is called MinGW. Dev-cpp, or rather Dev-C++, is an IDE using MinGW
Unfortunately, you can't mix C++ code from different compilers, see the description in the MinGW wiki. Only plain C is easy.
Wrong priority
Getting this API right is much more important.
I'd like to add "before_attack(attacker, defender)" and "after_attack" to the above list to point out that the event should be able to fully replace the game's attack handling code (before) as well as modify it (either before or after).
The same choices should also be available to a modmod - replace or keep and modifiy the "original" mod.
This will be an ongoing effort.
I'd like to ask for an overview documentation (maybe with a flow diagram of turn processing) that shows the different events and when and in what order they are called.
Additionally, there should be a strong collection of examples for common tasks - part of this can come from modders, but Stardock should pull commonly used examples into the official documentation.
While an IDE would be nice, it also is a huge amount of effort.
I'd settle for good error messages - both from the script interpreter and from the code around the events.
The code calling the event handlers must be ready for errors in the script and should also sanity check the values returned by the event.
That is object files, and it has to do with the name mangling. It does not matter as much for DLLs which should be using a more standard calling convention (namely that of the WINAPI). There maybe some incompatabilities, but there must be a way around them if the compiler is to make binaries that work with the WINAPI. This does put some contrains on the binaries that would not be needed otherwise, but assuming a will designed API it will not be a worry.
And an external tool to build scripts, so that people do not have to start Elemental to see if the scripts build correctly.
+1.
Where I work they built us an IDE and we have very bad error messages. I'd throw the IDE out the window to get understandable error messages any day.
This is usually impossible because the API changes pretty late in the game. Javadoc-like comments can help a lot though.
There are many great features available to you once you register, including:
Sign in or Create Account
