It is difficult to explain the issues of Sins have without going into too much "technical detail" which some may not understand. The main problem is that Sins is an older generation 32 bit game. With a two gigabyte memory limit. This is nothing new. All 32 bit games have a 2 gig memory limit. However Sins is NOT like other 32 bit games. Most games have an ability to load their assets "on the fly", or only as needed. Some games can also unload assets that are not needed. This is how they prevent themselves from reaching the 2 gig limit. Some games have been patched to where they can use up to 4 gigs of ram on 64 bit systems.
Sins also can load only what is needed at game start up, but Sins does NOT unload what it does not use. Any assets that are loaded remain loaded throughout the entire game! If you use all three factions (Six in the case of Rebellion), Then Sins will load ALL of its game assets at start up. Entrenchment, and Diplomacy in some games loaded between 1.7 to 1.9 gigs worth of assets at game start up! Graphic settings, The size of the Map, and the amount of Players, or AI opponents also determine how much ram Sins will use in any given session.In 2006 when Sins was first conceived 32 bit Windows XP was the standard OS of the time. Single core CPU's were also still the standard. Though dual core CPU's were starting to become mainstream, and 2 gigs was considered to be "plenty" of memory. So most game developers at the time didn't have to worry about hitting that dreaded 2 gig brick wall. As time went on Computers, and Sins itself evolved into their present state which is now Rebellion, and soon to be Windows 10 running in multiple core 64 bit systems. With 16+ gigs of ram. Things have changed so much over the years. However Sins still remained the 32 bit game as it was designed in 2006. That uses only a single core of your CPU. With a two gigabyte hard coded memory limit. Sins WILL CRASH when the game reaches that 2 gig limit.There is some misunderstanding about the 2 gig crash issue. The dump happens when Sins reaches 2 gigs of Ram Usage. That means the ram that Sins uses by itself. NOT what your entire operating system, and/or other programs you are running combined uses. Only the ram that Sins uses. No matter how powerful your system is. If it is 32, or 64 bit, or if it is Windows XP, Vista, 7, 8, etc. etc. Sins WILL CRASH when it hits its 2 gig ram limit. It is hard coded into the game.Nobody was aware that this would be a major problem back when the game was first designed, because Sins was designed based on the hardware standards of 2006-2007. Before 64 bit OS's, Quad Core cpu's, and 4+ gig ram systems became the standard that they are now. There is nothing that can be done about it. Except to re-write Sins in 64 bit. Which is out of the question, because it is too cost prohibitive for the developers to do this (Ironclad said so themselves). Sins is a great game, and it is a shame that this is happening. With the Entrenchment, and Diplomacy expansions Ironclad had unintentionally pushed the game over its own hard coded limitations with all of the new added content.
Over the years as of the release of Diplomacy 1.3, and Rebellion. Stardock, and Ironclad did indeed fix all of the major game crashing issues of Sins. Using the fix's from This Project as a guide. However, NONE of these fix's ever made it into Entrenchment, or Original Sins! While the Stardock, and Ironclad developers have fixed most of these issues over years. They can not do anything to change the game itself short of reprogramming Sins over from scratch. The developers can do no more, but there is something that WE can do as modders to prevent Sins from hitting that dreaded 2 gig brick wall, and improve the performance of Sins "just a little bit more". I have taken the initiative, and applied to Sins what i have learned from my Homeworld Modding days. I present to you....
The goal of this project is to improve your Vanilla Sins gaming experience, and to eliminate the 2 gig crash issue. No changes will be made to Original Sins, or any of its Expansions in game play at all. No stat changes will be made. Nothing "new" will be added. There are other mods that can do that. Original Sins will remain Original Sins, and the Expansions will remain as they are. What we are doing is taking what game assets that Sins already has, and we are "optimizing" them to improve the performance, stability, and to moderate the ram usage in game. The side benefit will be that some lower end systems can enjoy almost the same gaming experience as the high end users do (Especially Laptop Users).The project initially started out as an idea to reduce the texture sizes, and poly counts of some models. Not just arbitrary reductions. The goal was to maintain the quality of Sins in game. To make it look as if nothing has been changed at all. We started with optimizations to the Planet's, and Skybox's, because they were the "worst offenders" as far as ram usage, and performance was concerned. Unfortunately my modeling skills left much to be desired in Softimage XSI at the time, and there were un-fixable mesh errors on the optimized planet meshes. Despite that the "proof of concept" was established. A lot of ram was saved, and there was a very noticeable increase in performance.Then we studied the Particle files, and Textures. Many of the particle textures in our opinion were way too high in resolution for the purpose that they served. A reduction in texture resolution by HALF showed no visible difference in game at all! There were some exceptions where the particle resolution reduction difference was very noticeable in game, and those textures were reverted back to their original resolutions. Remember the goal was NOT to reduce in game quality. Also all of the particle textures were changed from TGA format to DDS format, because DDS uses a little less memory, and is much more efficient, because DDS format loads directly into video memory. The particles can also utilize the mip mapping features of DDS format. Where TGA format does not. Therefore enhancing in game video performance (increased FPS). As of version 0.07 no TGA format textures load in TSOP. With the exceptions of the Mouse Cursors, and Scenario Pictures. This is due to those textures being controlled by the hard coded exe. When you change the texture format those textures do not show up in game.The developers mentioned something about a Memory Leak. I heard that a memory leak can be caused by a file searching for something over, and over again if it is not there. Example a mesh file calling for a texture that does not exist. Vanilla Sins had MANY particle, and mesh files that had that issue. We removed all of the entries that pointed to non existent textures in the mesh, and particle files. We also removed the entries that have lines like for example "C;\g\main\datasource\textures\effects\aura4.dds", and just used "aura4.dds". This will default the search to Sins, or the Mods "Textures" folder, and speed things up a bit. Also various typo's were corrected in some mesh, and particle files. Some ships had mesh nulls that were Miss-Labeled. For example the Akkan had nulls, and flairs that were miss-labeled. The engine trails, and flairs didnt show up in game. the Marza was missing its bomb nulls. etc. etc. All of this has been corrected as well. If there is a ships mesh file in the mods "Mesh" folder then it had something wrong with it that was fixed. The Soundata, and Galaxydef files had errors too. Regardless if any of these issues caused any memory leaks, or not these issues MUST be fixed! (and they were as of Diplomacy 1.3, but NOT for Original Sins, or Entrenchment).After the Meshes, Particles, Textures, and the User Interface were optimized, and the very obvious issues were fixed there was a tremendous boost in game performance. Sins ram usage was greatly reduced. As of TSOP 0.07 I could run a game of Entrenchment, or Diplomacy ALL DAY, and I NEVER used more than 1.5 gigs of ram! This was using Maximum Settings. Including Bloom enabled. Tested on Huge Random Multi Star maps with 9 AI's. The 2 gig crash problem has been solved! Our primary goal with this project has been achieved! Without having to use any external memory increasing programs! Stardock, and Ironclad took notice, and they used TSOP as a guide in the Diplomacy 1.3 patch, and for developing Rebellion.Now that the Stardock, and Ironclad have fixed the issues of the game as of Diplomacy 1.3, and Rebellion. TSOP has been focusing more on Original Sins, and Entrenchment. Since the Ironclad/Stardock fixes never made it into Original Sins, or Entrenchment. However, That is not stopping us from optimizing Diplomacy, or Rebellion "just a little bit more" You can also use Large Address Aware to give Sins an extra gig to play with, but in vanilla Original Sins, and Entrenchment the Endless Search Loop issues that were never fixed will cause the game to eventually go over that extra gig (Run vanilla Entrenchment, and watch your ram usage keep going up, and up, and up). TSOP is a solution that will stop that issue.
____________________________________________________________________________________________________________________
Downloads
The Sins Optimization Project v1.0
For Original Sins 1.195
For Entrenchment 1.055
For Diplomacy 1.37
Stardock, and Ironclad fixed all of the issues that compelled us to make TSOP. Therefore there is no TSOP for Diplomacy
For Rebellion 1.82
As with Diplomacy, Stardock, and Ironclad fixed all of the issues that TSOP pointed out, and then some in the case of Rebellion. Sins: Rebellion is as optimized as any Sins game is going to get.
NOTE: TSOP 0.07 is still hosted on ModDB, and it still works in the current versions of Trinity (OS, E, and D), but you will be playing as if Sins was patched to an older version of the game. Plus some Icons and Buttons wont be visible.
Changes in TSOP 1.0
TSOP 1.0 updated to the latest versions of Original Sins, and Entrenchment.
The User Interface is completely overhauled to use pure DDS format.
Trade Ships, Strike Craft, Mines, and Constructor Unit Meshes, and Textures are Optimized.
Starbase Textures are optimized mimicking the Stardock Rebellion Optimizations.
Sounds, and Music are optimized.
All TGA format Textures are replaced with DDS format, and TSOP "force" loads them. The only TGA textures that remain are the Mouse Cursers, and Scenario Pictures. This is due to hard code in the EXE.
"Strategic Texture Reductions" for the Particles return using newer methods to create a better looking texture.
(Original Sins, and Entrenchment) Many Mesh Null issues, and various File Errors were corrected, and Unnecessary Text Entry's were removed in both the Mesh, and Particle files. Entry's pointing to Non Existent Textures in both the mesh, and particle files removed. See the various "Fixed Files" texts in the mods Read Me for details.
(Original Sins, and Entrenchment) The Diplomacy 1.3 graphic fix's have been incorporated into the Original Sins, and Entrenchment versions of TSOP (Pipeline Effects, "White Line" error fix, and Corrected Mesh files).
TSOP 1.0 for Trinity will see the return of Optimized Planets using the planet meshes from Rebellion! but with our Optimized Original Sins Textures!
There will be no version of TSOP made for Diplomacy, or Rebellion, because there is no need for it. Only Original Sins, and Entrenchment need TSOP, because the 2 gig issue, and various other issues were never fixed in those two versions of the game.
This is the FINAL version of TSOP that i will make. If the community wants to take over the project they are more than welcome to.
Community input, and contributions are more than welcome, and they are encouraged! This is a Community Project! All can pitch in!
Note: TSOP does not fix the known issue of "Late Game Lag". It does help prolong the lag until further into the game in some cases, but sooner, or later the lag will happen. The late game lag issue is a CPU issue due to the fact that Sins (including Rebellion) is optimized to use only single core processors. Therefore Sins will use only one core in multi core CPU's. To help with the late game lag issue we recommend the following:
Playing on smaller maps.
Using fewer AI opponents.
Disabling the Trade Ship icons (this actually helps out a lot)
Stacking the icons in the empire tree (same as with disabling trade ship icons the less icons rendered the better).
Using a Strike Craft Reduction mod (The many strike craft in the game are a big CPU hog. There are some good strike craft mods that reduce the number, but maintain the balance. Search for them).
You do not need to download all versions of TSOP. You only need to download the version of TSOP for the specific version of Sins you wish to play. The Retail version of Trinity is just all three versions of Sins bundled into one package. Installation for Trinity versions of Sins are no different than if you purchased each version of Sins separately.
_____________________________________________________________________________________________________________________
Other mods are more than welcome to use TSOP. No questions asked! In fact we encourage it! Learn from it! Use it as guide, and an example! Merge it into your projects!
TSOP is intended to be a Stand Alone mod. Run with no other mods enabled. There are issues enabling TSOP with other mods. Enabling with other "Graphic Enhancement" mods is NOT recommended, because the graphic enhancement mod will defeat the entire purpose of TSOP. All of our optimized files will be overridden by that enhancement mods "enhanced" files. Increasing your ram usage instead of decreasing it. Some enhancement mods like Distant Stars are incorporating elements of TSOP into their future updates.
Read the included READ ME text's for installation instructions.
I agree that the small team coloured icons (fleet tree and zoomed out icons) need to be 100% but IMO all else can be DDS.
Problem is that dds decompression takes place in hardware, which mean the game engine can store the compressed files in memory, whereas there is no hardware decompression for RLE TGA's (or jpeg, png or any other such format), so upon loading (if supported at all) the files would be decompressed and the size i memory would equal the uncompressed TGA files sizes.
Does anyone know if the Sins engine decompresses the UI elements in memory when accessed by positional brushes. If so, there is not any advantage to compressing the source UI images with DDS when it comes to memory usage.
I use this tool to compress now which results in a better quality. Gimp/Nvidia DDS plugin produces too many artifacts for me. The batch compress is nice as well.
Is there any downside to mixing texture resolutions on a planet or any mesh for that matter.
In this example the Earth light side is 2048x1024 and the earth dark side mask is 1024x512...
Other than a difference in quality, nope.
The planet is dark anyways ...
Nice, filling the Great Lakes with oil.
Ya, Obama finally authorized drilling in the upper east. Canada and the US are fighting over the reserves now.
It is advisable to keep the same resolutions just to ease burdens on hardware, and prevent possible annoying dev.exe errors (i have had some freaky errors using mixed res before). Sins doesnt like mixed res. Back when the dev.exe used to texture check we would get pop ups left and right for different sized texture, data, and normal maps.
Also if you use a reduced res you also take the chance of pixalation in the texture. depending on how low res that texture is.
I can see why you want to mix the res's up (especially for data textures), but again not advisable for the above reasons.
Yes they are throwing pink ribbons accross the borders at each other , and saying 'go away before i insult you a second time"
Thank you, I wondered if there was some issue like this as I hadn't seen it done previously.
First, it work with RLE TGA, i have test it...
Second, you need to understand how everything is processed...
First, the game load all the texture in the main memory... they are simply cached there ( faster access that on the hard drive ), having the same size that on the drive...
Once one texture is needed, the directx D3DX image file loader kick in... The D3DX image file loader functions ( D3DXCreateTextureFromFiles ) support BMP, TGA, PNG, JPG, DIB, PPM, and DDS files... the image loader simply process your texture to be send to the graphic card, using any hardware acceleration if enabled...
Once the texture reach the graphic card, the texture is in a format that the graphic card will accept...
TGA is no more supported with directx10... with directx10, you need to include code in your software for first convert the TGA... but sins is a directx9 game and have no problem with it...
So texture are only decompressed/processed when needed and send to the graphic card ( shader code use them )... what use a lot of memory is the texture cache in the main memory, who can enjoy to be converter from tga to rle tga...
On the graphic card level, when a texture is not more needed for the scene render, it is simply "released"...
Point is that in case a convertion to dds create too much artifact, it is always possible to use rle tga who spare some memory and keep quality at 100% ... with the compressonator, you are able to see the quality difference and decide if your convert to dds or not...
I would just as soon convert to DDS just to keep the standard and to use the mip mapping functions for everything except the UI. TGA doesnt have mip maps so will stay the same resolution no matter the zoom, or graphics settings. The mip maps DO help the game perform better even if the memory savings is minimal.
Thank you Thoumsin. So converting to DDS does help the memory situation through cache. The question is can the community live with "some" detail loss in the textures? Even if we try to keep it as minimal as possible? Is the detail loss in TSOP "that" noticeable to the average person?
I noticed no one really said a word about the TSOP fighters detail. I reduced the texture sizes of all the fighters in TSOP from 512 to 256 res. No one noticed. I reduced the Starbase texture sizes from a ridiculous 4096 to 2048. No one said anything about them ether. Same with the Trade Ships, and Constructors. I reduced the resolution of all but a few particle textures by 1/2 in TSOP. Nobody said a word. Nor did anyone say anything negative about the greatly reduced polys on the meshes SivCorp did as well. So that tells me that Yes, some detail loss is acceptable as long as it is not "Noticeable" to the common gamer.
The only negative i remember about TSOP was the early attempts to optimize the planets and skybox's which i will admit was an abismal failure.
I personally think DDS is better for 0.08, but keep the original texture resolutions (have a 1/2 res add on for those with net books or really low end systems). I dont know why IC mixed TGA, and DDS up in the same game. Most other games standardize their textures ether using strictly DDS or TGA, but not both. That is what TSOP tried to do as well. To standardize the texture system.
I think the pro's of DDS outweigh the cons. So TSOP, and TSCBFP will use only DDS format.
Yes, it is in fact, the only place where it help... a lot !!!
TGA processing is made by directx... in case of DDS,texture is load by directx and send for processing to the graphic card... it mean fewer resource needed by the processor... something needed for sins who is single core and need all the processor resource who can be spared... so a all DDS will speed up the game... no idea if it will help the mid game lag...
About GUI element, it is possible to make DDS without mipmapping :
- TGA 2048x2048 : 16 mb
- TGA 2048x2048 rle : variable compression ( very high compression in cartoon style picture ), quality 100%
- DDS DXT5 mipmap 2048x2048 : 5.3 mb ( artifact with normal map, hand drawn cartoon like picture, picture with a lot of red color, sky picture, light map, etc... )
- DDS DXT5 no mipmap : 4mb ( artifact with normal map, hand drawn cartoon like picture, picture with a lot of red color, sky picture, light map, etc... )
With DXT5, 16 pixels of 32 bits ( RGBA 8888 ) are compressed in one packet of 128 bits... where 64 bits are for the alpha channel and 64 bits for the RGB channel ( 2 x RGB 565 + 4x4 2bits loockup table )...
In case of normal map, the game use dxtn who move the red channel to the alpha channel for reduce the artifact problem...
Mipmapping increase the size of the texture ( 30% when compare to a dds without mipmap ), so it increase the memory cache use... but it speed up the work of the graphic card... today card have hardware able to resample texture but it use resource from the graphic card and lower the frame rate...
TSOP have not a easy job... find the right balance between memory use, main processor use and GPU use... for the actual TGA, some will be good with a DDS format without mipmap, some not... for these who have too much artifact, RLE TGA is a possible alternative... it is why tool like the compressonator help a lot since you can see the difference between the two picture ( in some case, need to amplify the difference windows by 3200% for begin see them )...
Well, already reply to these... for the delicate balance between memory use, CPU use, GPU use... sins was made ( several year ago ) for run on low end single core 32 bit computer using the old directx9 ( last directx version allow multi thread/core render !!! )...
Well, it is perfectly possible to reduce a poly count from a model without changing the shape of the model... usually, when i receive a model from someone, i can reduce the poly count by 30 to 50%... not true for all modelers, there is some here who already know how to optimize a model's poly...
For texture, same thing... it is possible to reduce the texture size without loose quality... mainly a UV work... lot of the texture from sins have large area of unused texture surface... some UV puzzle work will allow to use lower resolution texture... or if resolution is keep, to improve the quality ( it is how the B5 mod work, bigger surface for paint because of optimization lead to high quality )...
Back side is that these two job, model optimization and UV optimization ask a lot of time... and sins have hundred of model !!!
Anyway, a lot of thing is possible and until now, i have mainly read comment of people who are really happy with TSOP... so, it can only mean that you have make a very good job... more and more mods are using your optimization too, it mean for them more new model, more new texture, new races, etc...
Some mod was crashing with the very high graphic setting due to the 2gb thing... with TSOP, mod can run again in very high... so, a light lost on quality on a few texture have lead to a general on screen improvement...
Foundation is laid for TSOP 0.08beta it will be a release candidate for TSCBFP, because if all goes well then i will make the core game replacement files (converted to binary as well).
I need people to test 0.08 once released until their eyes bleed. Test ONLY TSOP. Not tsop with other mods enabled on top of it. We need to know if any problems exist with TSOP so we can fix it (if we can). TSOP is NOT meant to be run with other mods.
TSCBFP will be fully mod compatible, because you are replacing your core game files. TSCBFP will also include the text reference files so others can use, and study for their mod projects. Along with an itemized list of what exactly we changed, and why we changed it.
While TSOP focus's on the Sins engine issues. TSCBFP "may" include game play, and balance fix's. I will not do this myself. I need the rest of the community to do this along with me, and participate. After all it is a community bug fix patch is it not?
I will start a new topic for the bug fix patch once 0.08 is ready to launch.
Partially, yes. Once called by the DirectX function, the TGA is decompressed and handed over to the graphics card video memory *uncompressed* whereas the dds stays compressed in the graphics memory because it can be decompressed on the fly using the graphics hardware. (Corrections to mistakes of mine are welcome)
Just ran an interesting game. Experimental 0.08 TSOP on Original Sins. With full resolution DDS textures instead of the prior versions 1/2 scale textures. NO CHANGE in ram usage between the two versions. If there was a change it was only a slight increase that was so slight i didnt notice it at all. Original Sins still runs at 1.2 gigs on max settings even with vanilla resolution DDS's. (this is before i am incorporating Ironclads Diplomacy pipeline effects fix to Original Sins). Plus the mip maps in the particles are doing exactly as i thought they would. So that test was successful i think. You ask why i am testing on original sins.. cause i dont want to screw up entrenchment, and diplomacy just yet
EDIT: Incorporated the Diplomacy 1.2 pipeline effect changes into Original Sins, and it works! Not much of a ram improvement. I guess IC fixed that in the exe, but the white dots issue is fixed. Original Sins looks MUCH better. All i did was copy the files in the pipeline effects folder from diplomacy 1.2 into TSOPs pipeline effects folder overwriting the original files. Same can be done for the Entrenchment version of TSOP.
Is 0.8 beta out right now? I use TSOP all the time unless I am tweaking a mod. I'd love to run it. Happy to leave off other mods to contribute--fyi...please make an option to turn off the star effects when zoomed out, darn it! Absolutely essential for big maps.
I'd uninstalled TSOP on the last Sins beta build so as not to negate/overlap developers changes to the core game.
Well I hope you get to feeling better. I've had more time this past week 'cause I popped a bone in my foot slightly out of place (childhood injury). I feel some of your pain.
Kudos for making MSGM inthe first place and I think it's a valid argument to say it is a performance enhancing modification--though it does change the game display so it should be an option.
Having it incorporated into TSOP would be great though obviously rebellion might differ.
I completed the planets and sent them your way.
I need to finish the Entrenchment, and Diplomacy builds before i can release anything. I need to make sure everything works on my end with no dev errors before i can upload.
Should have something by this weekend, but no promises.
Good to to you back Stress.
0.08 for Entrenchment just finished, and first test with the diplomacy 1.2 pipeline effect fix, and full res textures results in ram holding so far at 1.5 gigs. 2 hours into the game. i am just watching 3 AI's fight. waiting for dev.exe errors to pop up.
There are many great features available to you once you register, including:
Sign in or Create Account