View Issue Details

IDProjectCategoryLast Update
0025626AI War 2Crash/ExceptionOct 16, 2021 12:28 pm
ReporterEluthena Assigned Tox4000Bughunter  
Severitymajor 
Status resolvedResolutionfixed 
Product VersionBeta 3.705 Blazing Collections And The New Spire 
Fixed in VersionBeta 3.709 Upgrades And Serialization Fix 
Summary0025626: Save-corrupting glitch relating to GameEntity_Squad deserialization
DescriptionI created a test campaign to test out a possible achievement bug that I had encountered a while back, but before I got to test it I encountered a glitch that appears to corrupt saves in the campaign. Note, I did restart the app numerous times while testing this to try and flush things out completely, it didn't seem to help.
- "Left off.save" is one of the corrupted saves
- "First Cap.save" is the uncorrupted save for reproducing this bug
- "Campaign folder" is the whole campaign, just in case it's useful
- "LastLobbySettings_usedforcampaign.save" is the lobby settings I used to create this campaign

Loading from the save "First Cap", if you let the game run for a bit (at least 1.5 minutes) without doing anything at all, then save, the resulting save file will be corrupt/unable to load. The exact steps I followed to reproduce this bug are listed below, along with the error message.
NOTE: Originally I wrote the line above to give a more specific time, but where the cutoff point for when the corruption happens seems to shift around a bit. When I started testing this, it VERY consistently happened to saves made any time after the game time reached 10m 33s, but after testing for a while longer the corruption cutoff point started happening later.

Increasing the sim speed may have affected when/if the corruption occurred, so make sure to test it without increasing the sim speed.

Steps:
1) Start up the application/game.
2) Load the save "First Cap" (if loading from the campaign, the save's game time is 9m 23s and it is near the bottom if sorting by date).
3) Unpause the sim (I used spacebar)
4) Wait until the game time reaches 11 minutes or so (time may vary); don't do anything while waiting, just let things run.
5) Open the Esc menu (I have "Pause game when opening esc menu in SP" set to on), give the save a name, then save and quit to main menu
6) Try to load the save you just made. If the glitch happened, the below message should pop up:
---
Errors Since Start: 2 (sometimes it says 3, depends on the save)
[Datestamp of load attempt] The savegame you are trying to load had the following fatal error:
GameEntity_Squad deserialization error at stage 19000 from serialized version 3.705 loading into new version 3.705, error: System.Exception: Could not read string, but did get this partial one: [seemingly random string unique to each corrupted save]
---

Other Notes:
- I did use the command "Forget the past" right away after starting this campaign, no other commands were used.
- I had the sim speed maxed out at 5x for a while, I cant remember if I started doing that before or after creating the "First Cap" save, which was the only uncorrupted save of the ones I made before noticing this save bug.
- At some point before I noticed the save bug (pretty sure it started sometime after the "First Cap" save was made anyway, but not 100% sure so I'm mentioning it just in case) there was a "slow background thread" message in the upper right corner. I cant remember what the name was, something like "persecondPlanningBlahBlah_BLAH" -- it appeared to be stuck/locked-up, as the message never disappeared until I exited to the main menu (so it was there for 2-3 minutes of play).
- During a couple of later tests (after a bunch of "load->save->exit to main menu" cycles), I also got the "slow background threads" message again a couple times in a row -- same deal as before, they didn't disappear until I exited to main menu. In the first one, the text was "LRP_8 15.0s" (the time was increasing), and the message popped up after the game had ran a bit. In the second one, the message popped up immediately upon loading, and had two lines "LRP_5 119.8s" and "LRP_8 139.9s" (again the times were increasing). I restarted the app/game completely and didn't get these messages again.

Game Install Info:
Game launched through Steam, steam cloud saving is enabled
Game is installed on C drive - nvme ssd - in the default steam location

System Specs:
Intel Core i7-8700
GTX 1060-3GB
RAM 16GB
Windows 10 (Version 21H1 / Build 19043.1237)
TagsNo tags attached.

Relationships

has duplicate 0025636 resolved Deserialization Continues 
related to 0025666 new Unable to Load Saved Game 

Activities

Eluthena

Oct 7, 2021 5:57 am

reporter  

Campaign Folder.zip (4,458,982 bytes)
First Cap.save (150,028 bytes)
Left off.save (181,355 bytes)

HarryT

Oct 7, 2021 2:48 pm

reporter   ~0062911

Same error/crash noted here.

BadgerBadger

Oct 12, 2021 12:52 pm

manager   ~0062918

I have also observed this on svn builds

Daniexpert

Oct 13, 2021 10:40 am

developer   ~0062922

3.706 still here
This is the useful part of the log
exception.txt (2,119 bytes)   
10/13/2021 4:30:49 PM	3.706	SINGLEP	GameEntity_Squad deserialization error at stage 19000 from serialized version 3.706 loading into new version 3.706, error: System.Exception: Could not read string, but did get this partial one: 't?? *sequence of <0x00>*'.  Was for field with name:' DeathEffectCausingDamageReceivedToEntity_Type'  Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
  at Arcen.Universal.ArcenDeserializationBufferModern.GetBits_InnerHelperChar (System.Boolean ReadFullUnicode) [0x000c7] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.Universal.ArcenDeserializationBufferModern.ReadString_Condensed (System.String FieldNameForErrors) [0x000e3] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.Universal.ArcenDeserializationBufferModern.ReadString_Condensed (System.String FieldNameForErrors) [0x00209] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeFrom_Inner (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String& NameFound, System.Boolean DoByIndex, Arcen.Universal.LookupSwapAllowed SwapAllowed, System.String FieldNameForErrors) [0x0029d] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeByInternalName (Arcen.Universal.ArcenDeserializationBuffer Buffer, Arcen.Universal.LookupSwapAllowed SwapAllowed, System.String FieldNameForErrors) [0x00001] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeByInternalName (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String FieldNameForErrors) [0x00001] in <41d20c6546954b9baaba0428856406cc>:0 
  at Arcen.AIW2.Core.GameEntity_Squad.DeserializeSquadIntoSelf (System.Int32 primaryKeyID, Arcen.AIW2.Core.GameEntityTypeData typeData, Arcen.AIW2.Core.Planet NewPlanet, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.Boolean HideErrors, System.Boolean IsLoadingTemplate, Arcen.Universal.SerializationCommandType SerializationCmdType, Arcen.AIW2.Core.PlanetFaction NewPlanetFaction) [0x01228] in <0a31df873aed43259043528e8a4839aa>:0 
exception.txt (2,119 bytes)   

x4000Bughunter

Oct 13, 2021 6:29 pm

administrator   ~0062928

Partial fix, but also making it worse:

* Broke savegame serialization again, but made future serialization more likely to be correct -- or at least error at the correct spot.
** Also cleaned up some serialization logging flags that were slightly inconsistent and which thus would mask errors.

x4000Bughunter

Oct 13, 2021 10:40 pm

administrator   ~0062929

There's no way I solved all of this, although I did find some errors. Unfortunately, they made the saves unreadable. If you wind up with any saves that are problematic in the next build, please let me know. Sorry about this!

* FireteamRequiredTargets are now properly pooled; previously they were small objects that just leaked.

* WormholeInvasionData was previously treated as a struct rather than a class, but after a lot of discussion it looks simpler to treat things like this as poolable structs.
** These also really need to use DeserializeUncertainNumberOfEntriesIntoExistingList in order to avoid a memory leak in multiplayer on clients, and now do.

* ProtectedDictionary has been split into ProtectedKeyDictionary and ProtectedValDictionary, since we might one to protect one or the other.
** Also, these just now use IProtectedListable, rather than IProtectedDictionaryable.

* WormholeWaveData has also had a similar update.

* Found a general serialization bug in StoredAIPurchaseCostByBudgetForSpecificUnits, which was blanking out all of the budget types on save to disk or network.
** The way this is stored and processed is now more efficient in MP and SP, but especially on MP clients.

* OutguardSpawnRequest is also now pooled, and also no longer leaks on MP clients thanks to DeserializeUncertainNumberOfEntriesIntoExistingList.

* I think that there are still other serialization errors for me to deal with, unfortunately. But things are better now than they were.

Daniexpert

Oct 14, 2021 10:50 am

developer   ~0062930

Got it again in 3.708
exception-2.txt (2,208 bytes)   
10/14/2021 4:44:57 PM 3.708 SINGLEP GameEntity_Squad deserialization error at stage 19000 from serialized version 3.708 loading into new version 3.708, error: System.Exception: Could not read string, but did get this partial one: '7?YOyVx5 *sequence of <0x00>*'.  Was for field with name:' DeathEffectCausingDamageReceivedToEntity_Type'  Exception: System.Exception: Tried to read condensed string format index 117 which is >= the length of 106 supported characters.  Some bad data just happened prior to this!
  at Arcen.Universal.ArcenDeserializationBufferModern.GetBits_InnerHelperChar (System.Boolean ReadFullUnicode) [0x000c1] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.Universal.ArcenDeserializationBufferModern.ReadString_Condensed (System.String FieldNameForErrors) [0x000e3] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.Universal.ArcenDeserializationBufferModern.ReadString_Condensed (System.String FieldNameForErrors) [0x00209] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeFrom_Inner (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String& NameFound, System.Boolean DoByIndex, Arcen.Universal.LookupSwapAllowed SwapAllowed, System.String FieldNameForErrors) [0x0029d] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeByInternalName (Arcen.Universal.ArcenDeserializationBuffer Buffer, Arcen.Universal.LookupSwapAllowed SwapAllowed, System.String FieldNameForErrors) [0x00001] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.Universal.ArcenDynamicTable`1[T].DeserializeByInternalName (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String FieldNameForErrors) [0x00001] in <2fa98ccb16b8446897da96d5067dc2f2>:0 
  at Arcen.AIW2.Core.GameEntity_Squad.DeserializeSquadIntoSelf (System.Int32 primaryKeyID, Arcen.AIW2.Core.GameEntityTypeData typeData, Arcen.AIW2.Core.Planet NewPlanet, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.Boolean HideErrors, System.Boolean IsLoadingTemplate, Arcen.Universal.SerializationCommandType SerializationCmdType, Arcen.AIW2.Core.PlanetFaction NewPlanetFaction) [0x01228] in <1301c755574c49a59aebb2737ebe902b>:0 
exception-2.txt (2,208 bytes)   

x4000Bughunter

Oct 14, 2021 11:07 am

administrator   ~0062931

Yep, I also was seeing it. Here's one fix, not sure if it is the last one. I'm doing some more testing:

* Fixed a serialization bug from the last few versions that was related to some new death-effect data.
** Additionally, while I was at it, made those serialize by index rather than name, since that will be more efficient given the nature of their data.
** Note that for hack types we're still serializing by name, since that's more efficient there -- there are many hacks, and few used at once, which is not a good candidate for by-index. But death effects are few in type but used many at a time, which is a great candidate.

x4000Bughunter

Oct 14, 2021 11:15 am

administrator   ~0062932

This seems to be the one.

Issue History

Date Modified Username Field Change
Oct 7, 2021 5:57 am Eluthena New Issue
Oct 7, 2021 5:57 am Eluthena File Added: Campaign Folder.zip
Oct 7, 2021 5:57 am Eluthena File Added: LastLobbySettings_usedforcampaign.save
Oct 7, 2021 5:57 am Eluthena File Added: First Cap.save
Oct 7, 2021 5:57 am Eluthena File Added: Left off.save
Oct 7, 2021 2:48 pm HarryT Note Added: 0062911
Oct 12, 2021 12:52 pm BadgerBadger Note Added: 0062918
Oct 13, 2021 10:40 am Daniexpert Note Added: 0062922
Oct 13, 2021 10:40 am Daniexpert File Added: exception.txt
Oct 13, 2021 6:29 pm x4000Bughunter Note Added: 0062928
Oct 13, 2021 10:40 pm x4000Bughunter Assigned To => x4000Bughunter
Oct 13, 2021 10:40 pm x4000Bughunter Status new => feedback
Oct 13, 2021 10:40 pm x4000Bughunter Note Added: 0062929
Oct 14, 2021 10:50 am Daniexpert Note Added: 0062930
Oct 14, 2021 10:50 am Daniexpert File Added: exception-2.txt
Oct 14, 2021 11:07 am x4000Bughunter Note Added: 0062931
Oct 14, 2021 11:15 am x4000Bughunter Status feedback => resolved
Oct 14, 2021 11:15 am x4000Bughunter Resolution open => fixed
Oct 14, 2021 11:15 am x4000Bughunter Fixed in Version => Beta 3.709 Upgrades And Serialization Fix
Oct 14, 2021 11:15 am x4000Bughunter Note Added: 0062932
Oct 14, 2021 7:19 pm Daniexpert Relationship added has duplicate 0025636
Oct 16, 2021 12:28 pm Daniexpert Relationship added related to 0025666