View Issue Details

IDProjectCategoryLast Update
0024383AI War 1 / ClassicCrash/ExceptionFeb 2, 2021 5:28 pm
ReporterLightjolly Assigned ToChris_McElligottPark  
Severityminor 
Status resolvedResolutionfixed 
Summary0024383: Hunter Fleet killed
DescriptionVersion 2.726

My specs rtx2070 super

i7-7700 4 core, 8 threads cpu

The match had civilian industries, Enemy Scourge and Kaizer Marauders, Macrophage, Reprocessors , Dark Spire minor factions. It was late game, where simulation speed would drop to 70 or even 45% and when it was at 100% it still had a slowness, stuttering to it. That error I attached popped up.

Saving and reloading the game actually alleviated the problem and even performance saw a decent bump

TagsNo tags attached.
Internal WeightNew

Activities

Lightjolly

Feb 1, 2021 1:36 pm

reporter  

error hunter.png (1,637,715 bytes)
ArcenDebugLog.txt (1,007,619 bytes)

Chris_McElligottPark

Feb 1, 2021 3:18 pm

administrator   ~0060435

Ah, this threw me off as it was in AI War rather than AI War 2! No worries, though.

Chris_McElligottPark

Feb 2, 2021 10:18 am

administrator   ~0060443

Okay, so the core error in question:

1/31/2021 1:05:53 PM 2.726 SINGLEP Thread 'specialFactionContext_85_HunterFleet_idx_4' is in state Running after 30 seconds, so killing it now
1/31/2021 1:05:53 PM 2.726 SINGLEP Exception in pathfinding debugStage 3800
System.Threading.ThreadAbortException
  at (wrapper stelemref) System.Object.virt_stelemref_class_small_idepth(intptr,object)
  at Arcen.AIW2.External.ArcenPathfinder`1[N].FindPathInner (N Origin, N Target, System.Int32 NeedToGetWithinXRangeOfTarget, System.Int32 RequiresNoMoreThanXRangeFromOrigin, System.Boolean DoDebugLog, Arcen.AIW2.Core.ArcenSimContext Context) [0x007e6] in <830aa89d750841fb821fbe515f1f873f>:0
1/31/2021 1:05:53 PM 2.726 SINGLEP debug code 500 during hunter fireteam logic. System.Threading.ThreadAbortException
  at Arcen.AIW2.External.ArcenPathfinder`1[N].FindPath (N Origin, N Target, System.Int32 NeedToGetWithinXRangeOfTarget, System.Int32 RequiresNoMoreThanXRangeFromOrigin, Arcen.AIW2.Core.ArcenSimContext Context) [0x0002a] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.SpecialFaction_HunterFleet.FindPath (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.Core.Planet Origin, Arcen.AIW2.Core.Planet Target, Arcen.AIW2.Core.PathingMode mode, Arcen.AIW2.Core.ArcenSimContext Context) [0x00035] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.Core.Faction.FindPath (Arcen.AIW2.Core.Planet origin, Arcen.AIW2.Core.Planet destination, Arcen.AIW2.Core.PathingMode mode, Arcen.AIW2.Core.ArcenSimContext Context) [0x00001] in <a6eaae2cbe304d19b73ecdb649f6a9d8>:0
  at Arcen.AIW2.External.Fireteam.GetDangerOfPath (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.External.ArcenLongTermIntermittentPlanningContext Context, Arcen.AIW2.Core.Planet source, Arcen.AIW2.Core.Planet destination, System.Boolean includeDestination, System.Int16& hops, System.Boolean includeStartingPlanet) [0x00001] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.SpecialFaction_HunterFleet+<>c__DisplayClass25_0.<AssignUnitToFireteam>b__0 (Arcen.AIW2.External.Fireteam team) [0x000e4] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.Fireteam.DoFor (Arcen.Universal.ArcenLessLinkedList`1[ItemType] List, Arcen.AIW2.External.Fireteam+ProcessorDelegate Processor) [0x00040] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.SpecialFaction_HunterFleet.AssignUnitToFireteam (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.Core.GameEntity_Squad entity, Arcen.AIW2.External.ArcenLongTermIntermittentPlanningContext Context) [0x000f5] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.SpecialFaction_HunterFleet.DoLongRangePlanning_OnBackgroundNonSimThread_Subclass (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.External.ArcenLongTermIntermittentPlanningContext Context) [0x001c3] in <830aa89d750841fb821fbe515f1f873f>:0
1/31/2021 1:05:53 PM 2.726 SINGLEP Thread specialFactionContext_85_HunterFleet_idx_4 with extra details '' was killed from running over time: System.Threading.ThreadAbortException
  at Arcen.AIW2.Core.Faction.Safe_DoLongRangePlanning_OnBackgroundNonSimThread (Arcen.AIW2.Core.ILongRangePlanningContext Context) [0x00038] in <a6eaae2cbe304d19b73ecdb649f6a9d8>:0
  at Arcen.AIW2.External.SpecialFactionPlanning.Execute () [0x0005f] in <830aa89d750841fb821fbe515f1f873f>:0
1/31/2021 1:05:53 PM 2.726 SINGLEP Thread specialFactionContext_85_HunterFleet_idx_4 with extra details '' was killed from running over time: System.Threading.ThreadAbortException
  at Arcen.AIW2.External.SpecialFactionPlanning.Execute () [0x000a7] in <830aa89d750841fb821fbe515f1f873f>:0
  at Arcen.AIW2.External.ArcenSimPlanningContext.InnerRun () [0x0006a] in <830aa89d750841fb821fbe515f1f873f>:0

Chris_McElligottPark

Feb 2, 2021 5:28 pm

administrator   ~0060452

Coming next build of AI War 2, version 2.727:

* FindPath() is being retired and replaced by FindPathFreshOrFromCache().
** This latter returns a new, cached, PathBetweenPlanetsForFaction object rather than just a list of planets.
** If a programmer alters that list of planets after they get the object, they need to be stoned lightly because it will break all sorts of things.
** The idea here is that we are calculating paths way too frequently, often thousands of times in a single second, and most of the paths we are calculating are the same during that span.
*** This is very true for things like checking the danger of a path (for some of the pathfinding variants), as well as for fireteams in general, as well as for ships that are undergoing fleet movement orders of certain sorts.
** This overall change should lead to a dramatic improvement in performance in a number of areas of the game, but mostly in the background AI processing for factions.
** The old FindPath() method is still there, so existing mods will still work, but it's been marked as obsolete so that mods will need to update to the newer, more efficient methods before they can be properly compiled/updated by the mod author for a new version.
** Also added a new PathfindingCacheForFactionOnContextObject that gets cached on ArcenSimContext, so that we can have thread-safe separate caches per context per faction (main sim, long term planning, whatever).
** In our innermost calculations for finding paths, we are also now reusing lists better, and we are using the [ThreadStatic] attribute for the first time in this program. This turns out to be extremely useful, and we just recently learned about it: https://docs.microsoft.com/en-us/dotnet/api/system.threadstaticattribute
** Additionally, even if the number of paths generated was the same (it's very much not), we're also now saving a lot of transient GC allocations, which is nice.
** Note to modders: if you were using FindPath, generally you are now using FindPathFreshOrFromCache(). This gives back a PathBetweenPlanetsForFaction object with a PathToReadOnly on there, and you should ONLY read from it, not ever alter it.
*** It is also possible for it to give back a null PathBetweenPlanetsForFaction object, so you do have to check for that now (that generally means an invalid pathing attempt, or already at the target, etc, and is there for efficiency's sake.)
** Another note to modders: if you were getting a pathfinder directly and then trying to call FindPath on the pathfinder... that's really something to be avoided, now, because it won't cache at all.
*** To work around that, you can call InnerFindPath_RawSinglePathfinder() and pass in the info as well as the pathfinder of your choice, and you'll get back either a populated PathBetweenPlanetsForFaction or null, all of it cached properly and thus more efficient.
*** There's nothing that will FORCE you to use pathfinders this way -- you just can switch to passing in a list of planets to be filled rather than just getting one in return -- but unless you are absolutely positive that this pathfinding call gets called less frequently than once every few seconds, you really should use the cached versions.
** In the escape menu, at the bottom of the performance section, we can all now see the number of path recalcs, and the number of paths from cache.
*** For reference, in a fairly large savegame with a fair number of late-game factions (5 hours in, near the end), running the game for about 20 seconds leads to 1200 path recalculations, and 4500 path pulls from cache. Each pathing call is pretty intense, and this is not remotely the most expensive scenario we have around, so this is a huge potential boost in performance, especially for older CPUs.
**** In the case of certain other savegames that we saw logs of, where the Hunter fleet thread was getting terminated for taking more than 30 seconds (yikes!) to do its work, it will be interesting to see what these numbers are like and if that is no longer a long-running thread. Either way, these performance improvements are worth it. But if not, then we can use that specific savegame to do some testing and tune things different ways for the hunter specifically.


Thanks! Please do give us a followup report and post the savegame (in a new ticket, in the AI War 2 game ideally) if you still see the hunter threads being killed after all of this.

Issue History

Date Modified Username Field Change
Feb 1, 2021 1:36 pm Lightjolly New Issue
Feb 1, 2021 1:36 pm Lightjolly File Added: error hunter.png
Feb 1, 2021 1:36 pm Lightjolly File Added: ArcenDebugLog.txt
Feb 1, 2021 3:18 pm Chris_McElligottPark Note Added: 0060435
Feb 2, 2021 10:18 am Chris_McElligottPark Note Added: 0060443
Feb 2, 2021 5:28 pm Chris_McElligottPark Assigned To => Chris_McElligottPark
Feb 2, 2021 5:28 pm Chris_McElligottPark Status new => resolved
Feb 2, 2021 5:28 pm Chris_McElligottPark Resolution open => fixed
Feb 2, 2021 5:28 pm Chris_McElligottPark Internal Weight => New
Feb 2, 2021 5:28 pm Chris_McElligottPark Note Added: 0060452