View Issue Details

IDProjectCategoryLast Update
0022090AI War 2Crash/ExceptionNov 7, 2019 1:07 pm
ReporterNRSirLimbo Assigned ToChris_McElligottPark  
Severitymajor 
Status resolvedResolutionfixed 
Product Version1.003 Sortable Objectives 
Fixed in Version1.006 Freedom Of Fleet Line Combination 
Summary0022090: Resource Leak
DescriptionFound a resource leak within the game where the RAM kept increasing seemingly without limit. Since the game didn't crash, only got slower and slower, I was able to make a screenshot of the debug readout in the escape menu and create a savegame.

An error seemingly related to issue 0021648 appeared right beforehand.

For those who can't read German, that's 12.3 GB reserved RAM and 9 GB actively used.
TagsNo tags attached.

Activities

NRSirLimbo

Nov 4, 2019 9:04 pm

developer  

20191105023707_1.jpg (384,294 bytes)
20191105024742_1.jpg (351,157 bytes)
resource leak.png (28,558 bytes)   
resource leak.png (28,558 bytes)   
s2 resource leak.save (2,266,727 bytes)

Chris_McElligottPark

Nov 4, 2019 9:45 pm

administrator   ~0054349

Can you post your ArcenDebugLog.txt from your PlayerData folder? I suspect the leak might be something along the lines of tons of error messages popping out for some reason...

Thanks!

NRSirLimbo

Nov 4, 2019 10:21 pm

developer   ~0054356

Oops, forgot that among the other stuff
ArcenDebugLog.txt (1,046,460 bytes)

NRSirLimbo

Nov 6, 2019 12:12 pm

developer   ~0054392

Ran into it again. This time the PC had a bluescreen with MemoryManagement (gonna guess it's because of the leak...)
ArcenDebugLog-2.txt (1,095,788 bytes)

Chris_McElligottPark

Nov 6, 2019 1:28 pm

administrator   ~0054393

There is definitely some sort of bug in the metal flow planning, related to "Error! PlannedMetalFlow from pool was already internalIsInUseNow" and "Danger! CurrentFrameNumber changed from 34405 to 34406 during metalFlowContext!"

I'm not sure if that's related to the UI, or what. Have you by any chance been using the metal flows window that you get when clicking the metal display in the top resource bar? If you don't use that, does this still happen?

I'm surprised more people aren't running into this, so I'm not sure what's different in your savegame or your style of play, one or the other. Either way we need to fix it, but just trying to narrow it down.

Thanks!

Chris_McElligottPark

Nov 6, 2019 1:30 pm

administrator   ~0054394

Oh, am I also correct in that the second time you ran into this (and got a bluescreen), you were in a new savegame and not the same old one? Looks like a fresh one was generated. I guess it's possible this is related to a specific faction you have enabled and some bug in them or some bug in a specific combo of ships. That save you posted is pretty large, so I'll have to look into it and see what's going on.

Chris_McElligottPark

Nov 6, 2019 1:47 pm

administrator   ~0054395

When I open this savegame, I get super smooth memory usage and no leaks when I am just panning around the battlefield and not really doing anything.

When I open up the metal flows interface, I have yet to get any exceptions, BUT the memory starts climbing like crazy (20mb a second or so, on average, but not wholly consistently and sometimes with some drops). Closing the metal flows interface causes the usage to go back down some, but not remotely to where it had been.

The number of "raw budgets" does seem to start going up inappropriately at some point, too, possibly after opening the metal flows window. In general the number of "raw budgets" seems incorrectly high (171k and counting after about 5 minutes on 10x fast forward, with 2.9m total). So there seems to be a leak in that area, possibly only somewhat related to the interface itself.

Badger, if you see this note, I'll look into this one since it spans several areas that are not all related to you.

NRSirLimbo

Nov 6, 2019 2:20 pm

developer   ~0054396

To answer a few questions: I usually don't use the metal flow interface and as far as I can recall I never had it open when these errors occur.

Every time a new update comes out I abandon the current game and I create a new one so there's no false positives on bugs or exceptions. The leak I ran in today was on a fresh save created just today on version 1.005.

I don't frequently check the RAM usage, but it seems to spike way harder than "only" 20mb/s, or maybe I just start to notice it at some point. What I could imagine is that there's some kind of infinite loop in there that simply mass-triggers the "small" leak?

I'll keep my eyes out when I play on how much RAM the game uses over time. I have been noticing a slow in performance over gameplay, though that could also be due to higher ship numbers.

What I have noticed at one point is a massive decrease in performance when I open the galaxy map. I have yet to check if that has anything to do with what we're tracking down, but it went from no number displayed to 0000143:0000050-60% when opened and then ticked back up to 0000154:0000070-80%, probably full performance again when closed.

Also, is there any way I can trigger a thread position state dump or however you call them (as in a file which writes down where in which code line all the threads currently are)? That might help locate the infinite loop, provided there even is one.

Lastly, I encountered these minor errors before, I do in just about ever savegame that survives for more than an hour, but not directly before it went out of control.

NRSirLimbo

Nov 6, 2019 2:20 pm

reporter   ~0054397

IDK what is up with mantis and tilde, but I meant to say 50-60% and 70-80% up there.

Ovalcircle

Nov 6, 2019 2:25 pm

reporter   ~0054398

My guess with the ~ (sorry for off topic) is that it makes it go the the comment number after the ~. So 0000151:0000056 would be issue whatever:000056

Chris_McElligottPark

Nov 6, 2019 7:28 pm

administrator   ~0054413

Correct on all those notes about ~.

Coming in 1.006:

* Fixed a pretty nasty memory leak that could happen around planned metal flows, particularly given that they were being used from a single pool but for two different threading areas.
** This was something that would affect things even when the metal flows UI wasn't open, because of race conditions between two threads. It was also causing some errors and other race conditions in general, all of which are now fixed.
** It was a very unusual sort of situation with a very special-case data structure (the time-based pool), and that was just... really an edge case thing. It also was not really causing errors or a memory leak in all cases, even though it was in some. Why it would be particularly severe for the one player who reported it, while nobody else had crashes related to this, is mildly baffling. But that's the nature of race conditions for you, honestly.
** Thanks to NRSirLimbo for reporting.

There's still another leak, though, so I need to also hunt that one down.

Chris_McElligottPark

Nov 6, 2019 7:56 pm

administrator   ~0054414

* Fixed a couple of memory leaks and other things that caused excessive memory churn while actively viewing the metal flows interface. These were things that would be given back to the player and thus not truly a memory leak in general; it was leaking on the heap and then coming back later.

Chris_McElligottPark

Nov 6, 2019 8:37 pm

administrator   ~0054415

Okay, coming tomorrow:

* The last of the memory-leaky-type things related to metal flows is now fixed. This one was introduced possibly as far back as September 2018 by Chris, or possibly more recently when changing another bit of code to make a copy per list instead of having a reference in another list. At any rate, we were creating a metric ton of extra metal flow data that was then getting churned out of the memory pool by the garbage collector, and generally wasting some CPU time and efficiency. Now things properly get returned to the pool.
** On the metal flows screen, there is still some climbing RAM that you see for a while, but then it stabilizes and eventually drops. That seems to be mostly related to temporarily cached sprite quads being used by TextMeshPro, and with such large amounts of text that changes frequently that's probably unavoidable. It's not a true memory leak anymore, and doesn't spike more than something like 60mb at peak in our current tests, where it was hundreds of mb in the same tests previously.

Thanks!

NRSirLimbo

Nov 7, 2019 6:15 am

reporter   ~0054419

I like the sound of that :)

Good work!

Chris_McElligottPark

Nov 7, 2019 1:07 pm

administrator   ~0054427

Thanks! Fingers crossed. :)

Issue History

Date Modified Username Field Change
Nov 4, 2019 9:04 pm NRSirLimbo New Issue
Nov 4, 2019 9:04 pm NRSirLimbo File Added: 20191105023707_1.jpg
Nov 4, 2019 9:04 pm NRSirLimbo File Added: 20191105024742_1.jpg
Nov 4, 2019 9:04 pm NRSirLimbo File Added: resource leak.png
Nov 4, 2019 9:04 pm NRSirLimbo File Added: s2 resource leak.savemet
Nov 4, 2019 9:04 pm NRSirLimbo File Added: s2 resource leak.save
Nov 4, 2019 9:45 pm Chris_McElligottPark Note Added: 0054349
Nov 4, 2019 10:21 pm NRSirLimbo File Added: ArcenDebugLog.txt
Nov 4, 2019 10:21 pm NRSirLimbo Note Added: 0054356
Nov 6, 2019 12:12 pm NRSirLimbo File Added: ArcenDebugLog-2.txt
Nov 6, 2019 12:12 pm NRSirLimbo Note Added: 0054392
Nov 6, 2019 1:28 pm Chris_McElligottPark Note Added: 0054393
Nov 6, 2019 1:30 pm Chris_McElligottPark Note Added: 0054394
Nov 6, 2019 1:47 pm Chris_McElligottPark Note Added: 0054395
Nov 6, 2019 2:20 pm NRSirLimbo Note Added: 0054396
Nov 6, 2019 2:20 pm NRSirLimbo Note Added: 0054397
Nov 6, 2019 2:25 pm Ovalcircle Note Added: 0054398
Nov 6, 2019 2:37 pm BadgerBadger Assigned To => Chris_McElligottPark
Nov 6, 2019 2:37 pm BadgerBadger Status new => assigned
Nov 6, 2019 7:28 pm Chris_McElligottPark Note Added: 0054413
Nov 6, 2019 7:56 pm Chris_McElligottPark Note Added: 0054414
Nov 6, 2019 8:37 pm Chris_McElligottPark Status assigned => resolved
Nov 6, 2019 8:37 pm Chris_McElligottPark Resolution open => fixed
Nov 6, 2019 8:37 pm Chris_McElligottPark Fixed in Version => 1.006 Freedom Of Fleet Line Combination
Nov 6, 2019 8:37 pm Chris_McElligottPark Note Added: 0054415
Nov 7, 2019 6:15 am NRSirLimbo Note Added: 0054419
Nov 7, 2019 1:07 pm Chris_McElligottPark Note Added: 0054427