View Issue Details

IDProjectCategoryLast Update
0020072AI War 2Bug - GameplaySep 25, 2018 5:33 pm
ReporterHeartHunter7 Assigned ToChris_McElligottPark  
Severitymajor 
Status resolvedResolutionfixed 
Product Version0.763 Supersized Ship Caps 
Fixed in Version0.775 Unit Stacking 
Summary0020072: lags
DescriptionI have 4 replayes with lag from one game. Save/load cant help. Last save is really hard lag.
TagsNo tags attached.

Activities

HeartHunter7

Aug 30, 2018 7:58 pm

reporter   ~0048628

cant load replay, seems that mantis have a little problem.
"no correct marker of defence"

HeartHunter7

Aug 30, 2018 8:07 pm

reporter  

lag.save (5,004,896 bytes)
lag.savemet (40 bytes)

HeartHunter7

Aug 30, 2018 8:07 pm

reporter  

lag2.save (7,321,133 bytes)
lag2.savemet (41 bytes)

HeartHunter7

Aug 30, 2018 8:07 pm

reporter  

epic lag .save (2,358,095 bytes)
epic lag .savemet (41 bytes)

HeartHunter7

Aug 30, 2018 8:08 pm

reporter  

lag3.savemet (41 bytes)

HeartHunter7

Aug 30, 2018 8:09 pm

reporter   ~0048631

File of last save so big.
https://dropmefiles.com/bWIv0

HeartHunter7

Aug 31, 2018 6:36 am

reporter   ~0048651

i think i find reason of laggs in the last save. Its..... MLRS corvettes, when AI killed(really ez) my planet, lags is gone.
20180831133407_1.jpg (516,043 bytes)

RocketAssistedPuffin

Aug 31, 2018 6:50 am

reporter   ~0048652

Seems simply too many shots being spawned. MLRS take so long to do a single salvo they start to fire more alongside the first. 1570 Corvettes means over 4500 missiles being spawned at a time.

Putting them to fire all at once would be...horrendous for performance (1570 x 16 = 25120 missiles every 2 seconds!)

This requires either a performance increase I think, or a change in how MLRS work to lower their incredible shot count.

Might need both if someone gets a ludicrously large swarm...

HeartHunter7

Sep 2, 2018 12:53 pm

reporter   ~0048726

Another example. I have laggs every 2-3 sec on 0.7 sec. And its not fighting laggs, i look only on galaxy map.
lag every 2 sec.save (5,634,601 bytes)

Chris_McElligottPark

Sep 3, 2018 4:03 pm

administrator   ~0048789

New:

* Reorganized the campaign stats section a bit, and broadened the game performance section (as well as moving it down).
** It now says if the game is paused or running, it says what the current frame is, the current entity count, the number of garbage collector calls, the total entities created, and the current command count (that last one is a rough estimate).
** This helps even non-developers tell at a glance what might be going on if the performance is low (tons of commands, tons of entities created or present, it lags every time the garbage collector runs, etc).

Chris_McElligottPark

Sep 3, 2018 4:06 pm

administrator   ~0048790

In this one from HeartHunter, a few things happen:

1. It's very laggy even when paused, even when on the galaxy map.
2. It has about 40k live entities.
3. Unpausing it adds another 20k entities pretty fast, which is probably MLRS shots.
4. It then becomes a slideshow, even on the galaxy map.
5. In particular there are some frames that are EXTREMELY long (3 seconds or so) that have no connection to the vis layer or the garbage collector being called.

It's worth also noting that this save doesn't have any factions enabled beyond the vanilla ones, so this should be an ideal test case. All those crazy numbers of shots from MLRS need to be toned down, of course, but the engine should be able to not choke and die on this amount of entity throughput -- particularly while processing this stuff while not even yet unpaused!
lagMLRS.zip (952,361 bytes)

Chris_McElligottPark

Sep 4, 2018 5:18 pm

administrator   ~0048818

Next version:

* Major internal refactor: GameEntity has been split into four classes:
** GameEntity_Base provides the new abstract class that is underneath the other three.
** GameEntity_Squad is for all of the ships.
** GameEntity_Shot is for all of the shots, and is now vastly more lightweight than it used to be.
** GameEntity_Other is for basically just wormholes, and is incredibly more lightweight than it used to be.
** Despite all of these changes, old savegames still work.
** The new data for shots is now faster to load into RAM when creating a firing shot, as well as much more lightweight when saving a game with a lot of shots in it.
*** A savegame that has 14k squads and 29k shots previously took 9.61MB, but now only takes 6.66MB.
** There are a few cases where code is now slightly more verbose, but overall the memory footprint is down notably, and the CPU footprint is down somewhat.

* The changes to splitting out the shot data, and thus making it not remotely so heavy to initialize, makes it so that when there are thousands of shots being spawned in a second, it no longer causes lag spikes like it previously did. It was actively causing huge amounts of hitching before, simply as a part of creating all those new objects in memory, it turns out.
** In our test case, with the prior version of the game it was hanging for about 3 seconds every 3-5 seconds, and it was running at something like 35% speed when not hanging. It now has no hitches in that same savegame, and runs at a solid 45% of full speed. This is even true as it progresses into having more than 45k active shots in the game at once, pretty much all of which are all on one planet (nightmare performance scenario).

* When saving to disk, some of the fairly-temporary data having to do with the plan for the next frame (that has not happened yet) is no longer saved. It simply isn't needed on disk, but it is needed for sync between clients in multiplayer.
** This brings the on-disk size of the large savegame that has 14k squads and 29k shots further down from 6.66MB to 5.29MB.

* A new function has been put into our serializer, where it "compacts" down the Int64 variables that are used for GameEntityIDs, by simply renumbering them 1-x as they are added into it. This only happens when saving it to disk, but it has the effect of majorly reducing the filesize of long-running games (or those with a lot of shots in them over time).
** This brings the on-disk size of the large savegame that has 14k squads and 29k shots -- which had had over 10 million entities in it since starting -- further down from 5.29MB to 2.68MB.

Chris_McElligottPark

Sep 4, 2018 8:10 pm

administrator   ~0048820

Further notes:

Puffin reduced the massive spam of MLRS. Although, surprisingly it had very little effect on the speed of the game running in the particular case here. The shots are apparently a negligible impact at the moment, somehow, in that savegame for me. Even when paused and no shots are present, I get the same amount of slowdown that I do when unpaused and there are 45k shots or 30k shots. His changes did drop the number of shots from 45k to 30k, however, which presumably will help out, particularly in even LARGER battles, and obviously it's good for balance. But it only improved the running speed by about 3%, if that, which is a big surprise to me. I think something else is simply the bottleneck right now.

HeartHunter7

Sep 4, 2018 8:36 pm

reporter   ~0048822

Feels like greanade launcher will next.

Chris_McElligottPark

Sep 25, 2018 5:33 pm

administrator   ~0049487

FINALLY got this completely, thank you!

* It is now possible for AI/NPC units/squads to be "stacked."
** A stack consists of 2+ identical units, with a little marker on them stating how many are in the stack.
** A stack is able to fire at up to 5x its normal attack power, depending on how many units are in the stack (1 unit in a stack = 2x, 5 units = 5x, 5+ is still 5x).
** A stack takes damage just as it would if it were not a stack. When the current item visible from the stack "dies," then the next item pops out and the stack count goes down by one.
*** Damage does not roll over from one unit in the stack to the next, so this makes AOE weapons weaker against stacks, as well as weapons with a lot of overkill.
** If a tractor is grappling a stack, then each stacked entity counts as 1/4 of a tractor target for purposes of the grappling capacity of the tractor beam.
*** This makes tractor beams substantially stronger against stacked enemies.
** The purpose of stacks is performance and visual decluttering. These keep the battles runnning quickly while still being large.
*** Note that ONLY mobile ships/squads can stack. Turrets and whatnot do not stack.
*** Stacks will typically only kick in when there are more than 50 of a single type of unit on a planet, owned by the same AI/NPC faction.
*** However, there is a max_count_per_npc_faction_before_stack flag that can be assigned to entities to make them stack sooner.
**** The various starships and guardians that have bubble forcefields now have a value of 3 for max_count_per_npc_faction_before_stack.
***** This keeps them from absolutely tanking performance if the AI somehow gets a hold of a ludicrous number of mobile shield generators on one planet, which has happened in the past.
** The stacking logic runs on a background thread on the game host only, and sends out orders across the network to cause units to stack when needed.
** It's worth noting that this also makes savegames WAY smaller during certain ginormo-battles.
** Additionally, when there are stacked units on a planet, and there are fewer than half the stacking cap, then stacks start being slowly split apart.
*** That way you don't wind up with a single stack of 50 units as the last entity, it will spread back out, first to two stacks of 25, and so forth and so on.
** Also note that the combining into stacks is done by mark level, so if there are multiple mark levels at a single planet it will not mix and match and change the difficulty on you.
** Thanks to these changes, our test savegames where we had awful things like literally 10% sim speed and 50+ second targeting cycles now run at full speed within 30 seconds of loading, and have quarter-second targeting cycles as well. In the 10% performance savegame's case, it also reduced the savegame file size from 8.6MB to 1.6MB.

Issue History

Date Modified Username Field Change
Aug 30, 2018 7:56 pm HeartHunter7 New Issue
Aug 30, 2018 7:58 pm HeartHunter7 Note Added: 0048628
Aug 30, 2018 8:07 pm HeartHunter7 File Added: lag.save
Aug 30, 2018 8:07 pm HeartHunter7 File Added: lag.savemet
Aug 30, 2018 8:07 pm HeartHunter7 File Added: lag2.save
Aug 30, 2018 8:07 pm HeartHunter7 File Added: lag2.savemet
Aug 30, 2018 8:07 pm HeartHunter7 File Added: epic lag .save
Aug 30, 2018 8:07 pm HeartHunter7 File Added: epic lag .savemet
Aug 30, 2018 8:08 pm HeartHunter7 File Added: lag3.savemet
Aug 30, 2018 8:09 pm HeartHunter7 Note Added: 0048631
Aug 31, 2018 6:36 am HeartHunter7 File Added: 20180831133407_1.jpg
Aug 31, 2018 6:36 am HeartHunter7 Note Added: 0048651
Aug 31, 2018 6:50 am RocketAssistedPuffin Note Added: 0048652
Sep 2, 2018 12:53 pm HeartHunter7 File Added: lag every 2 sec.save
Sep 2, 2018 12:53 pm HeartHunter7 File Added: lag every 2 sec.savemet
Sep 2, 2018 12:53 pm HeartHunter7 Note Added: 0048726
Sep 3, 2018 11:59 am Chris_McElligottPark Assigned To => Chris_McElligottPark
Sep 3, 2018 11:59 am Chris_McElligottPark Status new => assigned
Sep 3, 2018 4:03 pm Chris_McElligottPark Note Added: 0048789
Sep 3, 2018 4:06 pm Chris_McElligottPark File Added: lagMLRS.zip
Sep 3, 2018 4:06 pm Chris_McElligottPark Note Added: 0048790
Sep 4, 2018 5:18 pm Chris_McElligottPark Note Added: 0048818
Sep 4, 2018 8:10 pm Chris_McElligottPark Note Added: 0048820
Sep 4, 2018 8:36 pm HeartHunter7 Note Added: 0048822
Sep 25, 2018 5:33 pm Chris_McElligottPark Status assigned => resolved
Sep 25, 2018 5:33 pm Chris_McElligottPark Resolution open => fixed
Sep 25, 2018 5:33 pm Chris_McElligottPark Fixed in Version => 0.775 Unit Stacking
Sep 25, 2018 5:33 pm Chris_McElligottPark Note Added: 0049487