View Issue Details
| ID | Project | Category | Date Submitted | Last Update | |
|---|---|---|---|---|---|
| 0007188 | Valley 1 | Crash/Exception | Apr 20, 2012 8:33 am | Apr 20, 2012 12:27 pm | |
| Reporter | Toll | Assigned To | Chris_McElligottPark | ||
| Status | resolved | Resolution | fixed | ||
| Product Version | 0.970 (RC1) | ||||
| Fixed in Version | 0.971 (RC2) | ||||
| Summary | 0007188: Managed to crash the game | ||||
| Description | Sorry :( Fortunately (or perhaps unfortunately) it's perfectly reproducible though. Step 1: Enter a region (not sure if this step is necessary, but that's what I did both of the times I crashed it). Step 2: Think "Oh, right, that's what I forgot to do". Step 3: Pause the game and alt-tab out. Step 4: Return half an hour later and try to enter the game again. | ||||
| Tags | No tags attached. | ||||
| Internal Weight | New | ||||
|
|
|
|
|
Do you have to actually leave it for particularly long, or is it literally a half hour? From what I see in your logs, this is more or less what I know: 1. You didn't run out of RAM or anything like that. 2. The actual game logic doesn't look to be at fault. 3. The new render logic doesn't look to be at fault. 4. I notice that among the DLLs it says are active, there is a security scanner in there. 5. The crash itself is really nonspecific and _seems_ to originate in unity's underlying engine itself rather than the managed runtime that hosts our code. 6. Well, okay, actually the crash says that it tried to write to a piece of memory that it couldn't. Which is usually a sign of being out of RAM, except you weren't; and otherwise is just a sign of unity's black box engine... basically killing itself. From that I'm having to conclude for the moment that this is probably actually the security software starting to scan when you're idle on your machine for a certain amount of time -- a lot of them are set to automatically do that. And then during a scan, it looks like it might be doing something that temporarily locks some memory that unity wanted to use, causing unity to crash. That would actually be a bug with unity (I think) rather than the security software. But it's hard to be sure, and there's still a remote possibility that this is actually a bug in our code somehow -- that we were causing runaway memory allocations during alt-tab out or something. I think the error signatures would look a bit different if that were the case, though. But I can't be sure, because all I have are the unity error signatures (since this originates in their runtime) rather than our actual managed error signatures (which actually gets to code that Keith and I have access to). SO... this is good data to have, but I don't _think_ it's a generalized bug in the game. I would be interested if simply changing your security software to not scan while you are idle for only 30 minutes would fix this, or if disabling the security software temporarily simply as a test would do the same. I'm also going to get Josh looking at this and seeing if he can duplicate it, just to be sure. Thanks! |
|
|
I didn't leave my computer idle though; I just alt-tabbed out of AVWW. So I don't think it's a case of F-Secure starting to scan my system. I'll try to disable it completely though and see if that makes a difference. |
|
|
Okay, that's important clarification -- thanks. Will try this as well and see what I can see! |
|
|
Oh, and regarding the length of time: I don't really know, to be honest. Half an hour did the trick both times for me though, but I don't know if ten or twenty minutes works too. |
|
|
Three questions: 1. What is the shortest time you've been able to leave it and have that happen? 2. Were you running fullscreen or windowed? 3. Did you literally alt-tab out, or use some other keys (windows key, etc)? And what did you specifically do to re-enter the game? (Sounds like alt-tab for sure there). |
|
|
Fair enough on the time, thanks. |
|
|
x4000 said: "just a sign of unity's black box engine... basically killing itself." ...again |
|
|
So, Josh was able to repro it by tabbing in and out a lot. This may just be a glitch in unity, thought; looking into it more: http://www.interstellarmarines.com/forum/threads/id/352/ |
|
|
Well, the thing is, it's new. I'm not using tabbing out anymore than I was before. It might be related to the graphics changes you did recently. I used to be able to have the game open in the background for hours. So long, in fact, that I used to forget about it. I had a crash last night that I thought was related to a video I was streaming, but now that I had it again today, I'm pretty sure it was AVWW last night too. All I have to do is tab out of the game and let it sit. I don't know if it takes a whole 30 minutes, but, it takes more than 5 or 10, to be sure. Could it be a VRAM issue? |
|
|
It could be related to the graphics changes I did recently. The thing with the graphics changes is that it hands off a bunch of data to unity for unity to do whatever the heck it wants to with it. On the old way of doing the graphics, it wasn't a lot more directly making calls to the GPU. So if unity has some sort of issue in their general render stack related to this, then that could be related. Given you say you saw this last night, Josh, I don't think this is related to the steam wrapper dll that was added. That was one other suspicion of mine. This may not be fixable without upgrading to a newer version of unity (we're still using version 3.3, and they are now on version 3.5). However, version 3.4 was buggy as heck and I didn't want to introduce new unknown issues right before release by moving to 3.5. At this point I sure as _heck_ don't want to risk further instability by doing that. So... we may just be stuck, but I'm seeing if I can google up anything on other unity devs running into this. |
|
|
I'm about 96.783% sure that it's not related to the Steam wrapper. |
|
|
Same. A few things: 1. Some folks were thinking this was related to an ATI driver bug, but Toll is running nVidia I see from his log. 2. I still can't duplicate this, leaving it for 10+ minutes at a go. What I can duplicate is it hanging for quite a long while as unity gets its RAM back if it's been idle too long. It may be that when it tries to do this on your machines, it's coming up with not enough RAM or something. 3. I'm not running any antivirus. Apparently there's a good history of AV products causing various issues with various versions of unity, including alt-tab crashes (but those are mainly with the web player or unity editor): http://forum.unity3d.com/threads/78979-Anti-virus-problems/page3 |
|
|
I get the long hang too, but, I'm not sure how related to the crash that is. When it crashes I don't have to be trying to tab back into it. I'm just doing something else and I get a box that pops up that says "AVWW.exe has stopped running blah blah blah". Also, no AV at all? You are a brave soul. I'm running Microsoft Security Essentials. |
|
|
Okay, that's good info that it just happens in the background randomly. What else do you have running? Running any videos or anything else that might use hardware acceleration of the GPU? And yes, I don't use antivirus because I use gmail and I don't download strange files. I have AV installed and do scans if I'm concerned over something, but I've not had a virus in the last 8-9 years on my personal machine. Having had viruses many times in the past at work and at home, I've kind of learned my lessons on what to and what not to do. Not that I recommend that path to folks, but I'm a huge hypocrite on that. ;) |
|
|
I also had a crash today in the same situation; alt-tabbed out, leave for a prolonged time (around half an hour), crash, pop-up box, etc. No antivirus running here as well. Would attach the crash log if I could find it anywhere. |
|
|
Okay, I'm coming up with nothing. There are lots of crash reports that are similar on unity 3.1, 3.3, 3.4, and 3.5. Most in the editor, then in the web player to a lesser extent. I guess now that we're using their more mainstream render path we're running into that now whereas we were not before. I'm not sure this is solvable, leastways not prior to 1.0 without causing more problems than we fix. :/ On the bright side, the game would have already saved your world and stuff, so it's not like it loses progress if you do this. On the downside... blah this looks bad when this sort of happens. |
|
|
Well, last night I was watching the Tigers game, which streams through their own wrapper over Flash. Today I think I had hulu.com streaming, which I think is Silverlight, though I'm not sure about that. (yes, I stream a lot. I'm sure my ISP hates me.) Let me try this without streaming anything and see if it's still an issue. |
|
|
Minotaar: were there any logs in your runtimedata folder? And then in your AVWW_Data folder is there an output.log file with any extra data? The latter gets overwritten each time you start the game, so it would be lost if you've already played more. |
|
|
|
|
|
Uploaded the output log, didn't run the game since the crash yet. Nothing new in RuntimeData. |
|
|
Minotaar: Thanks, no details were in there, though. I just managed to get the same thing to happen to me. I'm not sure if I wasn't waiting long enough or if it was that I just fired up a youtube video. Going to test this more. |
|
|
So; I tested again, no AV running (I unloaded F-Secure), and disabled my internet-connection as well (yes, I'm slightly paranoid). Since Chris mentioned RAM, I had a look at task manager as well; during the first minutes being alt-tabbed, nothing happened, but after that it started going up at a rate of about 700kB/s, and it finally dies at about 1.8GB RAM. |
|
|
I did get a different error message this time though, and no error log was generated. The error message was: "Runtime Error! Program: This application has requested the Runtime to terminate in an unusual way. Please contact the application's support team for more information." |
|
|
I can also confirm the RAM increase that you're seeing -- it's the same for me. I'm going to look and see if I can track this down, but at least now: a) The error logged actually does make sense. b) There's a clear symptom of the issue that is easily tracked well before it crashes. Also, I will note that I've already confirmed that this is not managed memory that is increasing. So there's something going on in the actual render call stack that is causing this, or else possibly something we're doing with texture loading if it's causing textures to get invalidated and reloaded. Given that, I actually -- knock on wood -- have a few ideas on how I might be able to solve this! |
|
|
Yep, I'm getting the same thing. Good luck squashing this one! |
|
|
Well, the "good" news is that I can verify what is causing this: calling Graphics.DrawMesh while the game does not have focus seems to do it. Or something in that call stack leading up to it; still hunting back. But basically, we're back to the problem of not being able to detect when the unity editor has focus or not. I'm going to see what I can do, sigh, but this gets into pretty hacky territory to work around their bugs. |
|
|
AHAHAHAHAHAHHAHAHAHAHAA! Giddy glee! After... two plus years of trying to figure out a way to detect window focus in unity, we FINALLY have a way to do it. And it solves this problem. It's a hack, but it's a reliable one because it's one that we were unintentionally relying on prior to the new rendering model; will explain more later. And yes, we've bugged unity about this as far back as 2010, but they never have done anything about the focus detection fix. This sort of thing doesn't work: http://forum.unity3d.com/threads/87860-Lost-Graphics-Device But I found something that did, and I'll share for other unity devs to know as well. What a huge relief. KNOCK. ON. WOOD. Will have a build out in 20-30 minutes for you guys to bang on, after I finish de-instrumenting the code and making my own final tests on this one. |
|
|
Woo! Sounds awesome. So, about all those ideas that were denied because there was no way to detect focus... *reopens a hundred issues* |
|
|
What?!??! Really?!?!? Does this mean that eventually we could make AVWW pause if I click on my second monitor? (I know that's out for 1.0) |
|
|
BTW, there is a good bit of wood in this apartment, my knuckles are sore now. :) |
|
|
"So, about all those ideas that were denied because there was no way to detect focus... *reopens a hundred issues*" "Does this mean that eventually we could make AVWW pause if I click on my second monitor?" That is EXACTLY what this means. :D And why I'm soooo happy. This will be a huge boost for both AI War AND AVWW. |
|
|
Although, one clarification: technically I can't detect if the application has focus _literally_. If you're running the application in windowed mode and click to something else, for instance, I can't detect that to my knowledge with this trick. But if you're running fullscreen and click out of the application, I can definitely detect that now -- at least on windows. Haven't tested on OSX yet. |
|
|
Wait, does this mean those blasted "key is still held down when I switch out and then switch back in" errors in AI war can finally be fixed? Similar thing to the planet view still scrolling when minimized? HOORAY!!! This will help the first impressions of lots of new AI war players, and many long time players will be grateful as well. |
|
|
"Wait, does this mean those blasted "key is still held down when I switch out and then switch back in" errors in AI war can finally be fixed?" MAYBE. Definitely the scrolling when-minimized thing, and MAYBE the key-held-down thing. However, it won't help with the key-held-down thing with the Steam overlay, unfortunately, so that one will persist no matter what until unity actually addresses the core bugs. But this lets us work around at least a pretty good swath of the issues! |
|
|
Awesome. So who wants to be in charge of reopening the dozens of issues in all three games that could be fixed (or at least partially fixed) with this? Of course, they will need to be linked to this issue. |
|
|
Thanks again, folks! The explanation text: A crash bug was discovered this morning, but had apparently been in the game since the new graphics rendering system updates a few days back. This was a particularly tricky crash bug, as you can see if you read the history of the linked issue. However, it has now been fixed -- and as a side bonus, it led us to the holy grail of things we've been wanting to be able to do in unity: detect when the @#$#$ window doesn't have focus. This has been a bug in unity since pretty much forever, and we've told them about it as far back as 2010, but nothing has changed on it. Frustratingly. A minor digression: not knowing if the window has focus has all sorts of bad consequences for us. It means that if the application loses focus, we can't pause it. It means that if the application doesn't have focus, it still takes keyboard input. And, in the case of this bug, it means that we keep calling Graphics.DrawMesh and unity keeps queuing up more and more unmanaged memory that it never gives back until the application is closed or dies of its own accord. On average it would cause the application to leak about 700kb of memory per second while you were alt-tabbed out of a fullscreen copy of the game, and thus you'd get a crash after some amount of time (usually when the total RAM usage of the process hit about 1.8GB for whatever reason). As a side effect of this issue with unity, it made alt-tabbing incredibly slow. Apparently all those Graphics.DrawMesh calls were being queued up until you came back (or something to that effect), so then it would take forever to load when you came back, even if it didn't crash. Now that we are able to detect when the game doesn't have focus (in fullscreen mode only -- none of this, including the original crash, applies to windowed mode), we simply stop doing any drawing while it doesn't have focus. That makes it alt-tab back in really snappily, and prevents the memory leak. So how are we detecting the loss of focus? Well, a hack -- but a reliable one, based on the fact that this bug only came in once we started using Graphics.DrawMesh. Normally unity 3D keeps running its Update calls on its camera even when the game has lost focus (well, there is a way to disable that -- but then network connections get killed when you alt-tab out, so we can't use that sort of thing). So far so good, and this is what we expect. We just didn't have any way to know if we should be calling draw calls (because the window has a valid device state attached to it) or not during that update method. Normally unity doesn't expose the device-lost state to the game-logic level, because apparently they think we don't need to know. What I kind of stumbled into, after two years of off-and-on intensive searching for a workaround for this, was the following: OnPostRender does not get called when the device has been lost. That's where our Graphics.DrawMeshNow calls go, and why we haven't had this sort of crash before we switched to also using Graphics.DrawMesh (which was where all that performance boost came from a couple of versions back). Knowing that OnPostRender doesn't happen when the device is lost is the key to the whole solution: if there have been too many Update calls since the last OnPostRender call, you know your device has been lost. Normally if there have been more than 1 Update calls since the last OnPostRender call, you can probably infer that it's been lost. But I'm using 20 as the cutoff point just to be on the safe side, and only cutting of the render calls after that point is reached. Once you alt-tab back in it immediately starts doing OnPostRender again and then the Update calls to Graphics.DrawMesh resume, and everything is happy. |
|
|
... Wow. For a hack, that's a real neat one, I must say! |
|
|
x4000 said: "it led us to the holy grail of things we've been wanting to be able to do in unity: detect when the @#$#$ window doesn't have focus. This has been a bug in unity since pretty much forever, and we've told them about it as far back as 2010, but nothing has changed on it. Frustratingly." I can see you have been just as frustrated over this as many new and long time players are. The root cause may not be fixed, but nice work finding a way to detect a "side effect" of the window being minimized! :) |
|
|
I thought so, too! Okay, the new version just went live, though I've not put up the blog posts yet -- if you guys don't mind all retesting just to make sure it is indeed fixed for all of you, that would be enormously appreciated! |
|
|
"I can see you have been just as frustrated over this as many new and long time players are." Sure. I'm frustrated by getting bug reports I can't fix, on the one hand. And on the other hand, those bugs affect me as much as the next person -- I also run dual monitors, etc, so it's annoying on a personal level also. |
|
|
Ah this explains the black screen when I was alt tabbing back in on the new version. Gald this issue is also fixed. |
|
|
Alt-tabbing is back to being almost-instant. Yay! I missed that. Will leave it running for a while. |
|
|
That's weird, did you guys have it go from using the custom cursor to using the OS cursor when you updated? Or was that just me? |
|
|
Yeah, I also had my settings flip to the mouse cursor. It did remember after I changed it and restarted the game, though, so no big deal. |
|
|
So; updated, started, paused, alt-tabbed out. Wait 20 minutes, and basically no change in RAM usage (about 60kB at most). Success! |
|
|
Yeah, it's working for me too. My RAM usage actually dropped while I was tabbed out. It only did this once, so I'm assuming the GC cycle hit. :) |
|
|
Sweeet! :) Very strange on the OS cursor, though. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| Apr 20, 2012 8:33 am | Toll | New Issue | |
| Apr 20, 2012 8:33 am | Toll | File Added: Crash_2012-04-20_134452.zip | |
| Apr 20, 2012 9:18 am | Chris_McElligottPark | Note Added: 0022476 | |
| Apr 20, 2012 9:18 am | Chris_McElligottPark | Assigned To | => Chris_McElligottPark |
| Apr 20, 2012 9:18 am | Chris_McElligottPark | Status | new => feedback |
| Apr 20, 2012 9:25 am | Toll | Note Added: 0022477 | |
| Apr 20, 2012 9:25 am | Toll | Status | feedback => assigned |
| Apr 20, 2012 9:26 am | Chris_McElligottPark | Note Added: 0022478 | |
| Apr 20, 2012 9:27 am | Toll | Note Added: 0022479 | |
| Apr 20, 2012 9:27 am | Chris_McElligottPark | Note Added: 0022480 | |
| Apr 20, 2012 9:27 am | Chris_McElligottPark | Note Added: 0022481 | |
| Apr 20, 2012 9:31 am | TechSY730 | Note Added: 0022482 | |
| Apr 20, 2012 9:31 am | Chris_McElligottPark | Note Added: 0022483 | |
| Apr 20, 2012 9:36 am | tigersfan | Note Added: 0022484 | |
| Apr 20, 2012 9:40 am | Chris_McElligottPark | Note Added: 0022485 | |
| Apr 20, 2012 9:43 am | tigersfan | Note Added: 0022486 | |
| Apr 20, 2012 9:47 am | Chris_McElligottPark | Note Added: 0022487 | |
| Apr 20, 2012 9:51 am | tigersfan | Note Added: 0022488 | |
| Apr 20, 2012 9:54 am | Chris_McElligottPark | Note Added: 0022489 | |
| Apr 20, 2012 10:00 am | Minotaar | Note Added: 0022490 | |
| Apr 20, 2012 10:00 am | Chris_McElligottPark | Note Added: 0022491 | |
| Apr 20, 2012 10:00 am | tigersfan | Note Added: 0022492 | |
| Apr 20, 2012 10:01 am | Chris_McElligottPark | Note Added: 0022493 | |
| Apr 20, 2012 10:03 am | Minotaar | File Added: output_log.txt | |
| Apr 20, 2012 10:04 am | Minotaar | Note Added: 0022494 | |
| Apr 20, 2012 10:07 am | Chris_McElligottPark | Note Added: 0022495 | |
| Apr 20, 2012 10:14 am | Toll | Note Added: 0022496 | |
| Apr 20, 2012 10:15 am | Toll | Note Added: 0022497 | |
| Apr 20, 2012 10:22 am | Chris_McElligottPark | Note Added: 0022498 | |
| Apr 20, 2012 10:25 am | tigersfan | Note Added: 0022499 | |
| Apr 20, 2012 11:06 am | Chris_McElligottPark | Note Added: 0022500 | |
| Apr 20, 2012 11:16 am | Chris_McElligottPark | Note Added: 0022502 | |
| Apr 20, 2012 11:18 am | Toll | Note Added: 0022503 | |
| Apr 20, 2012 11:18 am | tigersfan | Note Added: 0022504 | |
| Apr 20, 2012 11:20 am | tigersfan | Note Added: 0022506 | |
| Apr 20, 2012 11:21 am | Chris_McElligottPark | Note Added: 0022507 | |
| Apr 20, 2012 11:25 am | Chris_McElligottPark | Note Added: 0022508 | |
| Apr 20, 2012 11:29 am | TechSY730 | Note Added: 0022509 | |
| Apr 20, 2012 11:36 am | Chris_McElligottPark | Note Added: 0022511 | |
| Apr 20, 2012 11:38 am | TechSY730 | Note Added: 0022512 | |
| Apr 20, 2012 11:48 am | Chris_McElligottPark | Internal Weight | => New |
| Apr 20, 2012 11:48 am | Chris_McElligottPark | Note Added: 0022514 | |
| Apr 20, 2012 11:48 am | Chris_McElligottPark | Status | assigned => resolved |
| Apr 20, 2012 11:48 am | Chris_McElligottPark | Fixed in Version | => 0.971 (RC2) |
| Apr 20, 2012 11:48 am | Chris_McElligottPark | Resolution | open => fixed |
| Apr 20, 2012 11:52 am | Toll | Note Added: 0022515 | |
| Apr 20, 2012 11:54 am | TechSY730 | Note Added: 0022516 | |
| Apr 20, 2012 11:54 am | Chris_McElligottPark | Note Added: 0022517 | |
| Apr 20, 2012 11:55 am | Chris_McElligottPark | Note Added: 0022518 | |
| Apr 20, 2012 11:56 am | Hyfrydle | Note Added: 0022519 | |
| Apr 20, 2012 12:13 pm | Minotaar | Note Added: 0022523 | |
| Apr 20, 2012 12:18 pm | tigersfan | Note Added: 0022524 | |
| Apr 20, 2012 12:21 pm | Minotaar | Note Added: 0022526 | |
| Apr 20, 2012 12:23 pm | Toll | Note Added: 0022527 | |
| Apr 20, 2012 12:25 pm | tigersfan | Note Added: 0022528 | |
| Apr 20, 2012 12:27 pm | Chris_McElligottPark | Note Added: 0022529 | |
| Apr 14, 2014 9:28 am | Chris_McElligottPark | Category | Bug - Crash or Exception => Crash/Exception |
