View Issue Details
| ID | Project | Category | Date Submitted | Last Update | |
|---|---|---|---|---|---|
| 0021978 | AI War 2 | Suggestion | Oct 27, 2019 10:13 pm | Oct 31, 2019 1:51 pm | |
| Reporter | Asteroid | Assigned To | Chris_McElligottPark | ||
| Severity | crash | ||||
| Status | closed | Resolution | not fixable | ||
| Product Version | 1.002 ARSes, Instigators, and Tech Vaults, Oh My | ||||
| Summary | 0021978: Opt-in telemetry for various player actions | ||||
| Description | I feel like a lot of discussions assume players play a certain way based on a very small sample set. It would be very good for the game to have actual data of how people play in hand when making potentially controversial design decisions. I propose introducing opt-in telemetry (properly explaining the benefits for the game in a *small* sentence) and to gradually expand that to all player actions we'd like to get data about. (I'm personally interested in whether people use c-click and r-view and where, as well as how they control or manage their units (fleet shortcuts or just bounding box selection, do they order around small or large groups or every fleet in a giant ball?). | ||||
| Tags | No tags attached. | ||||
|
|
I can't think of any useful way to code this, or any particular way to actually gather the data. |
|
|
Seriously, Chris, it's not rocket science for someone with your experience of the industry. Why do you make me spell it out? To gather the data, it's not really different from coding achievements or stats, and people have been asking for both all three systems could tie together. Gather stats about gameplay and upload them somewhere in addition to displaying them to the player in the form of achievements and on the game end screen. For instance I suggested that we make the tutorials more hand-holding. You replied by saying you're not sure it's needed for this audience, but that's the problem, you don't have data on who your audience. So instead of wondering, you add some code to detect: - How many people finish the tutorials, and if not which one they drop out on, and in which specific step of said tutorial - Whether people actually play after doing the tutorials, and how long they stick to it More widely, you could add code to the base ArcenButton that logs its name when clicked. That log is uploaded and then we can do data crunching to figure out areas of the UI that go unused and figure out ways to let the player know about them. Are most people even using the fleet panel to swap fleets? Well now we'd know. Much more scientific than reacting to posts by a minority of people that are not always representative of the player base, and tend to pull the blanket in different directions to boot. |
|
|
I don't know about you, but I don't want my games watching me and reporting back to the game devs. This sort of thing makes me very uncomfortable. Even if it is opt-in. I don't want to be part of that. |
|
|
I actually gladly share this kind of info because I know it gives data to the devs to improve their game. There's nothing secret or personal about how I play AI War 2, especially since you anonymize the data when gathering it. This info gathering is the standard these days to the point games don't bother asking you about it. But we'd be extra nice and make it opt-in. |
|
|
Asteroid, just a friendly heads up that the way you write is often very condescending. Not just to me, but often to others as well. I'm unusually patient and try not to let it color my opinion when people are condescending to me, because it happens all the time. But you're making a habit of it, and because I feel a certain level of comfort with you I feel I can offer the advice that this won't serve you well in life. It's not what you're saying, or your arguments in particular, but how you're saying it. I've seen some folks on the forums note that you were being condescending to them, so I considered not saying anything, but ultimately that would not be me doing you any favors. I like you a lot and I think you're full of great ideas, so hopefully this doesn't come across wrong. All that said, when it comes to telemetry: yes, it's not rocket science, and yes, I've been around the block on this before. Believe it or not, I actually have a huge background in this, not that it matters. I used to run (after fully architecting and doing most of the code) a large B2B web service application. Because things were all server-centric, naturally we could log everything and thus DID. We had more telemetry than any offline game could hope to have, because people could not opt out (they were all employees anyway of larger companies who also wanted the telemetry, and it was purely business metrics so it wasn't like it was anything personal they were doing). Automated bug reporting was handy. But I also know what those are like in offline apps, and I don't really care to go down that rabbit hole. Because guess what: I've done that before, too. Now I'm the one being condescending, if lightly so. If I was having trouble getting feedback from players, or not seeing large numbers of bug reports (things going unreported), that would be a big problem and telemetry might be warranted. But as it is, there are more reports than we can possibly ever address, and the number only increases. If nobody can be bothered to report something, I guess it's not worth me fixing. It's not like we're sitting around with nothing to do. We're already having to choose what to address out of the overwhelming number of things reported. "Can't be bothered to report it" items are a nice way of self-sorting that out. And that's BUG REPORTS, which are concrete and simple by comparison to user behavior. Contrary to your assertion that it's not rocket science, actually tracking data and then figuring out what to do with it is incredibly hard. You have to look at the context of the data. I know this from my extensive time trying to optimize things like advertising campaigns, and from having all the data in the world in my past software as a service, but still not being able to draw great conclusions without sitting down with departments of clients and talking to their users. This was over an 8 year span,so I had plenty of time to form opinions. Also, not that I'd normally bring this up, but data management and conclusions (for good not evil) are a big topic around my house. And there is indeed a huge science to it, and even WITH having huge amounts of context on medical records (in the case of this situation), it's extremely challenging and requires subject matter experts, data analysts, and programmers working together to make anything even slightly meaningful. I'm talking about relatively-concrete things like figuring out risk factors for people coming into a doctor's visit -- to make sure that the attending physician pays extra close attention to something that might be relatively rare that this person might be at risk for, but that based on their history could make a huge positive difference in their life if caught early. The complexity behind this, despite the voluminous data and the huge amounts of known things that increase risk factors, is insanely huge. Whole departments are devoted to this, and my fiancee just finished up a second master's degree (post-MD) focused on data informatics of this sort. If I was developing a game like Smash Brothers or Overwatch, then I would definitely want telemetry to tell me which characters are used most and least, and what their win/loss ratios are. I'd need context of what their matchups are, but beyond that the context is not too notable. Though you do get fuzz in the data from unlike skill levels of players, etc, but still the sample size is large enough to make conclusions. But I wouldn't need any more telemetry than those pieces of data right there, because beyond that it wouldn't be useful. I'd talk to players. "Why don't you use Waluigi?" Etc. There are some notable postmortems that folks have done on fighting games recently that talked about perceived inequity of a class actually causing it to perform worse when it was numerically the same. In the case I'm thinking of, the fix was actually to make the weapon sound effect more powerful and bassy, and then suddenly player behavior changed -- without any stat changes -- and they became something like 30% more effective with that class. It was a cool read, and a great example of how telemetry is inherently limited, while also being super useful in certain circumstances. And here we come to the last point: I don't care about the telemetry, or what the balance is in a grand scheme in this game. If I was making a Zelda title, or a roguelike, I'd feel the same way. If I was making Smash or Overwatch or Starcraft it would be different. Let me explain: in Zelda, for example, there are some things that are notably more difficult than others. There are some korok seeds in BOTW that almost nobody will find. Telemetry would tell me that it's horribly unbalanced, and that almost nobody ever gets all 900 seeds. This would suggest that the developers should balance those out, or make it more inclusive. But the reality is that they shouldn't care: some stuff is meant to be more rare or hard, and is more fun if it is. There are certain shrines that have a crazy difficulty spike, and that's absolutely okay. That's part of the freeform nature of the game. Verbal player feedback on something that is un-fun is a lot more useful, because you can read between the lines. But if I get telemetry that tells me that people overwhelmingly figure out to just cook a ton of Hearty Durians and not bother with the truffles... do I care? Players feel clever, and I've never seen anyone complain about that even though it is "unbalanced" in some ways. Or with a roguelike, some runs are supposed to be harder than others. Luck of the RNG is a thing. You want to clamp off the extremes, to some extent, but sometimes those are also just the way it goes. If something becomes a persistent part of the discussion about the meta and seems problematic, then you tweak it. But beyond that, there are difficulty spikes all over the place and a big part of the game is for players to navigate that. You're mentioning things like the UI, sure, but this goes back to my work with SaaS user behavior and UIs. Or even advertising campaigns. I had all the data at my fingertips, but couldn't tell WHY people behaved the way they did, even with full histories of every click they took. Did they take the long way around the interface because of a problem with the UI, or some personal preference, or something they wanted to see on the way? Did they not use XYZ report because it wasn't relevant to them, or because it was too hidden? I dunno! For things like C-click and R-click, if people are using that a lot or a little... what do I care? If people are playing and having fun and feeling like they have the information they need, generally speaking, then I'm satisfied. I don't use R-clicking, because I don't care to minmax to that degree. I also recognize the bias that is inherent in the data. I don't use C-clicking all that much because I have most of those things memorized. Doesn't mean there is a problem with either mechanic, but if you had excellent telemetry on how I play, it would look like there were problems with one or both. Only way to find out the truth is to ask me. There will always be some tools that are used more than others, and eventually discussions will come around to them if it's a true problem that bothers people. Otherwise it might just be a thing that isn't used as much. The jewelry in BOTW isn't all that useful by comparison to other aspects of the game, but it's still neat that it's there. The fact that I had to decide for myself that it's not worth it actually does add a little something, and it comes late in the game. With AI War titles, there's an ongoing flow where players will start out probably reading the basics and not much more, and then experimenting -- most of them, anyway. And then they may graduate to reading more details where they feel like they need it, or they may just stick with what they know. It's clear from talking to people that it goes all the various ways. My bigger concern would be if there are lots of people reporting that they use R-hovering and then find that it's not actually useful enough for them but they want it to be. Can that be made better? That's a written discussion with them that needs to happen. Are there people who want R-hovering type functionality and complain that it isn't there because they didn't find it? Again, a written discussion needs to happen. Hopefully this all makes sense. The fact that I'm rejecting telemetry has a lot to do with the fact that I've had extremely good telemetry in the past, and know its limits. And that I have a background view into the medical side of things where results matter a lot more and there's more money and staff, and I see how much THEY struggle with it when the data is more voluminous and the conclusions are more simple ("patient is at an increased risk of X, so think about this during their visit and use your judgement as to what to do next."). |
|
|
Appreciate that you took the time to write such a kind and informative long reply. I do like you as well and have a lot of admiration for the way you manage this community and the loyalty you command, and I hope my actions speak louder than my dumb words in this regard. What motivated my tone, but doesn't excuse it or make it appropriate is that I really didn't like your initial reply nor that to some other bugs of mine you insta-closed, and I'm on edge because I really dislike several of your decisions and approaches to development. "If nobody can be bothered to report something, I guess it's not worth me fixing." I am in maximum disagreement with this statement especially when applied to tutorials, and it makes me feel our fundamental philosophies are so divergent that it'd be pointless for me to keep arguing, I don't see anything positive possibly coming out of it. Overall, my apologies for stepping out of line again. I guess I inconsciously am in a bit of a "burning ships" (which may make the front part fall off) or "burning bridges" mode because I'm not happy about the project status. I'm happy that the launch seems to be a success for you though and that you might be out of debt soon if projections come through, so there's a silver lining to all this. I might send more thoughts in private, this discussion on a public forum is a bit awkward. |
|
|
Thanks for the notes -- we've carried on the discussion more in private, as he had noted. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| Oct 27, 2019 10:13 pm | Asteroid | New Issue | |
| Oct 28, 2019 11:06 am | Chris_McElligottPark | Note Added: 0054061 | |
| Oct 28, 2019 11:08 am | Chris_McElligottPark | Assigned To | => Chris_McElligottPark |
| Oct 28, 2019 11:08 am | Chris_McElligottPark | Status | new => closed |
| Oct 28, 2019 11:08 am | Chris_McElligottPark | Resolution | open => not fixable |
| Oct 28, 2019 12:26 pm | Asteroid | Note Added: 0054066 | |
| Oct 28, 2019 12:36 pm | BadgerBadger | Note Added: 0054067 | |
| Oct 28, 2019 12:36 pm | BadgerBadger | Note Edited: 0054067 | |
| Oct 28, 2019 12:41 pm | Asteroid | Note Added: 0054069 | |
| Oct 29, 2019 10:56 am | Chris_McElligottPark | Note Added: 0054103 | |
| Oct 29, 2019 11:34 pm | Asteroid | Note Added: 0054142 | |
| Oct 31, 2019 1:51 pm | Chris_McElligottPark | Note Added: 0054206 |
