This project explores solutions to the current challenges with playtesting virtual reality games.
The immersive, solitary environment of virtual reality applications poses a huge barrier to developers looking to get feedback on their games; many best practices for playtesting PC and console games, such as directly observing users play the game and eliciting verbal feedback on specific design decisions, are no longer effective if the developers cannot see what the players are seeing. Besides the VR world, there are several other factors that contribute to the gameplay experience – use of controllers, physical comfort, movement in the physical world – which are impractical for playtest facilitators to pay attention to all at once.
Throughout the semester, I will explore best practices for playtesting VR games and develop a system to aid developers in facilitating playtests. I will experiment with different methods of documenting the gameplay experience, such as recording video footage of the players’ interactions in both the VR and physical world, creating a shared VR world to enable direct observation, or enabling players and notetakers to annotate confusing aspects of the VR world.
Today was Design Expo! I had a great time explaining my project to other members of the UM community. Unfortunately, my HoloLens demo did not work outside of the lab where I developed and tested it (I suspect due to server issues), but I was able to walk visitors through the video I created. I had some interesting discussions with people about improvements to the existing implementation and future directions and applications.
Points of Improvement
As I was walking visitors through the video of the sample use case, we noticed some user experience issues, such as overlapping labels and markers in the MR visualizations and difficulty in telling the players apart, as they were visualized with identical markers. There are lots of quick visual fixes I can make, such as stacking event markers on the y-axis if they're placed in the same position and coloring the line renderer of the robot's laser eyes to match the color of the generic markers (assigned by the web-based dashboard when it sends data to the HoloLens).
I discussed my project with my research advisor, who co-designed and developed the mixed reality analytics toolkit which I extended for this project. He encouraged me to think about how the replay functionality could advance the overarching functionality of the toolkit, in addition to being applied to specific use cases, such as playtesting. With this mindset, perhaps it would have made more sense to implement the replay logic from the dashboard side, for example, streaming the data to MR devices in regualted timesteps instead of sending it all at once and having the Unity package regulate the timesteps. This would enable people to see the visualizations replaying on the dashboard side or with in-situ MR visualizations. I had originally chosen to implement the replay functionality on the Unity side so a single developer could facilitate the playtest visualizations on their own, without having to switch between the dashboard and the MR device. However, I like the idea of making this feature more widely available and applicable to other use cases, so I think it would be a good time investment to re-engineer the replay logic.
Future Directions
My faculty advisor posed the idea of algorithmically determining important camera angles based on the tracked objects in the scene or data collected during playtests. This could be particularly useful in terms of increasing efficiency of reviewing playtests, as developers would have a starting point of where in the environment is most intersting to pay attention to, rather than having to watch carefully and replay a few times from different angles to make sure they've observed everything important.
Perhaps this is not a future direction for my project specifically, but I had a good conversations with two students in a game development studio about utilizing data analytics to conduct more effective analysis for desktop games as well. They weren't familiar with playtest tools for desktop games, and were excited about exploring ways to incorporate a similar tool into their games. It was really exciting to see other people showing enthusiasm for my project and brainstorm applications to other areas of game development!
In Summary
All in all, I had a good time at expo, despite the disappointment of my demo not running. It was nice to hear other perspectives on my work and get some ideas for future improvements I can make. I also gained an understanding of my strenghts and weaknesses in driving a project independently, and am excited to improve these weaknesses as I grow into a more independent researcher.
I'd like to thank my faculty advisor, Austin Yarger, my research advisor, Michael Nebeling, and everyone else who has given me advice and feedback throughout the duration of this project. I really enjoyed working on Project Friday, both the challenges and successes, and I'm excited to see how my research skills will grow through future projects!
It's hard to believe, but Design Expo is tomorrow!! To get ready, I had 3 main focuses this week: testing on the Windows Mixed Reality headset (and possibly creating a VR demo), debugging the features that weren't working from previous weeks, and preparing a demo for the expo.
Trying out the WMR Headset
Since the original scope of my project was "Playtesting in VR," I did want to see how the playtest analytics system would apply to a VR game. So, I tested an existing version of the scene out on the Windows Mixed Reality Headset. I found that I had to do some repositioning of elements in the scene to make them visible in VR (for the HoloLens, the origin is the headset, while in the WMR, the y-axis is zeroed at floor level). There was also an issue with the gaze tracking where the starting point of the robot's laser eyes was way below the floor, but the cursor position seemed accurately positioned.
I weighed the pros and cons of switching to VR for the demo, and decided it would be too much of a hassle to debug my existing code and make a good demo with controller interactions. I was also worried about visitors putting on a VR headset in the busy, chaotic environment of design expo – it would be pretty hard to control where they walk with so many people around us. So, I decided to stick with the HoloLens for the demo, as I'm much more comfortable troubleshooting issues here.
Debugging Pause Behavior
I've been struggling to resolve the issues with pausing for a few weeks now. To recap, when the application pauses and resumes, the animation of the update events (player and object movement) are out of sync and choppy as compared to the placement of generic event markers. Last week, I tried to resolve this by using an open-source TaskManager to pause and resume coroutines in a more structured manner, but had no luck. I tried for several hours again this week, testing behavior when I paused for different amounts of time / different variations in the milestone times, adding additional yield statements in the replay coroutines, and checked for any skipped data points, but again, no luck in resolving the issue :( My best guess at this point is that the 2 coroutines that handle marker placement and animations are not being stopped at the same time, resulting in the out-of-sync behavior. I really don't know what could be causing the choppiness in animations, but for the future, I would aim to rewrite this part of the code.
Another issue I had was with pausing when a player reached a milestone – the system would try to pause all milestones at once instead of when the player actually reached the milestones. This was a pretty silly oversight on my part, so luckily I coudl resolve this bug quickly! After making the fix, the replay behavior was actually a lot less choppy and successfully paused/resumed the playtest visualizations on milestones! However, after the first pause, I could clearly see that the animations were no longer happening on an accurate timeline – the app would pause on a milestone a few seconds before the player was supposed to reach it. With expo tomorrow, I don't have the time to look into this more, and wanted to focus more on creating a polished demo.
Design Expo Demo
I decided to create a simple HoloLens prototype where the user can click on a couple moving animals in the scene – a bunny and bat – which very slightly resembles the mobile AR game Pokemon Go. I added custom markers for playtest milestones and the bunny and bat characters, so the visualizations are less abstract. During design expo, depending on how busy it is and how interested the visitors are, I would have them record some data by clicking on the animals and walking around a bit. Then, I could sync the session data and have them view the MR visualizations of themselves playing the game. I will also have a few pre-recorded sessions that I can use to show the multi-session visualization and pausing on a single milestone. With the session I recorded, the out-of-sync behavior is much less obvious, although I definitely want to keep troubleshooting this later on.
Here is a video of the final demo! It shows the data collection and visualization for 2 playtest sessions in the demo scene described above.
I'm really excited to present my work at Design Expo tomorrow! The last few weeks of the project definitely came with some technical challenges, but I'm happy that most of the issues are resolved, and I'm hoping the demo will run well tomorrow :)
This week, I focused on fixing a number of issues I was having last week: null reference errors when attempting to restart the visualizations and a broken implementation of custom synchronization points. I was able to debug a large number of these issues, but there are still a few remaining to fix. I also drafted a poster design for the design expo.
Debugging the StopCoroutine Issues
Last week, my code was throwing null reference errors when I tried to restart the replay visualizations, citing the StopCoroutine function. I attempted to debug this a bit more this week, but felt like my method of keeping track of coroutines and their paused/running status was extremely complicated. I decided to look for open-source solutions to handling Unity Coroutines and found this TaskManager written by Ken Rockot. It provides a simple interface for managing the lifecyle of a coroutine, from pausing to unpausing, ending, etc. Bits and pieces of my code resembled functions from this task manager, so I decided to try it out and see if this simpler interface could eliminate any errors I may have caused. Fortunately, it got rid of the null reference error!
Custom Sync Points
I also spent 2 days debugging my custom synchronization point implementation from last week. In doing so, I discovered a pretty big flaw in my logic - I was assuming that all players would hit Milestone 1 before Milestone 2, when in reality, Player A could hit both Milestone 1 and 2 before Player B hits Milestone 1. I refactored my code to account for this, but it is necessary for the milestones to be completed in the correct order (a chronological order must be enforced for a single player's milestones, so they couldn't complete Milestone 2 before 1).
This change fixed the majority of issues I was having, but there's still something off :( It seems that when Player A hits both milestones before Player B, the program thinks it's pausing on Milestone 2 when Milestone 1 occurs, which results in it never pausing on Milestone 2. Additionally, I was hoping the new TaskManager would solve my issues with choppy playback after pausing / unpausing, but this behavior is still there. After looking into it a bit more, I realized that the players actually don't end up in the same location as they would if I didn't pause the program at all, meaning some data points are getting dropped as a result of pausing. Not sure what could be going wrong here, but this is one of my priorities for next week!
Here's a quick video I took of the current implementation. We can see Player A pause on Milestone 1 and wait until Player 2 also reaches it. Then, both Players pause for 3 seconds, and playback resumes. The choppy behavior I described is evident after this 3 second break.
Design Expo Poster Draft
Here is my poster draft! I still need to update the images later on when I finalize the demo scene this week.
Next Steps
This is the last week of development before the design expo!! My focus this week will be preparing the demo scene for the expo, hopefully on the Windows Mixed Reality headset with a few interactions. I'll be careful to not spend too much time getting this to work (especially considering the issues I was having in previous weeks) and will use the HoloLens as a backup. I will also not spend too much time trying to debug the issues I encountered this week, as I feel as though the imperfect implementation is still a good proof of concept. I want to focus more on setting up an example game so visitors at the expo can see the playtest analytics system work in a relevant context.
This week, I set out to accomplish three goals: implementing the remaining replay menu functions, pausing the replay visualization at custom synchronization points, and cleaning up the dashboard user interface. Unfortunately, debugging was a huge time sink this week, so I didn't get around to finishing all of these tasks :(
Replay Menu Functions
In previous weeks, I implemented play, pause, and restart functions which the user can trigger by clicking on buttons in the replay toolbar. I wanted to implement the remaining functions (step backwards, step forwards, and skip to end) this week. However, I discovered a bug in the restart function which I must have introduced with all my refactoring last week – a null reference exception is thrown when you click the restart button. I spent several hours trying to debug this, with no luck, as the error seems to be occurring in the StopCoroutine function, which is internal to Unity and therefore doesn't provide very descriptive console messages. Since I had limited time this week, I decided to move onto another task; hopefully taking a second look on a new day will be more successful!
Custom Sync Points
My main implementation goal for this week was defining custom sync points, or milestones, within the game, where separate player visualizations will pauses until they all reach the milestone. This will enable developers to compare performance across specific tasks within the playtest, rather than the playtest as a whole, which could vary greatly in time and performance.
To accomplish this, I created a new event type for the data analytics library: PlaytestMilestone. Developers can choose when to log this event, and the default recorded data will be the milestone name (ex: Task 1, Object Collection, etc), as well as the player's position and rotation at this time. At the start of replay, the system will aggregate the milestone events across all playtests and pause when the same kind of milestone has been reached. I check that all playtests in the sessions contain the same milestones, and if not, don't consider these for pausing the visualization. After all players have reached the milestone, I will either unpause the application after a few seconds, or let the user manually unpause using the replay menu.
Unfortunately, I'm still debugging this functionality, there's something not quite right with the way I'm aggregating milestone events by session and name, in order to pause across the same milestone. I plan to finish it up within the next day or two. Video to come soon!
Next Steps
My first priority will be finishing the tasks I didn't complete this week, especially resolving the null reference exceptions when restarting the visualization and finishing up the custom sync points functionality. Next I will focus on the following tasks, reiterated from last week:
11/21 – 11/27 Sprint:
Implement VR interactions and test thoroughly on the Windows Mixed Reality Headset (this is what I would be using for the design expo demo)
Prepare the design expo demo: present an example game and instrument the game to collect data from the environment, and present a strong use case.
Clean up the design / user experience of MR application (UI, clarity of events visualization, etc)
With Thanksgiving break next week, I'll put in lots of overtime to get this project to completion!
My main goal this week was to resolve the issues I was having last week with visualizing multiple sessions at the same time and implement an MVP version of the feature. As discussed several weeks ago, I aim to enable developers to analyze playtesters' actions across different versions of the game (perhaps a few weeks of development apart) and analyze trends within the same version of the game.
Debugging!
First, I set out to figure out why multiple robots were spawning when I replayed multiple visualizations. After many print statements and examining the data stored in the database, I realized that I was assigning many unintended events with the robot "custom marker," so I adjusted the code and recorded a few new sample playtests. Problem solved!
The next problem I encountered was a choppy, out-of-order motion for the player, although the object update events were smooth and looked the same as in previous weeks. I expected this issue from the start, as I didn't have any support on the Unity side or server side for distinguishing between different playtests. So, I re-engineered my visualization code – described below!
Multi-session Visualization and Time Syncing
I re-engineered my code to more effectively handle the data received the dashboard, which involved creating new data structures to organize the data by session and event-type and writing a couple of managers to simplify the workflow I previously had. I also logged a "playtest ID" in addition to the shared session ID assigned by the server, so we can distinguish between indiviual playtests within a sessino. To enable time syncing, I set the replay function to consider the "time since startup" of the application rather than the timestamp when plotting data points, so the replay visualizations would play simultaneously even for playtests recorded hours apart.
An improvement I'd like to make next week is setting custom sync points. We can imagine a scenario where a player took a long time to complete a task, whereas another player took a short time. In this situation, the replay visualizations would may longer be synchronized in a useful manner, so I'd like to be able to pause certain playtests until everyone reaches certain milestones in the game.
Here is a video of the multi-session MVP! At the start, we can see the default visualization mode, where events are color-coded by playtest ID. Next, I start the replay visualization, and the two robots represent two playtesters who walked through the game during separate playtests. The start times are synced, so developers can view the players' actions simultaneously and compare behaviors (here we see different routes taken around the room, fixating on the cube at different times, etc.)
One drawback of using this application on HoloLens was the small field of view – it could be difficult for a developer to anticipate where the player was looking/moving and reposition themselves accordingly. So, a developer may have to rewind and replay the visualization several times to make sure they caught all details. This makes me think the application is better suited for VR, where both the user and developer would have a more complete sense of the environment around them and could navigate the environment more quicky.
Next Steps
It's hard to believe, but the design expo is only 3 weeks away! I laid out a development plan for the remainder of the semester:
11/4 – 11/20 Sprint:
Custom sync points: developers can define milestones within the game (perhaps a completed task or collecting an item) and log an event; in the visualization mode, they can set the multi-session visualization to pause separate players until they all reach the milestone
Implement all replay functions: step backwards/forwards, and debug the restart function
Clean up dashboard look & feel and interactions
11/21 – 11/27 Sprint:
Implement VR interactions and test thoroughly on the Windows Mixed Reality Headset (this is what I would be using for the design expo demo)
Prepare the design expo demo: present an example game and instrument the game to collect data from the environment, and present a strong use case.
Clean up the design / user experience of MR application (UI, clarity of events visualization, etc)
11/28 – 12/5 Sprint:
Poster preparation for the design expo
Any overflow tasks from previous weeks
Visualizing data across different server sessions (this is a stretch goal, I think the current implementation of multi-sessions is a sufficient proof of concept)
I set out this week to continue refining the replay feature, as well as make some headway on visualizing multiple playtest sessions at a time.
Replay Iterations
One task I had remaining from last week was troubleshooting the input system issues I was having, so that I could get the replay UI working in MR. However, I'm still looking into getting a different computer to connect the Windows Mixed Reality headset, so in addition to the HUD attached to the VR display, I decided to implement a statically-positioned UI for testing on HoloLens. It was a little tricky to figure out how to use the older version of MRTK (lots of the old documentation has been removed after the release of MRTK 2.0), but in the end, I got it working.
In addition to the replay UI, I wanted to address some feedback I received at our last independent study meeting: it was difficult to get a sense for where the player was looking while they were moving. This was a particularly straightforward fix, as the data analytics library logs cursor position over time in addition to the user's position. So, I used a Unity LineRenderer to draw a path from the player avatar's head to the cursor position.
This video shows both the replay menu in AR as well as the gaze tracking. It's a bit hard to tell from the video, but in person, the gaze visualization was surprisingly accurate – the endpoint of the line would hit objects around the room, such as the walls and tables, and hit the cube at the exact time the ObjectFixate event occurred. I think it would also be very helpful to visualize the player's field of view so developers know which areas of the environment were actually visible at a given time; however, the field of view is device-dependent, so I need to think more about how to approach this.
Visualizing Multiple Sessions: Try #1!
I focused some of my efforts on the server and dashboard side this week, attempting to aggregate session data and visualize multiple sessions at once. Another member of my research team implemented an aggregating functionality for a previous project, so I spent some time familiarizing myself with that part of the codebase and integrating it with my basic version of the server. Aggregating is accomplished by setting a shared session ID for all new instances of the application on the server side. The dashboard also had to be reformatted to distinguish between different sessions (by color-coding and labeling). For the most part, the server and dashboard edits went smoothly – there are a few formatting fixes I need to make to distinguish more clearly between users and events on the dashboard. However, I ran into some issues on the Unity side with the replay function not being able to distinguish between sessions:
I wasn't able to resolve these bugs this week, so I'll need to prioritize this for next week. Even after fixing these bugs, the implementation won't be quite what I'm looking for, as each session will be visualized consecutively instead of simultaneously since later sessions are currently stored with later timestamps in the database. I plan to log an event called PlaytestStart, which could be triggered through voice command, or possibly remotely by the developers, and sync the playtests based on this event time.
Next Steps
Clearly, I have some debugging to do with visualizing multiple sessions. I also noticed a bug only on HoloLens where after pausing the replay, the player's movement is very choppy. After resolving these issues next week, I will continue working on synching up sessions according to the PlaytestStart event.
This week, I set out to re-engineer my current version of the replay functionality, in order to address the differences in visualization behavior when running the application within the Unity editor versus on MR devices. Additionally, I worked towards a "minimum viable product" version of the replay feature, refining some aspects of the animation and user interaction.
Refactoring the Codebase
To recap from last week, the main issue with the previous implementation was attempting to access visualization makers before they were instantiated, a problem only apparent when testing on MR devices and not within the Unity editor. Re-engineering the codebase required a change to the user's workflow for replaying a playtest, as I could no longer assume that visualization marker game objects would have been created at the time I needed them to calculate the marker's path for the replay function. Now, the user must say "Visualize" in order for the playback UI to appear, then can use controllers to play, pause, step backwards, etc. Once the markers are created, the code maintains a list of the gameobjects, so that the replay function can access them later on without having to wait for new markers to be created. As another precaution, I sorted the list of events returned by the dashboard by timestamp and dispatched them in this order, to make sure the earlier-timestamped event markers would be available for use by the replay function, even if all markers are created asynchronously.
Refining the Replay Feature
After re-engineering my code and getting back to the replay functionality I had achieved two weeks ago (now, working both in the Unity editor and MR devices), I spent some time refining the feature and getting it up to an MVP level. First, I revised my animation function in order to achieve a smoother, more realistic movement for tracked objects. I made the decision to use a replay marker entirely separate from the visualization markers which are created and stored when the user issues the "Visualize" command, in order to have easy access to the the visualization markers' time and transform data. I moved and rotated only this replay marker, rather than showing and lerping each individual visualization marker at the correct timestamp, which resulted in a much smoother look to the animation.
Additionally, I implemented a few more playback controls, including pause and restart. These functions can be triggered when the user selects the corresponding button on the UI. However, this week I had issues connecting the Windows Mixed Reality headset to the laptop – sometimes, the headset refused to connect, and other times, the Unity editor froze when testing. So, I recorded the below video in the Unity editor, and will troubleshoot and look into getting new equipment for next week.
This video shows the new animation function, as well as pausing and restarting the visualization replay.
Next Steps
I noted down a few improvements to the replay feature that I will return to and implement later on if time permits:
Implementing the step forward and step backward functions – this requires slightly refactoring my replay implementation, so I put it off for now
Adding a "blur" to tracked objects so the user gets a better sense of where the objects were placed a few seconds before (I experimented a bit with adding a trail renderer, but it didn't look great at the time)
Creating a more realistic animation scheme for the player object – customizing rotation, in particular, so that only the head rotates along the y-axis rather than the whole body
However, I'd like to move onto visualizing multiple sessions at a time, in order to make sure I have enough time to implement an MVP version of this feature as well. My first goal for next week will be to ensure that user input is working on the Windows Mixed Reality headset, and troubleshoot any issues with functionality or user experience. Next, I will make some headway on visualizing multiple sessions at a time, which potentially involves delving into the web-based dashboard's codebase and adding support for synching multiple sessions with MR devices.
My highest priority for this week was addressing some of the issues I had last week with visualizations appearing in Unity but not on the HoloLens, as I was bottlenecked in the area of testing the MR interactions until this problem was solved. Unfortunately, debugging took 15+ hours this week, and I found a need to majorly restructure portions of the codebase and revise my approach to the replay function. In doing so, I redesigned the user's interactions interactions with the application, shifting more autonomy to the user rather than the dashboard operator.
Debugging Process
I started with developers' first instinct for debugging: print statements! However, the HoloLens poses some issues with this strategy, as the device isn't connected to a desktop, so it's difficult to see error messages. To get around this, I made use of a Mixed Reality Toolkit feature called the DebugEventLog: this prefab can be easily to any Unity project and debug/error messages are printed out on this MR console.
The DebugEventLog proved to be very helpful for finding differences in behavior between running the application in the Unity editor vs. on the HoloLens; however, the debugging process was extremely time consuming, as I had to rebuild and redeploy the application each time I added new print statements, which takes several minutes. I quickly found one issue – I was using an old recorded session with datapoints lacking a new property which I was checking for – for some reason, these error messages didn't show up in the Unity editor. After this point, the data resumed visualizing and replaying like it had the previous week. One bug down, but many to go!
A new issue presented itself – the lerp function was not working, and the application was throwing lots of null reference errors that were also not showing up in Unity. It took several hours for me to debug this issue by adding print statements after every line of code in my replay function. I finally realized that an event dispatcher was being used to place the markers in the MR environment, and while this appeared to happen instantly in both the Unity editor and on the HoloLens, it actually took much longer for the HoloLens to instantiate event markers. I assumed the markers had been instantiated and was attempting to access them too early, resulting in the null reference errors.
While I was happy to have found the issue, it unfortunately required a lot of refactoring of the existing code, as well as the new code I wrote. I started working on decoupling my new code from the existing visualization code by writing a separate replay visualizer view – hopefully this will make debugging in the future easier. I also revised my strategy for the replay function – I now require that the user uses the "Visualize" voice command to create all event markers before attempting to replay the data. This way, I can be absolutely sure that the HoloLens application isn't attempting to access game objects too early. Additionally, only the first marker in the time selection will be used, and will be animated to move to the locations of all other markers. This should hopefully improve the performance of the application.
With my remaining time this week, I implemented a large majority of these described changes, but didn't have the opportunity to test yet.
Overview of User Interactions
In order to give the MR observer more autonomy of the visualization features, I decided to implement a simple play menu which only appears when the replay visualizer is active. I also decided to revise the workflow I implemented last week, in order to prevent the issuse I described earlier – now, the user must follow these steps to replay game data:
Use the voice command "Visualize" to show all data selected on the dashboard.
Press the play button in the MR UI, which clears all visualizations and replays from the beginning of the time selection.
Use the pause, restart, rewind, and fastforward buttons to replay the data again.
At the end of the replay, the view will return to the generic visualization (same as when the user first said "Visualize"). This can also be accomplished by pressing the skip to end button on the UI.
Use the voice command "Clear" to hide all event markers.
Next Steps
Again, my highest priority for next week will be to get replay visualizations working again on MR devices. Next, I will focus on integrating the replay UI interface with the replay feature, aiming to implement pause and restart functions, as well as rewind/skip forward functions if time permits next week. If all goes well, I hope to complete a basic (perhaps less than perfect) version of this replay feature so that I can move onto exploring visualizing data for multiple sessions at a time, and refine the replay function later on.
My goals for this week were to further develop my initial implementation of the MR replay feature, as well as clean up the code base to make development easier in the upcoming weeks. I started out by implementing a way to easily switch between two modes for visualization – visualizing all events at once and replaying events at each timestep – through voice commands. The default commands, "Visualize" and "Clear," display all the MR events selected on the web-based dashboard and clear the visualizations, respectively. Saying "Replay" starts the step-by-step visualizations, clearing any visualizations which were displayed before.
When working on this functionality, I realized there's a need to redesign the visualization workflow, as the current version of the data analytics library might require a dashboard operator to be heavily involved when replaying a playtest. Especially considering playtesting VR games, it would be difficult for a single developer to manage the dashboard actions (synching the dashboard with the VR device, filtering the data, and selecting only the desired range of events). In upcoming weeks, I'd like to explore ways to give more autonomy to the VR observer rather than the dashboard operator, perhaps through adding playback controls (play/pause/rewind/fastforward).
Technical Experiments
I attempted two technical experiments this week: 1) animating the replay visualization to convey a more realistic movement of tracked objects and 2) displaying custom event markers corresponding to object prefabs.
Animating the Replay Visualization: I aimed to add a simple movement animation to the event markers by lerping each event marker to its successor's position. This required refactoring the data analytics library's code to split up events by type (for example, separating player update events from object update events), as the original implementation simply aggregates and returns all selected events. I also disabled event markers after a few seconds, so a small trail of markers would be left without overwhelming the observer.
Displaying Custom Event Markers: Additionally, I aimed to make the replay visualization more realistic through showing event markers which correspond to the actual objects being tracked. To do this, I started by implementing a ScriptableObject for the player and attempting to log the ScriptableObject along with the status update events (which record the player's movements). However, I found that there was no easy way to serialize the ScriptableObject, so I resorted to logging an additional string corresponding to the player prefab and storing a mapping from string to prefab in another script. This implementation was quick and dirty, so I definitely need to refine it over the next week, and consult more experienced Unity users to see if there's a cleaner way to accomplish this function.
The video below shows the results of the two technical experiments. Unfortunately, I must have broken something with synching data with MR devices, as I couldn't get any data to show up in the HoloLens. I'll troubleshoot over the next week, but for now, I recorded the video in the Unity editor.
Overall, I was happy I could make some progress on this feature, but there's a lot of improvement. The animation was choppier than intended (but I sort of liked that the observer can get a clear sense of the timesteps), so I'll work on smoothing the lerp function. Additionally, I need to refine the rotation lerping to make the player's motion look more realistic; the robot's whole body doesn't need to rotate, just the head, which could be solved through logging more data in the status update events and adding some additional logic to the custom marker display function.
Next Steps
The highest-priority goal I'd like to accomplish this upcoming week is mapping out a project plan and clearly define what I'd like to accomplish by the end of the semester (only 1.5 months away)! Implementation-wise, I will continue cleaning up the code and refining the replay functions I implemented this week. I definitely need to troubleshoot some of the issues I was having with synching data before I implement more features. After fixing these issues, I will aim to make the animations more realistic and implement some of the observer controls, such as pausing and rewinding.
After considering the feedback I gained from the interview last week and consulting with academic mentors, I decided the goals I was most interested in addressing were 1) assessing the impact of specific changes from playtest to playtest, and 2) enabling developers to replay the player's actions during critical moments in VR. Given these goals, I settled on a new direction to explore: collecting data from playtests and exploring visualization methods which may help developers better understand points of frustration or confusion in their games. By the end of the semester, I will aim to extend this library for use in a game-development context. In addition to replaying critical moments from the playtest, I will design data visualizations to help developers assess metrics, such as input accuracy, time taken on tasks, etc.
To reduce the barrier of entry in terms of setting up the data collection and network infrastructure, I decided to utilize a mixed reality data analytics library developed in a related research project. This library can be easily integrated with MR projects in Unity and enables developers to record pre-defined events, including user events (positional updates, voice commands, input, etc), object events (creation, positional updates, gaze fixation, etc). I spent some time this week cleaning up the code and setting up standalone components for my specific project, including separate versions of the server and the data visualization dashboard.
After doing some project setup, I implemented a quick prototype of the replay functionality. In the original version of the library, recorded data is logged to a database and visualized on a web-based dashboard. When a user syncs the dashboard with running MR apps, the data is sent back to the MR devices and visualized in the MR environment all at once using predifined markers and labels. I familiarized myself with the visualizer code and reengineered it to "replay" the data through displaying the markers at specific time intervals, rather than all at once.
This week, I recorded the video below on a HoloLens, and will implement VR controls on the WMR next week. The video shows the replay functionality after I walked around the perimeter of the room and fixated upon a moving cube.
This prototype was pretty straightforward to create, but has lots of room for further improvement to fit a game developers' needs. I looked into developing a higher-fidelity version through logging users' and objects' transform data through using scriptable objects. In doing this, we could visualize the actual objects – for example, the position and orientation of player's heads and arms – instead of using generic markers. Next week, I will aim to make more headway in developing this higher fidelity prototype, as well as develop a more concrete system design in terms of user interaction and defining critical moments in terms of data collection.
This week, I conducted an interview with a professional VR game developer in order to build a stronger understanding of what playtesting strategies developers have tried, and where there might be room for innovation in this area. Below are a few key takeaways:
The most important requirement for playtesting in VR is seeing what the player is seeing. The developer I interviewed accomplished this through casting videos to a desktop monitor inside the room, as well as setting up a live stream so other members of the development team could watch the playtest. He is primarily interested in capturing what the player is doing inside the game, but also records the player's physical movements through additional cameras set up in the room.
At this point in the interview, I floated my idea about having a 3rd-person observer present in VR during the playtest, and possibly annotating the scene as an alternate form of notetaking. The developer expressed that it could be useful for specific types of games (story-based, in particular), but not necessarily for all game genres.
He also had concerns about the technical overhead that would be required to instrument existing systems with this capability. While casting can pose issues with frame rate and doesn't fully convey the immersive environment, it's still "good enough" to give the developers a window into what the player sees.
This developer's primary goal during playtesting is to assess the impact of specific changes from playtest to playtest on the gameplay experience. In particular, he aims to track players' interactions with virtual objects, both in a quantitative and qualitative manner, in order to determine points of frustration.
He listens to the players' verbal feedback to infer their emotions. Since the player's face is covered by the headset in VR games, developers are limited to reading emotions through laughter, exclamations of surprise or confusion, or frustrated remarks.
Based on the feedback I obtained in the interview, I felt like I needed to rethink the 3rd-person observer/annotation solution I developed in previous weeks, in order to make sure the final solution I develop could be beneficial to VR game developers and fill an existing gap in this area. I spent a considerable amount of time analyzing the goals and pain points the developer mentioned, brainstorming new solutions to mitigate pain points I hadn't previously considered, and talking through the viability of these solutions with an academic mentor who researches in the AR/VR space. Here are a few of the ideas I'm considering moving forward with:
Overall, progress felt a bit slow this week, but I'm hoping that talking to a few more game developers and academic researchers will give me a clearer sense of direction in the upcoming weeks. As far as next steps, I plan to look into setting up a multiuser VR experience and saving game states next week, as I still see 3rd-person observation and replaying critical moments of the game as viable solutions.
Brainstorming Solutions
This week, I spent some time thinking about aspects of playtesting that are particularly difficult to carry out in VR, as well as brainstorming ways to mitigate these challenges. In particular, I focused on the different roles involved in playtesting: players, notetakers, and observers (perhaps other team members or expo visitors). In transitioning from a desktop game to VR, the notetaker's capabilities change most drastically, as they no longer have a real-time window into the player's experience.
Considering these challenges, I wanted to explore the notetaker's capabilities if they could take notes in VR, through sharing the VR world with the player during the playtest.
A few ideas I came up with:
Annotating the VR scene through painting and recording audio comments. This would enable the notetaker to see everything the player is doing while taking notes on the player's experience simultaneously. Using voice recording instead of typing would make the notetaker's work much more efficient.
Saving the game state during key interactions.This could take the form of screenshots, or perhaps a snapshot of the VR world at the time that developers could walk through at a later time. For developers who couldn't be present for the playtest, it could be very helpful to "replay" the player's actions in VR.
Enabling a "follow me" mode where the notetaker can shadow the player.This could make the notetaker's job much easier by taking away the need for them to navigate the VR scene themsleves, allowing them to focus all their attention on taking notes. However, I could foresee this causing nausea or sickness for some users.
First Technical Experiment
To make some headway on determining the feasibility of these possible solutions, I conducted a small technical experiment: implementing a VR painting feature to annotate the scene.
I started by deciding on which VR technologies to use. I was eager to try out the Mixed Reality Toolkit v2, which was just released a few months ago, so I started out developing with Unity v. 2018.4.9f1 and MRTK v2.0. However, I quickly ran into problems using the toolkit with the Unity editor - I could no longer fly around in game mode or rename objects in the Hierarchy panel (possibly a side-effect of developing on my MacBook Pro). Unable to resolve these issues, I decided to move on and ended up using Unity 2018.3.14f1 and MRTK 2017.4.3, which I've found mostly reliable in the past.
I faced a similar difficult decision in terms of which hardware technology to use. I'm a tiny bit familiar with Windows Mixed Reality Headsets, but also have access to the Oculus Rift S through the Duderstadt Visualization Studio at the University of Michigan. I visited the Visualization Studio this week, but found I didn't have permissions to download the correct version of Unity. For the sake of completing this small experiment, I used the WMR Headset, but I plan to attend an orientation session at the Visualization Studio next week in order to learn which device would be best for my project.
To get familiar with developing for the WMR, I completed the Mixed Reality 213: Input tutorial. This was really helpful in understanding how to process input from the controllers and how to run apps on the WMR in general. However, I ran into some issues with the starter code for the project - certain parts of the code were incompatible with the version of MRTK I was using, and I couldn't get the color picker to be visualized in VR. After working to resolve the compilation errors (but not the color picker issue), I added some game elements to the scene, such as some walls of a maze and collectibles.
The video below shows a possible use case for the painting tool: annotating the scene when you're confused by elements of the game, such as a door that you don't know how to open or powerups you don't know how to collect.
Overall, I was happy with my first technical experiment and see a lot of potential annotation as a method of documenting playtesting in VR; the interaction was fairly intuitive and quick to learn. There is still plenty of room for improvement, however; there could be a more efficient way to navigate around the room (teleporting or the "follow-me" method I described above), and it would be useful to have more control over when to use the painting tool. This experiment boosted my confidence in terms of the feasiblity of my ideas, but I'm a bit worried about the large learning curve I faced in getting this simple project set up - I should expect similar challenges in upcoming weeks as I decide on the VR technologies I'm using and familiarize myself with them.
Next Steps
My next steps for this upcoming week include:
Consulting with mentors and staff at the Duderstadt Visualization Studio to finalize which hardware and software technologies to use moving forward
Refining the current implementation of annotating in VR, adding UI elements to toggle the paintbrush, changing color, etc.
Stretch goal: making some headway on creating a multi-player VR experience and visualizing other players (I foresee this as my biggest technical challenge in the project)
To gather knowledge about the research area of playtesting in VR, I conducted an informal literature review of academic publications from the ACM Digital Library and Google Scholar. I started by simply compiling a list of any titles which seemed relevant to the research area or preliminary solutions I brainstormed.
After collecting this initial list of publications, I did a quick reading of all of the papers, focusing on the abstracts, metrics, usability evaluation methods, and results. I filtered the list to only include the most relevant publications, and summarized my findings below:
Carey et. al. developed a protocol for measuring player empathy in VR games. They found that the following methods were most effective for recording qualitative evidence of players’ empathic responses:
Utilizing video streaming and voice recording to observe the player’s first-person experience
Conducting open-ended rather than semi-structured interviews, in order to give the players flexibility in processing their emotional response to the VR experience
Using the Wheel of Emotions and the Other in the Self Scale to identify empathic responses, which can be used as a starting point for later interview questions
Karre et. al. conducted a literature review of 36 publications detailing usability evaluation methodologies which industrial researchers currently employ, in order to move towards a more standardized set of VR development and evaluation practices. Some key findings:
The most common usability evaluation method is conducting a controlled experiment followed by a post-questionnaire. Walkthroughs followed by feedback sessions, as well as heuristic evaluations, are also utilized.
The most common metrics recorded are qualitative, and therefore, subjective in nature: effectiveness, efficiency, and satisfaction with the VR experience. Few industrial researchers collect biometric data, which allows them to partially-automate user tests.
Suggestions resulting from usability evaluations are often difficult for industrial development teams to integrate into the current product, as VR development is expensive with regards to time and asset creation.
Playtesters are frequently asked to take a post-survey to assess the immersive nature of VR experiences; Schwind et. al. studied the effects of integrating these presence questionnaires into the virtual world, in order to prevent players from having to exit and re-enter VR. Playtesters completed both abstract and realistic first-person shooter levels, then completed a presence questionnaire either on paper or directly in VR. The team concluded that completing presence questionnaires in VR does not affect the measured sense of immersion, but may reduce the variance in participant responses.
Sutcliffe et. al. conducted two case studies and an expert evaluation of previously established heuristics for designing virtual environments. These heuristics include natural engagement, compatibility with the user’s task, natural expression of action, close coordination, realistic feedback, faithful viewpoints, navigation and orientation support, clear entry and exit points, consistent departures, support for learning, clear turn-taking, sense of presence. The evaluators and research team concluded that this heuristic system is an effective tool in assessing the usability of VEs, but designers should select only those heuristics which are applicable to their application (ex: entry/exit points and consistent departures are more applicable to desktop VR apps). Unlike the evaluation of Nielsen’s heuristics, there was no evidence of an increasing number of errors found with more expert evaluators.
Gabbard et. al. present a formal design and evaluation structure for applications involving virtual environments. This structure includes:
User task analysis: identifying critical tasks that users need to accomplish and dependencies between tasks through contextual inquiry and ethnography methods
Heuristic evaluation: using guidelines crafted by experts in the field to assess the usability of a system
Formative user-centered evaluation: performing an iterative process of usability evaluations, consisting of user interviews, walkthroughs, and other methodologies, then refining elements of the application based on user feedback
Comparative evaluation: assessing your application based on current case studies from industry and research
Bowman et. al. investigated the use of testbed evaluations, which tests a complex combination of tasks and independent variables, to analyze user performance and satisfaction for two virtual environment interaction techniques. The authors conclude that while testbed evaluations are time-consuming to implement and produce a plethora of data that is often difficult to understand, they are useful in quantifying usability factors and painting a more complete picture of complex virtual systems.
My research on the challenges of playtesting VR games could have the most impact on the video game industry, which has a projected revenue of $128 billion (Statista, 2019) and customer base of 2.7 billion (Statista, 2019) for 2021, globally. However, if we generalize playtesting to usability evaluation of XR experiences, this research has the potential to impact the XR sector as a whole, which has roots in the healthcare, automotive, education, and law enforcement industries, to name a few (Virtual Speech, 2019). The VR industry has a projected market size of $160 billion by 2023 (Statista, 2019).
The "big players" on the industry-side of this research area include companies which invest in VR game development as well as creating VR technology, including software and hardware components. Oculus is at the forefront of the VR gaming industry, unveiling several popular VR headsets in recent years and maintaining a large store of VR games, among other applications. HTC Corporation and Valve Corporation are also extremely prominent in the gaming industry, demonstrated by the popularity of the HTC Vive and SteamVR (GameDesigning, 2019). Another player in the industry is Unity, whose game engine – compatible with a wide range of VR hardware – is used to develop more games than any other technology on the market (AIthority, 2018). Additionally, MR quality assurance companies such as Applause and iXie Gaming have the potential to break into the industry in future years as the need for rapid user feedback increases with consumer demand for VR content.
Several products on the market aim to address the challenges with facilitating playtests for VR applications:
Applause: this quality-assurance company utilizes crowdsourcing, calling upon a network of software testers to evaluate clients’ applications. Claiming that “crowdtesting is the only viable solution to AR/VR testing,” Applause provides testing services specific to MR applications, including evaluating hardware and motion sickness.
iXie Gaming: similarly to Applause, iXie Gaming offers quality assurance testing for VR applications, focusing on accessibility trends across demographics and testing on a wide variety of VR devices.
HTC Vive Pro with Precision Eye-Tracking: while past iterations of this headset are primarily used for gaming, the eye-tracking feature of the HTC Vive Pro could potentially play a huge role in playtesting VR applications in the future.
Airtest: this automated testing tool is compatible with several game engines, including VR games built in Unity. Automated testing could help VR developers save time in identifying functional issues with their games, rather focusing on usability and accessibility during their playtests.
Twitch: while Twitch is not typically thought of as a playtesting tool, the streaming service provides a small window into what VR experiences actually look like for the player, giving it potential for use in user testing.
One major audience I’m aiming to impact through this research project is VR game developers, whose primary concern is creating quality game experiences for consumers. A major challenge that game developers face is short development timelines, caused by consumers’ demand for new games and the desire to get the upper hand over competing game companies, who could potentially be on the verge of rolling out a similar game. The resources available to VR game developers greatly vary on the size and stature of the company; prominent companies such as Oculus or Microsoft may have plenty of funds to dedicate to research, while up-and-coming companies likely do not.
Additionally, I believe my research will also be of value to academic researchers who focus on mixed reality systems. Currently, many MR researchers aim to make contributions in the area of MR development standards, including world design, usability evaluation, and debugging; these contributions have the potential to impact industrial developers who create consumer-facing products. Academic researchers face the challenge of limited resources, as most universities do not have nearly the amount of funds at their disposal as companies in the MR sector may have. This often limits researchers to assemble small development teams, which could greatly benefit from a more efficient process of testing MR systems.
