We had the original timeline of having an alpha by week 7 and a beta by week 8. Week 8 is when we were going to send out beta builds to New Intelligence’s client base (our end users) and start to gather analytics and improve the apps usability based on feedback. But unfortunately this never occurred, not even a little bit later.
There were some forms of play testing though while we were in the paper prototype phase. At some point the team and I were at a point where we were in between transitioning from marvel app to unity. In this transition I had already prototyped some menu stuff to get a feel for what it could look and feel like. This was much different to the menu stuff that we had within Marvelapp. During this transition, each of the team selected a section of the app that they’d prepared and paper prototyped (on post-it notes) each possible screen that a user interaction could lead to. It wasn’t assessing the testers knowledge of the app or assessing them at all. It was solely to asses the apps usability and how people interacted with it, to watch what they thought they could interact with.
Sorry for not having photos of the paper prototypes, they must have gotten lost in the fires. One of the most prominent issues was originally when I made the PROSPECT model unfold in a circular motion, I had a middle circle there to help me position the exact location of where the PROSPECT letters would sit. It didn’t look terrible and it didn’t look not (!)terrible. But every user that tested the paper prototype at one point or another tapped on that centre circle. It did nothing, and after watching every user tap on it made me realize it’s just a distraction and the PROSPECT model doesn’t need a center hub to rotate around,
There was also the point in time that we were far enough in the design process and marvelapp that we had a scheduled Skype meeting with NI. The Skype meeting was to give them a build of the marvelapp that they can run on their phone and to play test it. Marvelapp didn’t allow us to have the functionality that we wanted of the UI at all but allowed us to get a feel for layout, spacing and transitions. The skype meeting with NI allowed us to not only present what we had so far but gave them the opportunity to critique what they liked and didn’t like. But at the same time, run through the application with them and watch how they used it.
In the case that we would have been able to gather analytics on alpha and or beta builds it would have been awesome to gather:
- How long each user spent on each question.
- The frequency of which answers were chosen.
- The percentage of which users got answers right and wrong.
- How long they spent looking at feedback screens.
Seeing how long each user spent on each question might not give very specific information but would allow us to identify an average completion time and understand which questions are more time-consuming or difficult to answer.
The frequency would allow us to detect what users think the more obvious answers are or which they think are the correct answer.
The percentage of right and wrong would help us to determine if the way the question is being presented is too difficult or too easy. Or the users are having difficulty finding right answers, enabling us to iterate on design and aim to not just test retention but reform questions in ways that still do as intended but in a more decipherable way.
Detecting how long they spent on the feedback screens (in comparison to how many times they’ve completed that question) would be the most intriguing in my opinion. If the users spent a lot of time on the feedback screens (I’d hope) it’s because they’re actually reading the reasons of why the answers were right or wrong. If they’re practically skipping it, did they do it by accident or do they just not care? Because if they don’t care, this app is pointless. The point of the app is to give opportunities to those who have gone through New Intelligence’s training course and practice those skills whilst outside of work times. If they don’t care about the app, do they care about the training they’ve taken? Or will they even put it to use?
Until next time –