Continuing a series of posts on all things ‘QA’ here at Realmac HQ, I’ll be sharing a little about how we like to make the most of our issue tracker, as well as using the tickets logged within it to efficiently create detailed test plans.
If you’re not already using a system to keep track of bugs, I highly recommend looking at a service like Lighthouse, a system in which we also manage projects in - far from using it as just a bug tracker. We sweat the details to ensure we ship great apps - we also place great value on being efficient behind the scenes too. Playing down the value of getting the most out an issue tracker could cost a lot of time that would otherwise be better spent crafting the actual app!
Within an ideal ticket, duplicate bug reports are avoided, concise steps to reproduce a bug are detailed by the first person to experience the issue, and explicit explanation of expected behaviours for new features should always be present. Of course, that is the ‘ideal’. It’s also important to tune and refine the ticketing process, striking the right balance between getting enough pertinent information before taking up too much of the bug raiser’s time.
We’re currently using a fairly common template to log bugs within Lighthouse (based on Apple's own bug reporting best practises) where we request a few specific fields of information that’s used to look at how and why a bug is present.
We could request much more information by default but, as mentioned, it’s vital that logging a bug doesn’t become a chore - keep the process quick! For example, have you ever used an app, particularly during a beta phase, and come across a seemingly obvious bug and thought “the developer must know about this already, I won’t waste the time taken to log it when it’ll probably be a duplicate.”? If the bug wasn’t already noted as being a known issue in the app’s release notes it probably hasn’t been raised yet. It’s natural to zone in on particular areas when working on projects but failing to step back to view the app as a whole poses a risk of missing a bug that, perhaps, would have been seen after stepping back from the project.
If I weren’t in a position where time can be dedicated specifically to testing, I’d want to ensure that any ad-hoc or beta testers are made aware of the fastest path to raise a bug, so that even the small or apparently obvious bugs may still be reported, avoiding leaving it to chance at being picked up. Providing clear steps on how best to raise an issue within release notes will help encourage a higher response rate.
Of course, this process can be made even more straightforward by making use of the extremely useful HockeyApp where app crash reports are automatically collated from an app, whether or not they’re in beta or the App Store. We’ve recently integrated Lighthouse, our bug tracker, into HockeyApp to allow for quickly creating tickets which automatically include all the information gathered and organised from within HockeyApp.
Fast-Track Test Docs
For every ticket raised in Lighthouse, be it for a bug, feature, view or behaviour in an app, I add a test case to a test plan that’s specific to that particular app or project. I find that by using the existing tickets from within Lighthouse, I can fast-track the addition of new test cases to a test plan, to both save time writing them from scratch as well as ensuring as much of that app as possible will be tested.
As mentioned previously, the Lighthouse tickets I use as a starting point for a test plan should already include the required input (steps, files or gestures) to perform an action as well as the expected output i.e. the resulting behaviour from our input. Likewise, with bugs found during development, tickets should also contain steps to reproduce the issue as well as any other anomalies to look out for when testing. This is essentially the core information that makes up each test case.
As new tickets are assigned over to Quality Control I check the ticket and create the new test case within a test plan, had a prior test case for the issue not existed. Our test plans are hosted on a private Wiki here at Realmac HQ (you can find out a little more about our network setup, including the Wiki, in this post by Keith).
Unique ticket IDs that link directly from the test plan back to the Lighthouse ticket are also added to each test case. This makes checking for regressions much more efficient as I can quickly check the build number of an app where a reoccurring issue was fixed previously. Integrating commits from Beanstalk within tickets means, in theory, it’s possible to pinpoint the code responsible for a bug within seconds. This, coupled with the inclusion of ticket IDs in release notes, saves time previously taken up by inadvertently creating duplicate tickets.
I’ve recently been looking at further ways to create effective test documentation. For example, creating visual flow-documents for some of our apps, where every view and the connections between them are laid out. As well as flow-docs, I’ve looked at MindMapping, a great way to check that there’s no areas left unchecked within a test plan. A nice example of an iOS testing MindMap can be found here, created by @noir and @lelchuk.
An Evolving Process
Everything can always be improved in some way, especially in QA. More detailed test cases can be created and added to a test plan, new bugs will always be found. That’s a fact that can take time to get used to. In his recent post on the human impact of bugs, @jury sums this up well:
“Realistically, we are never going to fix all of the bugs in our software. Anyone who tells you they don’t ship until all the bugs are fixed is either lying or naive.”
Not only is the fixing of the little bugs a never-ending process, but so too is the refinement and evolution of all the steps it takes behind the scenes to design, build and ship great apps!
At a time where engineering and design focussed resources and articles are readily available for Mac and iOS development, there’s plenty of room for more open discussion on QA. So if you’ve got any thoughts, comments or questions on anything I’ve mentioned above, I’d love to hear them! I’m @hamishmcneill on Twitter.