Continuing a series of posts on all things ‘QA’ here at Realmac HQ, I’ll be sharing a little about how we like to make the most of our issue tracker, as well as using the tickets logged within it to efficiently create detailed test plans.

Tuning Tickets

If you’re not already using a system to keep track of bugs, I highly recommend looking at a service like Lighthouse, a system in which we also manage projects in - far from using it as just a bug tracker. We sweat the details to ensure we ship great apps - we also place great value on being efficient behind the scenes too. Playing down the value of getting the most out an issue tracker could cost a lot of time that would otherwise be better spent crafting the actual app!

Within an ideal ticket, duplicate bug reports are avoided, concise steps to reproduce a bug are detailed by the first person to experience the issue, and explicit explanation of expected behaviours for new features should always be present. Of course, that is the ‘ideal’. It’s also important to tune and refine the ticketing process, striking the right balance between getting enough pertinent information before taking up too much of the bug raiser’s time.

We’re currently using a fairly common template to log bugs within Lighthouse (based on Apple's own bug reporting best practises) where we request a few specific fields of information that’s used to look at how and why a bug is present.

We could request much more information by default but, as mentioned, it’s vital that logging a bug doesn’t become a chore - keep the process quick! For example, have you ever used an app, particularly during a beta phase, and come across a seemingly obvious bug and thought “the developer must know about this already, I won’t waste the time taken to log it when it’ll probably be a duplicate.”? If the bug wasn’t already noted as being a known issue in the app’s release notes it probably hasn’t been raised yet. It’s natural to zone in on particular areas when working on projects but failing to step back to view the app as a whole poses a risk of missing a bug that, perhaps, would have been seen after stepping back from the project.

If I weren’t in a position where time can be dedicated specifically to testing, I’d want to ensure that any ad-hoc or beta testers are made aware of the fastest path to raise a bug, so that even the small or apparently obvious bugs may still be reported, avoiding leaving it to chance at being picked up. Providing clear steps on how best to raise an issue within release notes will help encourage a higher response rate.

Of course, this process can be made even more straightforward by making use of the extremely useful HockeyApp where app crash reports are automatically collated from an app, whether or not they’re in beta or the App Store. We’ve recently integrated Lighthouse, our bug tracker, into HockeyApp to allow for quickly creating tickets which automatically include all the information gathered and organised from within HockeyApp.

Fast-Track Test Docs

For every ticket raised in Lighthouse, be it for a bug, feature, view or behaviour in an app, I add a test case to a test plan that’s specific to that particular app or project. I find that by using the existing tickets from within Lighthouse, I can fast-track the addition of new test cases to a test plan, to both save time writing them from scratch as well as ensuring as much of that app as possible will be tested.

As mentioned previously, the Lighthouse tickets I use as a starting point for a test plan should already include the required input (steps, files or gestures) to perform an action as well as the expected output i.e. the resulting behaviour from our input. Likewise, with bugs found during development, tickets should also contain steps to reproduce the issue as well as any other anomalies to look out for when testing. This is essentially the core information that makes up each test case.

As new tickets are assigned over to Quality Control I check the ticket and create the new test case within a test plan, had a prior test case for the issue not existed. Our test plans are hosted on a private Wiki here at Realmac HQ (you can find out a little more about our network setup, including the Wiki, in this post by Keith).

Unique ticket IDs that link directly from the test plan back to the Lighthouse ticket are also added to each test case. This makes checking for regressions much more efficient as I can quickly check the build number of an app where a reoccurring issue was fixed previously. Integrating commits from Beanstalk within tickets means, in theory, it’s possible to pinpoint the code responsible for a bug within seconds. This, coupled with the inclusion of ticket IDs in release notes, saves time previously taken up by inadvertently creating duplicate tickets.

I’ve recently been looking at further ways to create effective test documentation. For example, creating visual flow-documents for some of our apps, where every view and the connections between them are laid out. As well as flow-docs, I’ve looked at MindMapping, a great way to check that there’s no areas left unchecked within a test plan. A nice example of an iOS testing MindMap can be found here, created by @noir and @lelchuk.

An Evolving Process

Everything can always be improved in some way, especially in QA. More detailed test cases can be created and added to a test plan, new bugs will always be found. That’s a fact that can take time to get used to. In his recent post on the human impact of bugs, @jury sums this up well:

“Realistically, we are never going to fix all of the bugs in our software. Anyone who tells you they don’t ship until all the bugs are fixed is either lying or naive.”

Not only is the fixing of the little bugs a never-ending process, but so too is the refinement and evolution of all the steps it takes behind the scenes to design, build and ship great apps!

Open Q(&)A

At a time where engineering and design focussed resources and articles are readily available for Mac and iOS development, there’s plenty of room for more open discussion on QA. So if you’ve got any thoughts, comments or questions on anything I’ve mentioned above, I’d love to hear them! I’m @hamishmcneill on Twitter.

As part of a series of posts going behind the scenes here at Realmac HQ, I’m going to be introducing a topic that’s often seen as the black box of software development, QA.

I’ll be featuring more specific aspects of my role in a mini-series of QA related posts while this article will give you a little introduction to my overall role, I hope!

As Quality Assurance manager, I’m responsible for overseeing all things quality related; that means both Quality Assurance and Quality Control. As the line between these two areas can be blurry at the best of times, we often use the term ‘QA’ to loosely encompass both roles. That makes QA a a pretty wide ranging subject that would be challenging to detail what my role covers, if asked to, in a single sentence.

Some of the key areas that I’m involved in day-to-day include writing test and support documentation, issue tracking and, of course, testing all of our apps to ensure they are reliable. One of my overall aims is to ensure the apps we ship are not only bug free and a joy to use, but also that the means by which we build them are the best they can possibly be. I love asking questions as to why things are the way they are. Asking questions is a great way to begin improving something. Why is it that some parts of an app’s development cycle take longer than others? Why is it that one framework or API is particularly error-prone compared to another. If there’s no justifiable reason for something, should it be there in the first place?

Although QA covers the entire development cycle, I personally am not always directly involved at the beginning of an app’s development cycle. For instance, low-level code testing is handled by the project engineers, who independently peer review the code. The analysis of such tests are available to be fully explained and justified to the non-engineering team members. I tend to be more heavily involved once a general feature-set has been settled and when early prototypes become slightly more stable internal releases. At this point I set about forming more detailed test documentation in preparation for more user-based testing.

From the outside, documentation and testing may sound like fairly dry subjects but I like to think a sense of humour plays an important part in good testing. I take great pleasure in throwing the documented processes to the side to deliberately create scenarios that may throw up unexpected behaviours and edge cases in an app. This is an area where human ingenuity can truly trump more rigid automated testing!

When it comes to creating test plans and manifests, I treat each project independently and don’t force them to conform to any processes used in another. However, one useful set of guidelines that I highly suggest you look to base any potential support or issue ticketing process on are Apple’s Bug Reporting Best Practices. The more relevant and detailed information you can provide in a ticket, the more likely an engineer will be able to diagnose and resolve an issue or accurately implement a feature request.

A commonly used word at Realmac HQ is “detail”. We pride ourselves on getting the smallest details right in everything we do from handling errors at code level to designing for retina user interfaces. It’s important to point out that detail doesn’t need to be associated purely with visuals - as QA Manager, I like to think that the details are the individual steps taken to build and ship an app that will delight our users.

Apple released a recruitment video last year that got a lot of coverage online. The video is littered with statements that ring true to anyone who strives to achieve process perfection. The overriding tone of the video was that the focus was not solely on the quality of the end product but really about having the right processes and people in place to enable them to go on and create the best products.

Infinite Potential

The poster above hangs on the wall across from my desk. It reads: “The infinite potential of every little bit”. To cut a potentially long story short, if someone ever was to ask me to encapsulate the mindset and reasoning behind QA at Realmac in a single sentence, I’d point them to this poster.

If there happens to be a topic mentioned here that you’d like expanded upon in my next post, please feel free to comment below or reach me @hamishmcneill on Twitter with any questions.