The initial spec documents that you put together will be a great help at this point. Bringing it all together you should now have:
- Established who will be the main tester/testers on this project;
- Identified other stakeholders who should get copies of the build;
- Shared all your specification documents with your supplier (on Google Docs or similar); and
- Created a shared folder for designs that everyone has access to.
This is a great start but you should now set-up a few more shared documents (again Google Docs is recommended) for the testing process:
- Sign-off Sheet (structure & functionality) – A sign-off sheet for each ‘page’ of the app and each piece of functionality. The easiest way to create this is to simply duplicate your Functionality List spreadsheet but add an extra column on the right, where you can mark off things as complete as they are tested. Using this sheet as your basis not only ensures that every page and every single piece of functionality is tested before launch, but that the app does not stray too far from the original brief.
Extract of an example Sign-off Sheet, the full functionality details (in the far left columns) are not shown here
- Sign-off Sheet (content feeds) – optional – If you are pulling in content as feeds into your app (for example if you are producing an app for a newspaper) then you will likely want to produce a separate sign-off sheet for this as well (you can also use the previously supplied Content / XML Feeds Tracker spreadsheet). This will work in exactly the same way as the previous sign-off sheet but each row on this sheet will represent a feed going into the app, which is then tested and marked when complete
- Snag List – Although you have your Sign-off Sheet, it is usually much easier to record any bugs or missing features and list further details about each one in a single document. This is where the Snag List spreadsheet comes in. To do this you can use or adapt the template at the back of this guide or download the original file from the Members Area (under Resources) of app-outsourcing-guide.com
Set this up as a Google Doc (or on whatever project collaboration system you are using) and then share it with your testing team and your supplier’s project manager/development team.
Extract of an example Snag List
(Make sure also that your testing team and developer all have access to these documents and encourage them to use these as the only place to report bugs and follow-up any issues, as things can quickly get lost and missed on email alone).
Whenever your supplier releases a new build (either by emailing the files, placing them in Dropbox or in a folder in your project collaboration system) they should also email a brief summary (bullet points are fine) to you and your tester explaining what has been added/fixed/changed from the last version. (If you are including other stakeholders in every build you should also either get them copied in these emails or distribute builds to them as you see fit also).
Your tester should then go through every part of the app that has been worked on since the last build and compare it with your Sign-off Sheet, making sure that each area fulfills the specified functionality exactly. They should then also compare it with the signed-off designs in the shared folder to make sure that everything is correct there too. If this is the case then that part of the app should be marked as complete on the last column of the Sign-off Sheet.
If there are some issues however these should be added to the Snag List by the tester, who should complete all necessary fields (including a full but clear description of exactly what the issue is) and marking the item as Open in the ‘Status’ column of the sheet.
You (if you are acting as project manager) and your supplier should then review the Snag List at least daily (it can form the cornerstone of your daily catch-ups) and agree priorities. To do this, change the Severity column to match the importance of the issue as you see it, my suggestions:
- Critical – must be fixed as an urgent priority, cannot launch the app without this
- High – cannot launch the app without this, but less of a priority at the moment
- Medium – want to fix before launch but may be able to wait for an app update if necessary
- Low – fix if we have time but can wait till a future update of the app
How comes everything isn’t critical? Although most bugs will be marked critical or high, there may also be parts of the app that can be delayed if other pressures creep up.
For example, if you have a commitment to a sponsor to launch by a certain date and you are having problems with the Twitter functionality you might decide to leave this part out so you hit the bigger goal of you overall launch date. The Twitter functionality could be re-visited a week later with an update to the app. It’s your project and your call, but adding everything as ‘Critical’ is not going to help your supplier help you. As your supplier corrects issues on the Snag List, they should mark them as ‘Fixed’ to alert your tester to re-test them. Your tester team should then look at each issue again and mark it either as:
- Closed if it now manages the original spec and the problem is resolved (remembering to also update the appropriate line of the Sign-off Sheet if it closes that page/piece of functionality completely); or
- Re-Opened if it requires further work (with an explanation if necessary)
As part of your daily catch-up, you should also review any Sign-off Sheet with your supplier to see if you’re on-track overall and to discuss strategy if you seem to be behind schedule.
If, as a result of your testing process or other feedback you realise you need to change the spec or one of the designs, then make sure you keep your master documents (your Functionality List, sitemap and every design) in your shared folder up to date.
You can create a separate ‘Archive’ folder for the previous documents if you wish. Make sure though that any updated documents make it to the shared folder and that everyone is clear that: firstly, there has been a change; and secondly, that they should now only work from these. With these simple daily checks you can stay up-to-speed with your project and be able to move everything towards completion.
Feedback and criticism
Doing some solid testing against your spec is all very good for keeping your project on track but you need some ‘real world’ feedback too!
A good starting place is making sure you are regularly testing some common user processes on the app. If it’s a game you should have people playing it all the time. What drives them crazy about it? What do they love? If it’s a news app, can people find what they are looking for? Is the way to share articles intuitive or could it do with some work? Are several people saying the same thing?
Depending on your app there are some common steps you will be expecting most users to carry out in it (go back to your original App Product Statement sheet and see what you put against each audience group). Test them all, particularly with people who have not looked at the app a lot (or at all!) as they will be looking at it with a fresh pair of eyes (and will be viewing it without the benefit of knowing your spec or what should happen). If the impression they get is way off what you hope for then you know something is wrong and you might have to revisit that part of the app.
Ask people you respect for their feedback. Don’t ask them ‘do you like this?’. If they know you well or are friends they might just say ‘yes’ to be polite or out of concern for offending you. Instead say ‘what didn’t you like about the app?’, or ‘what was the worst part?’ or ‘how can I make this better?’. These kind of questions are more likely to give you a useful response. Don’t look for a pat on the back, that won’t help you. Listen to how you could improve the app instead.
If you don’t agree with their criticism or don’t have time to work it in – that’s fine but at least you have taken some time to get feedback and are launching the app with some ‘real world’ opinions already in hand. The earlier and the more frequently you can do this in your testing process the better.
If you have the time and budget at this stage of the project to refactor things and improve them then do it. It may also be worthwhile at this stage doing your own check of the app against the original goals you set for it in Part One to make sure that the original concept of the app is being followed.
Shutting it down
If you are starting to get a majority of items on your Functionality List ticked off, you’ll want to start doing some final testing so that you know nothing has been overlooked and that when you submit the app to Apple it will be bug-free. How deeply you go into this is up to you, but I would over-check, rather than under-check, even if it may be repetitive. Better this than a rejected app by Apple (delaying you) or a bad experience for your first users (killing their first impression and any word of mouth marketing you might be hoping for).
A simple way to do this, is to add another column on the far right of your Sign-off Sheet and put today’s date on it as the header. You should now get your tester to go through every part of the app again, checking everything is complete and matches the spec. If anything that was previously marked as complete now has an issue it should be added to the Snag List as an ‘Open’ issue immediately.
You should then repeat this every day that you get a new build. Add another new column (with that day’s date at the top) and let the testers go through it again, repeating the previous process. This should then continue until you have every page and piece of functionality signed off on the app. From this sheet you will also be able to see day-by-day the progress you are making towards final sign-off. (If you get a problem later you can also refer back to the Sign-Off Sheet and see the first date something was signed-off as correct, in an effort to find out when the problem was introduced).
Going back through previously checked stuff is necessary. Not only does it give you piece of mind that everything on the app has been at least double-checked, it also ensures that, during a potentially lengthy project and testing process, nothing that was previously OK has been accidentally ‘broken’ by one of the developers while fixing another issue on the app (an unfortunate but still common occurrence in many software projects). You may also want to do these checks yourself in tandem, as an extra safety net, if you are not the main tester on the project or have not worked with this particular tester before.
When you have every page and piece of functionality 100% signed off you should then do some final final checks. First of all if you haven’t been already been doing so, go through and do exactly the same checks yourself. Then find at least one other person who has never looked at the app to go through it and do the same. There is nothing like a fresh pair of eyes to help you see what everyone else missed. Now is also the time to pass the app to other stakeholders and get their final sign-off and make sure everyone is happy before you think about submitting the app. Before you pass it on to Apple though, if getting you app onto as many devices as possible is something you’re keen to do, then there are a few more things you might like to think about first. All of which are covered in Part Four: The Push.
