The weird and wonderful bugs that get thrown up when real users first start using your code never ceases to amaze. There’s always some odd edge case that had been overlooked, despite you think about little else for several weeks. We’ve been through this many times and concluded that beta testing is the solution to our problems.
Here are 7 things you can do to get the most out of our your beta tests:
Ask for a commitment to provide feedback:
Response rates will be higher if you ask your beta testers upfront to commit for providing feedback. This doesn’t have to be formal, it could be just a part of an application form. But having agreed to it, people are more likely to follow through.
Do not release with known bugs:
Most beta testers will only provide feedback once so you don’t want to burn any tester to just hear about known issues.
Allow enough time:
Use the following as a rough guide. For a major development effort, say about a year’s work, you’d want to spare 10-12 weeks for beta testing. Decrease as necessary – so if it took a month to develop, then, around a week will suffice.
Be feature complete:
Only beta test when your feature complete. Adding in things as you go sets you back to the start. Otherwise, it just means the new code and its impact on existing functionality isn’t as well tested as the rest. Something you’ll regret later.
Make it easy to get in touch:
You want to make it as easy as possible for your beta testers to provide feedback. Give them direct emails and offer to jump on a Hangout/Skype if they’d prefer.
Follow up but don’t annoy:
While your product might be front and center for you, it’s not going to be that way for your beta testers. You’ll want to remind them along the way. However, don’t overdo it, they’re helping you out so you don’t want to annoy them with too many emails.
Don’t forget to provide feedback:
Make sure to send them updates during and after the tests about how you are putting their feedback to use. People like to know that their time wasn’t wasted. And don’t be tight with the swag – a free t-shirt can do wonders!
You don’t need to argue over code style and formatting issues. There are plenty of tools which can consistently highlight those matters. Ensuring that the code is correct, understandable and maintainable is what’s important. Sure, style and formatting form part of that but you should let the tool be the one to point out those things.
Everyone should code review
Some people are better at it than others. The more experienced may well spot more bugs, and that’s important. But what’s more crucial is maintaining a positive attitude to code review in general and that means avoiding any ‘Us vs. Them’ attitude or making code review burdensome for someone.
Review all code
No code is too short or too simple. If you review everything, then, nothing gets missed. What’s more, that makes it a part of the process, a habit and not an afterthought.
Adopt a positive attitude
This is just as important for reviewers as well as submitters. Code reviews are not the time to get all alpha and exert your coding prowess. Nor do you need to get defensive. Go into it with a positive attitude of constructive criticism and you can build trust around the process.
Code review often and for short sessions
The effectiveness of your reviews decreases after about an hour. So putting off reviews and doing them in one almighty session doesn’t help anybody. Set aside time throughout the day including breaks not to disrupt your own flow and help form a habit. Your colleagues will thank you for it. Waiting can be frustrating and they can resolve issues quicker whilst the code is still fresh in their heads.
It’s OK to say “It’s all good”
Don’t get picky, you don’t have to find an issue in every review.
Use a checklist
Code review checklists ensure consistency – they make sure everyone is covering what’s important and avoid common mistakes.
Keep the code short
Beyond 200 lines, the effectiveness of a review drops significantly. By the time you’re at more than 400, they become almost pointless.
Link to any related tickets or the spec. There are code review tools that can help with that. Provide short but useful commit messages and plenty of comments throughout your code. It’ll help the reviewer and you’ll get fewer issues coming back.
Whether you’re working on an existing or new application, you’ll often find yourself playing catch up when it comes to tests. Soon deploying code changes feels like poking at some ugly, sleeping code monster — you aren’t sure what’s going to happen, but you know it won’t be good.
Here are the 4 things you should do first to tame the beast and improve test coverage:
1. Add the Right Tests
Start by adding tests in the areas where it is easiest. It’s important to consider the order in which you do this to make sure you get the most out of your scarce resources. You want to start adding tests in the following order:
i. Create tests as you fix bugs
Add tests to prove that your specific fix is working and keep them running to show that this does not break again. This prevents from being somewhat targeted — you are creating tests in your weakest areas first. The weaker it is (i.e. more bugs) the faster you will build up tests.
ii. Create tests with all new features
All new features will need to include tests created to prove that the feature works as expected. If you’re covering the new aspects of your application, then at least things aren’t getting worse.
iii. Create tests as you add to old features
When updating old features, add tests as you go to show that the older functionality is not breaking in unexpected ways.
iv. Create tests in identified risk areas
Talk to the developers and testers on your team and ask them to point out any weak spots or issues they have experienced. Also, you talk to your support team — they are an excellent resource with a direct line to the customer. They’ll know the features of your product that frequently causes issues.
2. Turn on Code Coverage
Code coverage is a tool included in most continuous integration systems (or one that can be added with a plugin). This tool will instrument and monitor your code as you run the tests to determine how much of your code used by the tests. For this to be useful, follow these steps:
Start running code coverage against all your code
Get a baseline
Find out what the tool can see, where you are currently at etc.
Determine areas that you want to exclude.
There are likely areas of your code that you don’t want to cover — third-party tools, ancient code untouched for years etc.
Determine coverage goals
Sit down with your team and discuss what your current coverage is and what your ideal can realistically be (usually 90% or above).
Work-out steps to improve your coverage
You aren’t going to fix this problem overnight. Put in place some specific tasks which are going to help you achieve your goals over time.
Determine your pass/fail criteria
Is staying the same OK, or should it always go up? Do you define any drop as a fail?
Run Code Coverage constantly
Use automation to run your coverage test and use the criteria you determined as a team to report a pass/fail and do this constantly. It is a lot easier to add tests when the code is still front and center in your mind than later on.
3. Run your Tests on a Scheduled Basis
You should run your tests regularly, on several schedules:
Run them on every check-in
Use CI tools like Jenkins to run (at least) your unit tests on every check-in. Run them simultaneously if they are taking too long to run at this frequency.
Run them on every build (package)
Depending on how your systems work, your CI infrastructure can help you with this. This could be on every check-in if you are on Continuous Deployment, or every day/week/month that you use. This should be a clean install on a test environment and a full run of all your automated tests.
Run them on every deploy
You should run all your automated tests against your environments immediately after a deploy.
Run them every X days/hours/minutes
Run your automation suite against your constant environments as often as you can (Production, Staging etc). This should be at least a “once a day task” and take place during ‘off-peak’ times when it does not interrupt others too much. You can increase the frequency further if your tests are short, just be mindful not to overload the system.
4. Provide a Button to Run the Tests
Again, use a tool like Jenkins to turn test runs into a self-service operation. A developer should not be delayed by asking a QA person to run a test for them. Get a system in place where your tests will run and just give them a button to press. Remove as many barriers for everyone to run the tests as possible.
If you follow these steps, over time, you’ll see that you are able to turn an unwieldy application into something more manageable. First, by adding tests to the key areas, then making things as easier as possible, you can build confidence around your code changes and deploys.