Looking Behind the Curtain: QA Testing CORE IMPACT Pro

May 9, 2011

Those who know me are aware that I wasn’t born with perfect dress sense (being colour blind doesn’t help) and as such I have a tendency to rely on sales assistants at clothing shops to help pick out new outfits. Experience has taught me that while the salesperson may insist that an outfit will “suit you sir” or complement me perfectly, I should really get someone I trust to give me a onceover before I go public with my latest rocking outfit.

Even more important is the quality of the products we release here at Core Security.  In fact, it is really important to everyone here and, as you would expect, we have always put a lot of effort and attention into the quality testing of each version of our products. However, as a result of all effectiveness of our testing, most of our customers are blissfully unaware of the hard work our testing teams do to help prevent our customers from experiencing any issues.

I sat down with Laura Balian and asked her to tell me about her experiences working in testing for the last 5 years. Given that testers only have a finite amount of time to test a product, I wanted to understand the thorough process Laura and the ~15 other testers use when assessing new features and product versions. How do they ensure the tests performed both emulate the way our customers will use the product, and how are they most likely to uncover any issues that need to be addressed? Here, in her own words, are what Laura considers to be the important elements of quality testing.

- Alex Horan, CORE IMPACT Product Manager


A Core QA Professional’s Viewpoint, by Laura Balian

Years ago, when I started to work in testing, it was difficult for me to see at a glance how the test cases for new products were devised. I realized from the start that you can quickly understand “why” we were doing testing, but “how” is a level of detail that is trickier to grasp. You can read and learn from almost anywhere and teach yourself or be taught. Ultimately, things work better when you have that light-bulb moment when everything falls into place and you say “I’ve got it!”  No matter how much you have read, studied or practiced, everything makes more sense after that moment.

In my case that moment came over when one workmate told me, “Just try and break it!” Now, I am not telling you testing is a matter of just “breaking” things, but the mindset helps. When we are testing a product we try to make it work as expected by using it in the normal manner, but we also design tests to see how the product reacts when it is not used as we anticipated. At least it helps me to think, if you break it first, then a user is less likely to experience the frustration that comes along with when a product breaks or fails and instead experiences what they paid for it to do. I think that my previous statement explains the “why” aspect of testing, which is to avoid users having broken pieces of your product in their hands – and is commonly translated in the business world as “good quality”. But of course, breaking sounds like much more fun!

So, how can you maximize the good quality of the product you make and reduce the risk of negative results as much as possible? Using a professional experienced testing team is the most direct way. The amount and quality of resources you invest to test has a direct and measureable impact on the final result and will strongly decrease the odds that your software is “buggy.” Those resources are not financial; what I am talking about is time and people – and it’s great to be a member of such a well-staffed and talented QA group here at Core.

Nowadays, the willingness and desire to automate everything is rampant, especially in testing. However, many companies and testers are forgetting about the value of key parts within the testing process; analyzing the new product or feature, thinking and planning how new features will be tested, both individually and in the larger context of the product, and then executing and watching carefully (both for what you expected to see and anything unexpected). Of course I am not denying that automation saves time and helps testers quickly check for basic issues and perform routine activities, but you cannot automate if you don’t first plan and design what to “break” – and on the first execution of any new feature or capability, human eyes cannot be replaced.

Fortunately, our team has both types of testers and each group handles the part they like. I am part of a group of people working as analysts, designers and executors; we work closely with another group of people that automate test cases and constantly develop better automation tools.

We are given documents at the beginning of every release, which we read carefully to start getting ready for the upcoming features. Once we get the finalized requirements, we start working closely in small groups per the particular feature. Testers, developers and automators dynamically form new groups on the fly to assure the quality of the released product.

Developers help us to understand the technical aspects of what they are doing and add information about the test scenarios they consider important. Automators work closely with us to think about the best ways to automate both new test cases and also old ones for regression purposes. I think this is the best way QA work can be done. However, during the years I have been working as a tester, I have heard people say manual testing is something a robot can do and that “executing” is a “monkey” issue. But you can only automate properly only once you have seen the product as a user sees it and run it from start to finish with a clear goal in mind.

Alex asked me what the important qualities of a testing team are. I say for a number of reasons the importance of people’s minds, eyes and hands being directly involved in the tests should never be underestimated. First of all, you need a good team to take software requirements, analyze them carefully and understand them in the context of “Why are we making this feature?” and “How will it help our different types of customers?” Only then can the team plan what they are going to test, how and in what time. Second, we all know software can vary from requirements to the final product and that variation requires the testing team to be flexible enough to both understand and absorb the impact of any sudden or deep changes and adapt their tests on the fly. Third, it is only the tester’s eyes that can see those tricky issues that a user will experience because they use the product – not just running a single script and observing the results.  It is only when you run a test case manually, actually “touching” the product, that you can see the elements of the product working together as well as tricky combinations that you wouldn’t have thought of combining.

Last but not least, every large team must have good communication. A good tester has to be able to communicate properly and negotiate with their different stakeholders. Testers should also be able to interpret what their product manager is asking for on behalf of the customers. They should also communicate properly to the development team on how to reproduce a bug and explain why they think it is not a proper behavior for the software to exhibit. Testers have to be prepared to handle discussions where developers may defend their work and not accept some issues as bugs or disagree on an expected behavior. And most important of all, a good tester should understand the user’s point of view and be capable of seeing and using the software through their eyes.

So to wrap up, people are the most valuable resource you can ever have. I would say a good quality product requires quality testers and a good quality tester needs to be open minded, flexible, willing to be challenged and to learn, analytical, empathic, and a good communicator. I believe that the Impact Pro QA Team has all that and more.

- Laura Balian, QA Tester

  • Penetration testing

Ready for a Demo?

Eliminate identity-related breaches with SecureAuth!