The author will show you how to organise work on product testing. As the QA Manager one can have own personal choices of features and changes but your main job is to get a starting idea of what resources and how much time you will need to get the new release tested according to your company's high quality standards. He presents the Grantt table and all what is needed to make you work easier.

Preparation is the Key: Things to do, and not to do, when you are told that your product is going to be developed across multiple platforms.

You walk into the requirements meeting for the next major release of one of your company's products. Everyone in attendance has their laundry list of features, enhancements, and changes that they feel need to be added to the product for this release. As the QA Manager you have your own personal choices of features and changes but your main job is to get a starting idea of what resources and how much time you will need to get the new release tested according to your company's high quality standards. The Vice President of Development opens the meeting by announcing that, while everyone will get their chance to give their input for new features and fixes, it has already been decided that this release will include native GUI clients on all supported platforms added to the Windows client that already exists. In order to add these platforms the underlying windowing code will be converted from MFC to Qt in order to support the new clients. Also, since many of the company's customers are worldwide, the product will now support the Unicode standard for native language input. Your brain begins to work furiously trying to calculate all of the combinations and permutations that this particular scenario has just produced. Just before your higher brain functions shut down from overheating you hear one last phrase: Our target release date is 9 months from concept design.

Now is the time when you should close your eyes, take a deep breath, and repeat my favorite mantra: Don't panic… don't panic… don't panic… . Actually, there really is no need for panic if you know how to prepare yourself, your team, and your resources for this Herculean test effort.

First prepare yourself. Whether you are a QA Manager, a lead tester on the project, or the developer who is assigned to organize the company's testing you can get through this in an organized manner.

Gather a list of all platforms that are expected to be supported by the new clients. Windows has XP, 2000 Server, 2003 Server, NT, ME, and 98. You may decide not to support some depending on what systems your target customer base is operating on. Apple has OS X over several versions. If your list includes Linux then be prepared to ask some tough questions. There are many, many different distributions each having many different versions. My suggestion is to pick the ones you feel are being used or are most likely to be used by your customers and test against them. One method that we developed for testing a large number of Linux distributions is to pick five distributions (we chose SUSE, Mandrake, RedHat Fedora Core, RedHat Enterprise Linux, and Debian) and perform full release testing on the most recent versions of those platforms. Earlier versions and other distributions will get a limited regression testing every year to insure that we can still support them.

Insure that you create test matrices for all new, changing functionalities in the release. This isn't just a list of things to test but a spreadsheet that takes into account the functionalities, platforms, required database integrations, integrations with other programs, etc. An example test matrix is shown in Figure 1. Have this matrix reviewed but several people from different departments such as QA, development, support, and sales. Yes, I did mention sales. Sometimes a person who is more removed from the effort is the one who will catch exclusions from the list. They are also likely to have "pet functionalities" that their customers have been asking for that may be in the release and they will be happy to make sure that particular item is getting tested.

Figure 1. Sample Test Matrix

Now that you know the scope of your test effort your will need some way to track all of this work as it is being done. In my case I found that one of the things I hated the most became my most useful tool… the Gantt chart. I never fully understood their usefulness until I had to track several parallel efforts going on at once with a limited number of resources to perform within those efforts. If your company runs a Microsoft shop then you would probably use Project as your planning tool of choice. For those of you with a bent toward the open source world might try Planner (formerly MrProject) which is a Gnome desktop application. KDE also has a tool in development named KPlato but as of this writing it hasn't yet released version 1.0. More about putting the Gantt chart to good use a little later.

Planning the Approach: How schedules, lists, and matrices can save your sanity.

For every testing effort there are two major questions: "What are we testing?", and "How are we testing it?". I have described the first question and we have prepared a lot in order to answer the second question. So just how are we going to test this beast? Now it's time to plan your approach.

Take a look at that test matrix. Whether it fits comfortably on a standard sheet of printer paper or takes up a half of a ream of paper covering an entire wall of the development meeting room it all needs to be tested in an organized manner. If it is feasible to do so I would honestly suggest posting your matrix in a conspicuous place. This gives the project exposure to everyone involved, makes a nice backdrop for progress meetings, and it has a limited tendency to discourage feature creep.

Unless you included each platform under test as a separate variable in your test matrix then you must remember that the amount of work represented by the matrix will need to be multiplied by the number of platforms that you will be testing. Depending upon the development schedule the QA team may find themselves testing more than one platform at a time. Concurrent testing in several platforms will take more planning in order to insure that everything gets tested in a timely manner. On the other hand, it can be beneficial to test two different platforms at the same time. If someone finds a defect that is probably not platform-dependent then it can quickly be verified on other platforms.

Do you know what your team looks like? How many people will be testing full time? Part time? Will there be developer testing sessions? What skills does your team currently have? These are the next questions that need answering. Staffing will have the single largest effect on your schedule.

First look at your team's skill set. If you are a Windows shop that is starting to develop for different platforms then you probably do not have someone who is knowledgeable about testing on Linux, Mac, Solaris, or whatever other platform you are developing for. If you don't have anyone then it may be to your benefit to hire a contract person or two to perform your testing on the other platforms. If you do have one or more current testers who have experience on those other platforms then you are ahead of the game.

One piece of advice here: if you do not have people with experience on Linux, Mac, or other platforms then now is NOT the time to start training. Take it from someone who has tried this approach. It will make for a longer and more painful testing process if your testers are trying to learn Linux commands and test releases at the same time. If you have the time before starting the test effort then acquire training for members of the team or have team members cross-train each other if that is an option. Allow your team enough time to learn everything they will need to know about working in the environment before they have to work there under the conditions constrained by the testing schedule. Your schedule will slip horribly if your testers are looking up how to install the software or how to set system parameters in an unfamiliar environment.

Once you have a list of your team members and their relevant skills you can assign tasks appropriately.

Putting Your Plan Into Action: Tracking progress, dealing with problems, and keeping everyone in the loop.

Now that your team is set up and the testing is started the name of the game changes to "Track and Report". For me, personally, this is the hardest part of leading a test effort. Although I consider myself to be organized and conscientious about details, I am also more of a performer and not much of a "watcher". Keeping track of my own part in the test effort was easy. I do that every time I work on a project. Keeping track of everyone else's progress on a project this large takes everyone's commitment and a certain amount of oversight on your part.

If your company does not use some form of defect tracking then now is the perfect time to start and also to see the benefits that can be derived from defect tracking. Since Seapine Software is the producer of automated lifecycle management tools and since we are a firm believer in the old adage of "eat your own dog food" we use our own tool, TestTrack Pro to enter , assign, track, fix, and verify defects. It allows us complete control over our workflow configuration, keeps a running history of each defect as it runs through the workflow, and also keeps a log of all changes to a defect during the lifecycle for standards compliance. TestTrack also integrates with many source control management tools on the market, including Surround SCM, also from Seapine Software. Integration with a SCM tool allows developers to associate code changes with the bugs that they fix.

A defect tracking system is one of your best tools for reporting test status to management. Knowing the amount of defects entered, open, and fixed gives managers a point of reference about the health of the project. Tracked build-over-build a trend can be established for how effective the developers are being in creating good code and how thoroughly the test team is exercising the functionality. You just have to remember that it isn't the amount of defects found that gages the effectiveness of the test team. The quality and thoroughness of the testing is what you are looking for.

Another tracking tool that I like to use is weekly, or even daily, written status reports from each member of the test team. I don't expect long, rambling reports that point out every detail of what they have done that week or that day. I am looking for what they accomplished, what they didn't get to that was assigned to them, what held them up, and any major issues they came across. All of these status reports are then condensed and reported up to the upper management in order to keep them in the loop as to the testing status. By doing this you are also building your justification if you need to plead your case for extending the testing schedule and creating a journal of problems and successes to refer back to for future test efforts.

If you ask the team members to include things that kept them from getting assignments done then you can deal with them in a timely manner. Having the documentation of problems is much better than having an employee just stop by and tell you about them. Chances are, you would have them document the problem anyway after they take the time to come and complain.

Figure 2. Sample Gantt Chart

I am sure that you have forgotten about that Gantt chart by now. I sure would have tried to forget about it. Now is the time to call it back up, dust it off, and start updating it. I recommend a weekly update. Daily updates are too time consuming and if you have multiple shifts working or some sort of flex time implemented then it is often hard to get everyone to get you their status on time. If you update beyond weekly then it is very possible that you won't catch schedule slips or changes in the critical path until they become major problems. If your Gantt chart is broken down into tasks by platform (a very good idea, looking back in retrospect) then you can see how much work needs to be done on each platform and reassign resources accordingly. This is an effective strategy especially if you are ahead on testing one platform and behind on others with the first platform's next build coming ahead of the rest of them. You could possibly put more resources on the almost complete platform to finish it before the rest of the platforms have their new build ready. Reports from the Gantt chart are another way to keep upper management in the loop. This gives them a timeline of testing events and a way to forecast future planning for other efforts. Figure 2 Shows a good example of a Gantt chart that breaks down tasks by platform. Note that this is a simplified example. The actual project that this article is based on lasted many months and the Gantt chart I used had well over 400 rows spanning several more platforms than what is shown. Creating the chart was well worth the effort.

Looking Towards the Future: Building on successes and learning from mistakes.

The crunch time has gone away, the checklists are all filled out, the customers are using the new product, and the testing is over (but it is never really over). You now have time to look back, take a long deep breath, and laugh deep and hard. Now you should take some time to look at what went right and what went wrong. What worked and what needs to be changed for the next test effort.

As I stated in the last section. I would have definitely broken down my Gantt chart by platform so that I could more easily track the testing progress on each individual platform. Using one chart for all of the platforms became too confusing and it was difficult to follow through the clutter of the chart. You will want to make a list of the procedures that you came up with that worked. Discuss the procedures with your team then add them to your existing process documentation.

Also list the things that didn't work and make sure to document those items so that you don't make the same mistakes twice. You will, more than likely, have to go through this whole process again with another release some time in the future. If it isn't you then it will be whoever takes it over when you move on to other things. You only make everyone's life easier by leaving behind a record of the whole wonderful mess.