What do testers test?

[BizAnalyst]

One of the audiences for testers is requirements. But... do testers test requirements or do testers test design?

Or both?

Assumptions: I have a set of requirements from which a design spec was built. The design spec specifies everything the system is to do from a user's point of view. Testers have a limited amount of time to test, and need to get the biggest bang for their testing buck.

Do testers test that the design was followed, or that the requirements were implemented?

[gescober]

In my organization, we do both. The test group is called Validation and Verification. Validation refers to ensuring that requirements were met - that the software designed and built meets the intended use of the system. Verification tests that what was built works as designed or specified.

As an Analyst, I work on both levels. On one level, I help facilitate the writing of the business requirements that are subject to Validation. This is where the Product Managers capture what product requirements.

At another level, I write "detailed requirements" or "product specifications" that are the subject of Verification. I wouldn't call this quite "design" level, but it's not purely implementation neutral either.

However, it should be noted that I work in a regulated environment, so we need to do both, and the size of our V&V organization reflects that.

[rcauvin]

Originally Posted by BizAnalyst

Assumptions: I have a set of requirements from which a design spec was built. The design spec specifies everything the system is to do from a user's point of view. Testers have a limited amount of time to test, and need to get the biggest bang for their testing buck.

Do testers test that the design was followed, or that the requirements were implemented?

You pose a very important question. Most organizations that are broken test only against design specifications. (In fact, most organizations never bother to document real requirements.) Often, the result is that the system meets specifications but fails to solve problems for customers.

In general, a healthy organization will test against both requirements and design specifications. If there is too little time to do so, the organization will scope down the requirements to free up more time for testing.

Given the somewhat false choice in your hypothetical, though, I would answer, "It depends." Test against requirements if it's feasible to get sufficient coverage. However, sometimes a large number of requirements are testable in principle but not directly testable in practice. In those cases, you may be better off testing against design specifications.

[bkuhn]

Both should be tested. Agree with Roger - some requirements aren't written in a testable state (though they should be). Similarly, it may prove impractical or too costly to test all design aspects as well.

[BizAnalyst]

Interesting. I guess another way of asking this is... is it the testers job to make sure the system meets the design or the requirements? I just realized I agree with the answers I've received, but would add more info:

In the design phase, the job is to make sure the design meets the requirements.

In a final test phase, the job is to make sure the system meets the design.

I sidetracked to this topic because loading documents with "the system shall.." statements is often justified as needed by the testers. I'd rathaer see less time going into "the system shall..." statatements and more time into design. i.e., really nail the business requirements, but then start moving to design, rather than an SRS loaded with "the system shall".

[rcauvin]

Originally Posted by BizAnalyst

In the design phase, the job is to make sure the design meets the requirements.

In a final test phase, the job is to make sure the system meets the design.

In theory, this sequence works well. However, most organizations run into big problems when they test only against design.

This conversation relates to another one we're having on traceability. My point in that thread is that testing against the requirements ties everything together and makes for some implicit traceability. No matter how many formal traceability documents you compose, you are likely to have some gaps between your requirements, design, and implementation. If you don't test against requirements, you may never expose these gaps.

[bkuhn]

Agree with Roger again on this point - testing only against the Design document presumes that the Design document hit the requirements 100%. While that is certainly the goal, that's a really big assumption to be risking the project on.

[ddelancey]

A good tester does both. And then some. The best testers I work with are those who say, "I'm going to do everything in my power to break it."

[MGoyal]

How about this?
QA does system/design testing (unit tests, regression test, etc.)
UAT tests against the requirements (using UAT Test Plans that can be traced back to the requirements.)

Originally Posted by BizAnalyst

One of the audiences for testers is requirements. But... do testers test requirements or do testers test design?

Or both?

Assumptions: I have a set of requirements from which a design spec was built. The design spec specifies everything the system is to do from a user's point of view. Testers have a limited amount of time to test, and need to get the biggest bang for their testing buck.

Do testers test that the design was followed, or that the requirements were implemented?

[bkuhn]

Originally Posted by MGoyal

How about this?
QA does system/design testing (unit tests, regression test, etc.)
UAT tests against the requirements (using UAT Test Plans that can be traced back to the requirements.)

Certainly agree with the second point, but system testers/QA should also test against requirements (in addition to design). I've run too many UATs where the software broke on the first test due to developers/system testers not running the UAT scripts themselves first.