How To Handle Requirements That QA Cannot Test

[bkuhn]

We were recently conducting a training session and had a great question from a participant - and I wanted to get people's input here. We were discussing testability of requirements and the question was this: "how do you handle documenting requirements that you know QA doesn't have the means to test?"

The question is not whether to write testable and verifiable requirements - we should. But if we know that a certain requirement would require an automated test tool or a production-like set of test data that QA doesn't have, for example - what should our approach be in documenting the requirement? Write the requirement in such a way that QA can test it? Or write the requirement without considering QA capabilities?

So my take is to write the requirement without considering QA capabilities, but then call out somewhere in the SRS known risks about those requirements (we should be documenting requirement risks anyway, so this shouldn't be new). Here are my reasons:
- The requirements team doesn't own testing, QA does - so it should be QA's responsibility to respond to, and develop a plan for, verifying the requirements.
- Just because QA doesn't have the required capabilities, it doesn't change the underlying business reason for the requirement. Of course, if the requirement doesn't map back to a business objective, then we have another problem.
- It may be that those particular requirements must absolutely be tested and that the project will fund whatever tools/resources/etc... QA needs to test them. We can't make the assumption that the status quo will always be.

B

[MAlexander]

Having worked in both camps (QA and requirements), I would wholeheartedly agree with B's take and rationale. The bottom line is that the requirement is the requirement, whether QA has the (current) ability to test it or not. We should do what we can to help QA understand the requirement so they can do their work, but we also need to rely upon them to be the QA experts and come up with novel approaches to the task.

[MTalbot]

The bottom line is that the requirement is the requirement, whether QA has the (current) ability to test it or not.

Are requirements completely independent of IT's ability to implement them?

We set scope based on IT's ability to develop. Why would we not also set scope based on IT's ability to test?

[rcauvin]

There are three important distinctions to keep in mind:

1. Testability in principle versus testability in practice.
2. Direct versus indirect testing.
3. Requirements versus acceptance criteria.

I address your question and these distinctions here and here.

[MAlexander]

Are requirements completely independent of IT's ability to implement them?

We set scope based on IT's ability to develop. Why would we not also set scope based on IT's ability to test?

Good points. I've been on some projects where the scope of the system to be built and the scope of the requirements weren't completely overlapping, though. We had business requirements (typically with low priority) which simply couldn't be met by IT, but they were documented anyway so that they could be addressed if/when the technology caught up with the stated need.

So in projects where the scope of the requirements is set by what IT can develop, I would agree that a similar boundary could be useful for what IT can test. I'd just personally prefer to stay away from both types!

[gescober]

Are requirements completely independent of IT's ability to implement them?

We set scope based on IT's ability to develop. Why would we not also set scope based on IT's ability to test?

How much do you limit yourself up front in terms of IT's ability to develop/test when you're gathering requirements? Granted, most projects start with a vision that generally defines scope, but I generally don't limit things within that scope. Once we've got the requirements, then we'll negotiate a solution with IT.

And, really, what's the limitation on the ability to develop/test? Is it technology? Or, is it money/time/resources? If it's technology, then you're stuck. If it's time/money/resources, then the business will decide scope based on cost/benefit.

Going a little off topic - how do people generally set the scope for their projects?

• Do you get an executive telling you, "You have 9 months and $2 million dollars; build all of the requirements you're going to gather." Then, 6 months later, start the 'rapid de-scoping' phase of your project.
• Or, do you get - "Here are the requirements, let's figure out how long and how much it'll cost, and then budget for it."
• Or, do you go Agile, and say, "We don't know how much it will cost or how long it will take, but we'll keep developing iteratively until we're done; which will probably be cheaper and better than the other way of doing things."

[JGrapes]

I think you do write the requirements. Your issue will be whether or not QA will sign-off. QA will probably only agree to sign off with risks well-understood and documented.

Also, even before requesting QA sign-off, you want the business to understand that there may be risks/issues as you go forward.