RE'08 Follow-up: Teaching About Unknown Software Requirements

At the REET08 workshop, Ray Barnes presented a paper “Teaching the Unknown and the Unknowable in Requirements Engineering Education” written by himself, Donald Gause, and Eileen Way.

In their paper, they talk about the need to accept that the "unknown" and "unknowable" exist in requirements.

Confusing? That is - "You don't know what you don't know."

And while it is challenging for a requirements engineer to deal with, it is still a very necessary skill. Their paper is specifically focused on teaching students about the uncertainty and how to work with it. They have the students do “real world” problems and give them specific techniques to deal with the uncertainty. The students seemed to experience the expected frustrations of not having all the answers in these problems.

I thought it was interesting that when they taught a specific technique to the students (through lecture), the students still didn’t apply it, even amidst their frustrations. The paper gives an example of teaching students about Toulmin’s model of logical argumentation multiple times. Finally the instructor helped 1 student use it, and later that student offered to present it to the class. It wasn’t until this last step - the student explaining the concept and value - that the rest of the students really got it.

That’s probably pretty typical of learning in general, both in that they have to experience to understand it and peers can be very influential in teaching. In fact, one of our favorite training games is called Each Teach from Thiagi, where we split the class up and lecture them on different topics, then ask them to teach the topics to one another. We have had outstanding success in students’ perceived understanding and actual understanding of the topics.

I found it to be an interesting talk and had some questions that came out of it:

1. How does unknowable relate to the untestable in requirements? There was an INCOSE paper that talked about the idea some requirements cannot be written in a testable format, so you can do risk analysis on these to mitigate the issue. Similarly, you might identify the areas that seem most unknown in your requirements and focus more time on those specifically. Or weight your risks and priorities accordingly.


2. How do you document the unknown or unknowable? If you have an area that you know is still fully undiscovered or in some way vague, it should still be tracked. Perhaps in an RM tool, or even in an issue tracking list as things to be further discovered. And certainly a risk list, as per point 1 above. Ideally you would document as much as you do know and flag it in some way for both follow up and just as a warning.


3. Do other industries have this issue with the unknown? Surely it’s not just systems and software, so what do other industries do to deal with the unknown? Probably a lot around risk analysis, but this is just a guess.


4. In teaching something like this topic of uncertainty – both in awareness and the skill to navigate it - how much of it is something that just has to be experienced in order to learn it? There isn’t always a black and white answer to how to approach the unknown. My thought is it’s probably 95% experience, with 5% teaching them helpful tools to try out.


5. When doing the real world exercises, can you write scripts for the instructors to use in role playing on how to be vague? I haven’t really tried this formally, but it sounds fun. And hard! In fact, related to the unknown problem, how can you predict what the students will ask in order to have pre-scripted responses?


6. I think one of the most important things in an experiential exercise is to debrief afterwards. I’m not sure if they did this here, but it wouldn’t surprise me. You put students in what is probably a frustrating exercise, and afterwards you need to talk about the experience, let them explain what was hard, and have them explain what they learned.

RE'08 Follow-up:Questions About Reinforcement Pattern in Teaching Software Requirements

At REET08, Sascha Konrad presented “The Reinforcement Pedagogical Pattern for Industrial Training” by Brian Berenbach and Sascha Konrad from Siemens Corporate Research. They outline a pedagogical pattern called the “Reinforcement Pattern” in which you teach the students a concept (including any definitions), give them a couple of examples of the concept in context, do one or more exercises together, and then relate a set of concepts together using a team exercise.

Their paper talks about how this works well in industry requirements education.

The questions that this paper made me think about:

1. Can you use this pattern when you are not always with the students in person? This is something we are just beginning to develop more thoroughly, but I hope the answer is “yes”. I know of someone at DePaul University who does a distance learning course where many of her students are remote. She does her best to make the examples and exercises work for remote students. They seem happy with it as well. However, there probably is some impact to not all being physically in the same room. It’s harder to share the students’ practice work and teach from it if not everyone can see it. The team discussions students have will not be as easy if they cannot all just gather around a table for 3 minutes in person.

2. Is it more important that the instructor have experience at instruction or at requirements engineering? This is something we have long debated. Currently we put an emphasis on instructors having requirements engineering experience, and we teach them to be good instructors (or select the requirements engineers that are naturally good at it). However, a good instructor should be able to learn enough about the subject matter to facilitate a good learning experience even if they haven’t done it. My concern is they don’t have the stories from practice to relate to the students. But perhaps those can be learned.

3. Is customizing the exercises to be specific to the context of the students important? For example, if you are teaching students from a mobile phone group, should your examples relate to mobile phones specifically? We have courses that are taught in a public forum, so we cannot be specific to all students in your examples and exercises. Therefore it seems to work to use generic examples and exercises, but be ready to address how they can translate the practice in the course back to their job.

Live from RE’08: All Done, But There’s Nothing Wrong with a Little Fun!

The conference is now over, and it’s been a great one! Now we can all look forward to RE’09 in Atlanta. But before we leave the live posts behind, I want to share a fun story about some of our favorite requirementers.

Wednesday night they had a conference reception at the Museu Nacional d’Art de Catalunya (that’s National Art Museum for those that can’t read the language!). It was fun, we got to see a part of the museum even. Well after the party, we decided to go for a drink and of course invited others to go along with us. So off we headed with some of our favorite requirements experts, to find a drink in the streets of Barcelona. Along the way we managed to find some more of our favorite requirements experts wandering the streets. We found a nice little bar where we could sit outside and order cerveza and vino.

We also heard some more statements to add to the stupid thoughts list.

• "When I got here, I only knew how to say 3 in Spanish - which meant I had to be really committed to whatever I was asking for." Really?
• While looking at the murals in the Romanesque Gallery, which were from various small churches, one person says “ So, then you think they moved and re-built these here?” … um, as opposed to building the museum around them?
• And what happens when you ask a very literal person (we are at an RE conference after all!) a specific question. I was surprised to hear this person only brought one pair of shoes. So I asked “You wore your shoes on the plane then?” thinking maybe he packed a dress pair and wore another pair. His response was perfect “Well usually yes, I worry about foot fungi. I mean if it’s a long flight, I might consider taking them off, but it depends on the feeling I have at the time.” To his credit he answered the very question I asked!
• In the context of guessing the length of the tunnel "How far are we now?"..."We're at one.."Where did we start?"....."Uh, zero - where else would we have started? Negative?"

And with that, thanks for the good memories RE'08 friends! We learned a lot, some about requirements, a bit about ourselves, and a lot about our colleagues.

Live from RE’08: The Towel Effect

The closing talk today was by Jean-Pascal van Ypersele from the Universite Catholique de Louvain-la-Neuve and Vice-chair of the IPCC working group 1. Say that 10 times out loud fast! Anyway, his talk was “Climage change: Challenges and Opportunities for Software Requirements Engineering”. In general, he discussed the background science behind the warming trends they have measured, and I by no means am going to try to recreate that here!

But I had a couple of interesting take-away thoughts. He discussed what we call the “towel effect”. I’m sure everyone has seen the signs in their hotel rooms suggesting to save resources by reusing your hotel towels. He suggested the results of a study on the effectiveness of such programs would be interesting. Partly because he thinks the presentation of the program is usually quite awful, most people probably do not participate, and even if they did, what kind of impact does it actually have? Dan Berry made a comment that he very much tried to do this in every hotel he stayed in, and yet not once did the cleaning staff save his old towel – they always gave him a new one anyway. Anyway, the speaker’s point was this is a marketing program by the hotel to look better, but proposes that they may not be terribly effective programs for actually helping the environment.

And the other more relevant take-away discussion is – what can we in RE do? Can we help come up with good solutions that solve the towel effect problem? But there really is an interesting large scale system problem here – there are many different stakeholders with variant priorities – how will we ever align them to do the right thing for the environment (whatever that “right” thing is!). You have companies who want to make money and reduce costs. You have individuals who want to recycle everything. You have governments who need to look at their global position on this, find money to fund projects, and of course they have to be elected. The point is there are a lot of different interests, so it’ll be interesting to see if we can come up with solutions that are accepted by all users.

For more related thoughts, see the green post: part 1 and part 2.

Live from RE'08: Creativity Workshops

Neil Maiden presented a paper Use and Influence of Creative Ideas and Requirements for a Work-Integrated Learning System by Sara Jones, Perry Lunch, himself from the University of Manchester and Stefanie Lindstaedt from the Know-Center.

The basic premise behind their paper is that there is an issue with elicitation, in that it assumes that the users know and can articulate what they want. They’ve integrated creativity workshop techniques to address this.

In short summary, they ran a two-day creativity workshop with 16 stakeholders, 1 facilitator, and 2 scribes. Through the workshop they do divergence and convergence of ideas, repeated several times over. I believe the idea is that people come into the workshop with some narrow ideas, so you broaden those ideas. Then in time you converge them back down to something more specific and useful. Some of the techniques they used include:

• Round robin brainstorming, using creativity triggers
• Constraint removal – a systematic technique to remove constraints, even if it is not feasible to remove them for real, but pretending they aren’t there allows for more creativity
• Solution presentation – use technology ideas they already do have to inspire new ideas

They did quite a bit of data analysis and at least can show some tentative amount of “creative ideas” came out of these workshops. Either way, I do think this is important. It reminds me of the work by Innovation Games: Creating Breakthrough Products Through Collaborative Play by Luke Hohmann, which we’ve had some success with.

Live from RE'08: Recreating Industry in an Academic Course

I wanted to spend a few minutes writing more about the first paper from this morning, again that was Requirements Engineering Education in the 21st Century, an Experiential Learning Approach, presented by Gil Regev. In his talk, he described the design of their Requirements Engineering masters course. This class effectively simulates an end-to-end practical project for the students. In the last 2 years, they increased the time spent in this project on the business, RE, and specification topics from 7 to 11 weeks of the course (out of 14 total weeks).

He goes on to explain that industry is full of “wicked problems”, where the problems are difficult to define for all stakeholders, there are no clear stopping rules, they have strong moral/political/professional dimensions which cannot be easily formalized, and there is often no objective measure of success. Their course design is a bit unconventional – which I love – the intent is to reflect industry realities by maximizing confusion, not giving the rules of the game out, and stating goals that are not the actual learning objectives. Also, they do not grade the experience in order to create an active fearless learning.

They give the students vague a problem statement around management believing sales are suffering because of customer support problems. The instructors play the part of subject matter experts, going as far as to dress in costume to clearly separate their role as non-professor. They point out this probably does requires business savvy instructors.

In the end, the students say they would have done better if they knew the rules, if they had theory about RE before they did the exercise, and the course is chaotic. If only! Though at least one student shared a retrospective in which she recognized after the course while working at a company, that this course was in fact simulating the real world.

This class is going to be very hard to replicate in industry (as a training), since it takes 14 weeks and 6 hours a week. And even for the instructors, it is very time consuming to teach it – partly because they play the role of subject matter experts, and partly because they need to do some amount of redesign every year on the scenario so that students don’t hear ahead about how to do the exercise – as it would take away from the learning.

All that said, I like it. I like that they are trying to recreate an industry problem and teach it in academia. And sure, they can’t simulate it exactly, but this is one of the better attempts I’ve seen at it.

Live from RE'08: Learning Research

This morning, Mike and I attended the Learning Research track where we heard 3 research papers presented on the topic of learning.

The first paper was Requirements Engineering Education in the 21st Century, an Experiential Learning Approach by Gil Regev, Don Gause, and Alain Wegmann. I’ll talk more about this one in my next post.

We also saw Gameplay to Introduce and Reinforce Requirements Engineering Practices presented by Renel Smith and Olly Gotel from Pace University. This is based on Renel’s research which lead him to develop a game called RE-O-Poly, to be used to train students on some basic requirements knowledge. In case it’s not obvious from the name, it is based on the game Monopoly. I've been watching Renel's progress on this over the last year and it's nice to see the game growing. I'm certainly a big fan of using games to train, so I think there is some interesting work here.

Finally, Thomas Alspaugh presented Marginal Notes on Amethodical Requirements Engineering: What Experts Learned from Experience by himself, Susan Sim, and Ban Al-Ani. This talk was based on interviews and survey they did in 2006 of various RE’s – what their most important skills were, experience levels, etc. The 5 themes that came out of their feedback were: spanning bridges, communication, selective process, doing less, and business value.

Live from RE’08: Learning Days for Us

Yesterday was the Requirements Engineering Education and Training (REET08) workshop and today was the beginning of the main conference. So when I say they were “learning days”, I mean more than 2 days of us learning - I also mean that most of the talks we heard were about learning, a topic near and dear to my heart.

Mike summarized the workshop well, however he neglected to mention the talk he gave during the workshop. He presented our paper titled “Effective Design and Use of Training Games”. This talk was about specific games we designed for our Req101 training course. Given the nature of the workshop, he was able to actually play one of the games with the participants, and they absolutely loved it. In fact, some of the participants are looking to use the game ideas in their own training programs. One of the neat things about this year’s workshop is we spent end of the day sharing classroom activities with one another for reuse. I had co-chaired this workshop with Jane Cleland-Huang from Depaul University in Chicago and Didar Zowghi from University of Technology, Sydney Australia - so it was really exciting to see it finally come together and have so much participation (25 people!).

Today’s first session was “Training & Lessons Learned”, with the first paper presented by Brian Berenbach on a survey he did of a requirements engineering training program he did at Siemens. He was able to survey the students quite awhile after they took the course, which in itself is a neat thing – to find out what the students found most useful to them from the training course. This would be great data in order to influence the design of your courses in the future. The last talk of the session was by Sascha Konrad, also of Siemens and was about the challenges and solutions of doing requirements engineering on large-scale systems. The challenges they face at Siemens are certainly common to many large organizations – for example, there are many requirements to be managed, their customers’ expectations can be hard to manage, and the teams are globally distributed. And the talk in the middle was actually my talk, based on a paper I wrote with Mike, titled “Games-Based Requirements Engineering Training: An Initial Experience Report”, in which we describe the learning theory behind using games in training and talk about specific games and experiences from our visualization course. The talk seemed to go over well, and I got some interesting questions.

• If you have large teams in your activities, how do you deal with the problem of some people checking email or wandering out of the room and letting the rest of the team do the work? The short answer is that you talk to them individually if it happens. But at the core of it, we are using games in our training in order to avoid this exact problem – we want the students to be so engaged that they want to be there.

• Have you used the games internationally, how well does that work? We have not done much internationally. One of our courses uses Jeopardy, and we did teach it internationally. They understood it but were not as excited by that particular game. However, over the next few months we’ll be internationalizing our courses, so we will have more to say shortly.

• Have you thought to use different types of members of the audience to play different roles in games (i.e. RE vs developer)? We have not. We do have different types of audience members, but typically they are all there so they can learn to speak the same language about requirements, not to use the content differently per se. That said, there is a possibility to do this, but I would have to give it more thought. And we do one exercise in our Req101 class that does actually put the students in the role of a developer.

• What is Jeopardy? (this was a follow-on to our discussion about using games internationally from someone non-American) We laughed at his question, as it made exactly the point about international audiences not understanding USA game references. Most of our games are not actually based on US games, though some are and will need altered. But to answer this, I did explain Jeopardy.

The final session of the day was the panel on “Transforming the RE Classroom Experience”, with Jane Cleland-Huang, Don Gause, Olly Gotel, Zhi Jin, Pete Sawyer and Didar Zowghi. They each presented a snip-it of information about their university courses and then opened it for discussion to the group. My favorite question was (and this is biased since I asked it), how did Jane change her classroom activities for her distance learning (DL) students to be able to do them. She records the courses and they watch it online the day after the in-course stuff. And it turns out she does let the DL students watch the activity performed by the local class, so that they can learn from the local experiences. She then asks them to do the activity on their own. Her feedback has been only positive from the DL students in how this works. There was also a good discussion around whether universities should teach current industry practices or new ideas from research – the consensus is that it’s mostly 80% current practices with a little bit of new to send out into industry.

All in all it was a fun-filled day!

Live from RE'08: Requirements Engineers are smarter than Rocket Scientists

Did you know that you're smarter than a rocket scientist? Seriously! In a presentation at the Requirements Engineering Education and Training (REET08) workshop today, one of the presenters talked about his experiences with teaching Requirements Engineering (RE) techniques to European space agency employees, most of whom had PhD's in math and/or physics -- actual rocket scientists! As part of the presentation, he said that those scientists said that they weren't able to do RE, and that it should be left to people who had those skills. So kudos, RE community -- you're smarter than rocket scientists!

In other (and only slightly less exciting) news from today's workshop, we heard a lot of great ideas from academics and industry professionals who teach the next wave of Requirements Engineers. Tony Gorschek presented his current conundrum in teaching RE, namely that he has only 5 weeks of instruction time to deliver the only RE course in his school's program. Which topics are most important? Do you teach students to write clear, concise, testable requirements, or do you teach them to be great product managers who understand business realities and can plan product releases accordingly? Yes, we did suggest that he teach both, but time constraints won't allow it. And appropriately enough, time constraints in the workshop kept us from discovering the answer!

Several presenters explained that they teach their programs' RE courses at or near the end of students' educational programs. The rationale for the placement of the classes is that students must understand how to be good software engineers before they can understand, apply and appreciate RE skills and practices. At first I thought this was the right order, but as the day went on and we heard about more and more challenges of this approach, I began to wonder if placing RE education at the end of software engineering education sends the right message. If something is as fundamental as we believe RE to be, then shouldn't we make it a fundamental (rather than a capstone) element of RE education? Would the answer be different if the RE courses were part of a different degree program, like Industrial Engineering or Information Systems?

Finally, we talked quite a bit about the disconnect between industry's needs and education's focus, including training both in academia and in industry. Education in RE seems to provide controlled, well-formed, somewhat artificial problems for students to solve. Unfortunately, most of the projects we deal with in industry are subject to political forces, impacted by fickle market conditions, and just plain messy. How can academia best provide Requirements Engineers with both the problem-solving skills and the ability to deal with messes needed to do their jobs well.

Clearly, we identified more issues than definitive answers, but also we came up with a number of great ideas and new tools to help teach Requirements Engineers (and improve our own work on projects). All in all, it was a very educational day.

It's been a great first two days at RE08 -- stay tuned for more exciting adventures of the Seilevel team!

Live from RE'08: Managing Requirements Knowledge, or RE as KM

OK, yes, it's a confusing title. But stick with me, and I promise it'll make more sense.

Welcome to Barcelona! Day one included the Managing Requirements Knowledge (MaRK) workshop. The presentations covered a WIDE variety of topics in Requirements Engineering (RE), some of which seemed directly related to Knowledge Management (KM). Many of the presentations initially appeared unrelated to what I think of as KM, but part of my interest in the workshop was to get a sense of what our community considers KM, so I stuck it out ... and I'm very happy that I did.

My typical understanding of KM is influenced by Information Science and corporate KM endeavors, in which the whole of a company's knowledge about specific topics is cataloged and managed in a central repository. So, I was expecting KM in RE to be similar. What I found, though, was that KM in RE is in many ways more fundamental to our work than KM is in my expected, corporate setting. Sebastian Meyer talked about automatically generating glossary terms from existing documentation. Joao Araujo presented research on automatically generating use case names, again from existing documentation. Jane Cleland-Huang and Oezguer Uenalan each discussed some initial work on using wikis for requirements elicitation. To me, these topics all seemed to be "everyday" RE, not specifically KM-focused.

What I was looking for was something new or different about RE practice, given the focus on KM. What I found instead was a new way of looking at RE as KM. These unique approaches may be focused on eliciting or generating requirements, but those activities are the first step in managing knowledge about requirements. By slightly changing my perspective on the topic and the work we do, I was able to see the entire RE process in a new light. Expect to hear more about RE as KM from these and other researchers.

One of my favorite parts of the day was the small group discussion sessions we had at the close of the session. In my group, we delved deeper into the use of wikis for elicitation, documentation and management of requirements. We got into some interesting topics of anonymity, power relations, and the role of the RE in the overall development process. Again, keep an eye out for Jane and/or Oetzger to give us more to talk about in the near future!

It's been a great start to the week, and I expect the remainder to be as insightful -- stay tuned!