3 Basic Tips to Data Migration Requirements for your Software Project

It's late stage of a project, so it's time to start worrying about your data migration requirements if you haven't already! If you have done it, good for you! You can just stop reading.


But so many projects push this to the end and then panic when it doesn't go well. I have seen project after project have late deployments because the data was not properly migrated and tested. Here are a few tips to consider:

1. User your data models to identify what data should be migrated. For example, you can use the boxes in Business Data Diagrams (BDDs) and you can use the data stores in Data Flow Diagrams (DFDs).
2. You can actually do a two-way check here - if you get data dumps and have fields that are not in your data models, then you may be missing a requirement.
3. Plan for a lot of time to test this. Surely someone will test the migration scripts themselves and someone is likely testing your code, but you also want to plan for testing the integration of the migrated data to the code. So often properities are not setup correctly and so the data doesn't show up in your software, even thoug it's in the database behind the scenes.

RML™ Requirements Model 5 - The Data Flow Diagram (DFD)

The Data Flow Diagram (DFD) is a very useful part of the Requirements Modeling Language (RML™). The Structured Analysis Wiki contains a great explanation of how to create a DFD, so I’m not going to cover that information here. Instead, I’m going to provide one answer to the question “How do I know when to use a DFD?” This answer comes from my own (unique) view of the world, so some of you probably won’t relate to it, but others will--I’m sure there is at least one more out there…

Sometimes I have what I think of as an "equation" in my head. In vague terms, I may be thinking “Customer Data + Product Data + Input from Sales Rep + Taxes = Order”. But that's not really right, nor is the equation a good representation of the information.

And, the “equation” also misses a few other particulars about creating an order that I’d like to convey: Sales Reps update customer data, the Finance staff maintains the rules for the tax calculations, and orders flow to the Order Fulfillment system after creation.

The data flow diagram is great for representing all of this. Here’s my “equation,” expressed as a data flow diagram:

I have found that business users, as well as developers, react well to this model—it provides a “big picture” with which to begin a conversation about creating an order. It paints exactly the picture I want to convey and validate when I’m thinking “Customer Data + Product Data + Input from Sales Rep + Taxes = Order”.

Display this picture, and you’ll get some interesting questions and comments:

• How is customer data populated initially?


• Does the order fulfillment system update the order store with information about the fulfillment of the orders?


• Where does the product data come from?


• Are all of the tax rules manually entered, or is there also an electronic source for them?


• And, maybe, “No, that’s wrong, updating customer data isn’t a separate process. Customer updates need to automatically flow out of changes made when creating the order.”
  

One note: it doesn’t have to be technically perfect to be useful. I often provide “conceptual” DFDs, in that I intentionally provide conceptual, but not technical information. For example, conceptually, there is a “products” data store. Technically, there may be multiple stores: product list, product descriptions, etc. The important thing is that they work together to provide product data. Developers and architects are very receptive to this; they understand I’m illustrating the behaviors of the system without defining the implementation (which, after all, is their job). Oh, and how does the audience know it is conceptual? I put the word “conceptual” in the title!

You may notice I didn’t number my processes. That’s because I rarely decompose them and many people tend to take the numbering as ordering. So, for my usage, numbering adds confusion rather than clarity.

Happy diagramming!

Related Articles:

How To Choose A Software Requirements Model

Six Requirements Models For Agile Projects

Live from REFSQ: Deriving Information Requirements from Responsibility Models

Tim Storer from St. Andrews University in Scotland presented this paper. His underlying assumption is that in large scale socio-technical enterprise systems, you are constrained by the design of the platform you select, integration to other systems, and systems of systems factors. He contends that the functional requirements in these projects are more useful in the procurement process where you select the system to implement, and specifically less useful in the actual implementation. The reason for this is that the system is already constrained by behavior based on what you select. While I don’t entirely agree with the de-emphasis of functional requirements he implied, his overall point is absolutely valid in that you often are in a situation where you must configure the system and adapt business processes around the system.

This paper discusses how beyond the functional requirements, you also need what he calls “information requirements” which help you to configure the purchased system for the given context. These requirements tell you:

• What information is needed


• Who needs the information


• Who produces information for the system


• Flexibility for access to that information


• Consequences of incorrect information flow

And these requirements influence platform configuration, organizational changes, and system integration.

To identify these information requirements, they use “responsibilities” as a starting point. They have defined “responsibility” as “a duty, held by some agent, to achieve, maintain or avoid some given state, subject to conformance with organizational, social, and cultural norms.” They are not goals, but rather more abstract and less formal. They are not concerned with specific types of agents. Then for each responsibility, they determine the resource needs, ultimately leading to information requirements.

In his example, a self service check-in terminal has responsibility of “provide boarding pass”. Then they can look at the resources needs– blank tickets, ticket database which is fed by ticket server, etc.

For the information resources, you then have to look at what happens if it the resource is unavailable, inaccurate, incomplete, late, and even early. And ultimately you get to the details mentioned above to form the information requirements.

What I found useful in this paper is helped validate some of our own thinking we are applying on projects in that we look at something very closely related with data requirements. We’ve written before about our People, Systems, Data (PSD) approach to discovering all requirements using visual models from RML™. In this case, if we break down the Data component of PSD in combination with the People component of PSD, we have something very similar to what Tim spoke to. Very briefly, we use the People models (e.g. Org Charts) to identify the people using the system and then look at what stories they need to execute in the system (e.g. User Stories). Now we can also look at the top-level data model (e.g. BDD – a Business Data diagram), and for each data entity in the diagram, we look at how it is:

• Created


• Viewed


• Changed


• Removed


• Copied


• Moved

(And yes, I deliberately did not call out the CRUD because I’m not a big fan of the acronym!). In doing this, we can actually complete the list of user stories and identify system integration points. Similar to above we would also use the user stories to cross-verify the BDD was complete. Our data actions closely map to the list in Tim’s presentation:

• What information is needed -> the BDD entities


• Who needs the information -> Who views it, changes it, copies it, moves it


• Who produces information for the system -> How is the data entity created


• Flexibility for access to that information -> Who views it


• Consequences of incorrect information flow -> exceptions in the stories that come out of the list

Now in a recent example, we were working with a system that we did not know well. There were six main data objects and a list of about 10 user stories for the system. I wanted to validate whether the list of User Stories was complete, so I proceeded to walk through each of the 6 actions above against each of the six data objects. Not every data object could have those actions performed in the system, but it was useful to deliberately check each. In the end we identified about five new User Stories.