Gathering Data

I gather both quantitative and qualitative data about organizations. I use commonly accepted practices for collecting and interpreting qualitative and quantitative data. I employ the technique of ethnological study to software organizations. Ethnography consists of looking at what people do, what they say, and the tension between what they really do and what they ought to do, as well as what they make (artifacts)1.

Quantitative Data

I try to gather up as much quantitative data as possible. These are hard facts about an organization. I gather data at the project and application levels and it is rolled up to the organizational level. The following is a list of some of the quantitative data I gather:

Metrics and Ratios

I develop a set of metrics (ratios) using the quantitative data. One of the metrics I calculate is organizational cost per function point. This metric is based upon the total organizational cost divided by the total number of function points. I also calculate project cost per function point or hours per function point. Often, there is a large gap between project costs per function point and organizational cost per function point.

Too often the only costs that get reported and recorded are programming costs not total organizational costs. Actual programming costs represent between ten and twenty percent of total software development costs.

The internal IT hourly rate is around $150 per hour. This puts total organizational cost per function point at $5,000 per function point. Most IT organizations are understating their costs by orders of magnitude because they are not including everyone’s time. As an example, it is common not to include management or user time as part of project costs. Often, there is a large gap between organizational cost per function point and project cost per function point.

I try to observe and collect other types of quantitative data too. I make notes on the consistency of documentation, naming conventions used, and verb usage.

Qualitative Data

Over the past twenty years I have developed a list of fifty qualitative factors that impact productivity and I list these in Appendix A. Over the years I have refined the list by getting feedback from both academics and practitioners. Each of these factors is given a score between 1 and 5 where 5 is the highest possible score.

I am often asked to explain how I came up with my evaluations, so I need to have supporting reasons and documentation. The qualitative scores are based upon actual artifacts where possible. It is important to have supporting detail especially artifacts. When I am working with a company I do not makes requests similar to, “Rate your project management skills on a scale of 1 to 5” because everyone knows the right answer. Instead, I rely on artifacts where possible. If a company is a five in project planning, it will have project plans to examine.

My analysis of a company starts during the first conversation I have with a client. I keep copious notes on clients with notebooks full of interviews and data. I generally conduct informal interviews. I like to talk to people, and I ask a lot of questions. I do not like to sit in a conference room and call individuals in for an interview and start asking a series of questions. I have learned it is best to gather information during informal conversations while drinking coffee or enjoying a meal.

I need to know if I am getting good data from people. When people respond rapidly with confidence to my data requests, it is usually an indicator that the data is of high quality. On the other hand, when a person is moving his head back and forth and exhaling as he is explaining the data, it indicates potential problems.

I use a technique developed by behavioral psychologists called Conversation Analysis (CA)2. CA is a method to transcribe conversations using a detailed notation system. I notice what a person says and how it is said. Sometimes, people respond to questions with an elevated voice, as if they are asking a question. I denote this with an upward arrow followed a question mark. It is common for people to sigh before making a statement; they either inhale (.hhh) or exhale (hhh), and I make notation of this too.

Qualitative Scores And Productivity

There is a strong correlation between high qualitative scores and productivity3. It is not by accident that I use the word correlate and not the word causes. Correlate and causation are not the same thing at all. It is similar to figuring out that wealthy people listen to classical music and concluding that if you listen to classical music, you too will be wealthy. Listening to classical music is a characteristic of the wealthy; it is not what made them wealthy.

Project Success and Project Failures

Large projects with high qualitative scores in the area of project management skills4 were more likely to be successful than projects with low scores. Those large software projects with more than 10,000 function points are more likely to be successful when they brought in skilled project managers from the outside.

1Creswell, John W. Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, Calif: Sage Publications, 1998.

2Denzine, Norman K., and Yvonna S. Lincoln. Collecting and Interpreting Qualitative Materials. London: SAGE, 2003

3The R2 is in excess of .80 with a p value of less than .01.

4I list about a dozen or so project management skills in appendix A