Many in the statistics profession, including myself, do not believe in random events. Some statisticians and mathematicians go mad trying to find patterns. There is no need to go mad in the domain of software development because many of the patterns are blatantly obvious. The process only appears to be random because it has not been studied. It is common for a clinical psychologist to see patterns in an individual’s behavior while the individual is totally blind to his or her own patterns of behavior. One of the functions of a clinical psychologist is to get people to recognize these patterns and behaviors.
The most common way for psychologists to get people to recognize their patterns is through measurement and keeping a journal.
Many software organizations write journals which are more commonly known as lessons learned documents. I often review lessons learned documents, and it is painfully obvious the same patterns repeat over and over again. If all the past lessons learned documents are pointing to the same problems, then it is safe to conclude the next lessons learned document is going to point out the same problems. There are only three possible reasons for software organizations to keep repeating the same behaviors: the first is denial and ignoring the glaring fact that the same bad stuff happens in every project, the second is “not my fault syndrome” or nothing can be done about it, the third and final reason it is just too hard to change.
Psychologists often talk about the elephant in the living room. The person or family has learned to ignore something as obvious as a giant elephant in the living room. When bad behaviors are pointed out to individuals, the initial response is often denial or anger, and later on, it is justification. Individuals begin to justify bad behaviors due to circumstances beyond their control. Software organizations generally blame customers for poor requirements. Now, if nothing can be done about the lack of requirements or changing requirements, then there is no point in trying to measure the requirements process; right?