Critique of a Mixed Methodology Research Study
The purpose of this paper is to critique the mixed methodology research study, Identifying useful project management practices: A mixed methodology approach, authored by Gabriela Fernandes, Stephen Ward and Madalena Araujo.
The title of the article is concise and clearly represents the research project as well as the methodology. The abstract describes the study as a mixed methodological research approach; the authors specify the importance of the research as having the potential for increased contribution to project management performance. The research methods are discussed, including the qualitative as well as the quantitative components. The rationale for mixed methods research is not clearly articulated in the abstract (although it is described later on in the article). The authors identify a research question as: What are the most useful project management practices? This is an appropriate research question for the type of study being done in that the authors’ objective is to identify the list of the most useful practices. The authors report the results obtained, which align with what they set out to do.
The authors clearly discuss the importance of this research and provide statistics from Standish Group International showing that in the year 2008, only 32% of all software projects surveyed succeeded, 44% were challenged, and 24% failed. These statistics provide a compelling argument for the need to find those project management practices that will improve project performance. The purpose of the study is to make a contribution to the identification of those practices that improve project performance so that organizations can make decisions as to which practices to adopt.
The literature review cites 41 articles, 17% of which were published within the past five years. Generally speaking, research is considered relevant when 85% of the literature review is taken from articles published within the past five years, for that reason, the literature is considered to be dated. The existing literature was used to provide a basis for comparing the findings of the study (the list of useful tools). Although the literature showed consistency in the identification of the most useful tools, it showed a distinction between the value of some of the tools, which is identified as intrinsic value (present extent of use + potential improvement). The authors advise that for the purposes of this study, the more relevant information pertains to intrinsic value, as this research study is focused on identifying those practices that are most useful in project performance.
The research methodology is explained in detail. The authors explain that this is a mixed methods study which uses two research instruments: semi-structured interviews and questionnaires. Mixed methodology is appropriate for this study in that the researchers intended to use the results of the qualitative component to augment the development of the survey instrument for the quantitative phase.
The first phase of the study was the qualitative component and the authors specifically discuss how the interviews were conducted, as well as the questions that were asked. This level of transparency adds clarity to the study and allows the study to be replicated. The authors also do a thorough job of detailing the results of the interview and create easy to read tables identifying the findings. The second phase of the study was a worldwide on-line survey questionnaire. The questionnaire is described in detail, including instructions for participants as to how they should score their responses, (researchers used a five point Likert scale). The list of project management practices identified in the survey was derived from the results of the qualitative component as well as the literature review. The authors also added project management practices that had been discredited in the literature review which demonstrated an unbiased approach to the research. The survey instrument is described in enough detail to allow for replication, adding credibility to the study.
The questionnaire was distributed via the researchers’ professional contacts who also were encouraged to recruit additional participants, (this is known as the snowballing technique). This technique has the potential to significantly increase the number of participants (in effect, reaching a population). Since the purpose of the study was to cover Project Management practitioners all over the world, this type of technique is appropriate for pursuing this objective. Approximately 3,000 surveys were distributed and 793 completed questionnaires were received from practitioners from 75 different countries, yielding a 26% response rate. Given the length of the survey, which took around 15 to 20 minutes to complete, this is an acceptable response rate. The participants’ organizational roles were reviewed and the authors reported that 70% of respondents were either Project or Portfolio Managers or Project Team Members, as such, this is the appropriate respondent for this research. The fact that participants were from 75 different countries adds credibility to the study in that this is a study attempting to identify useful practices from the project management community.
The researchers provide descriptive statistical analysis of the quantitative component to identify the list of practices, including the mean, median, mode and standard deviation. Standard deviations show low values which indicate a low variability of answers. They include data as to missing values, none of which are significant (the highest number of missing values is 89 and that is of 704 responses, 13% missing). Using descriptive statistics to formulate a ranked list is appropriate for what the researchers set out to achieve.
The authors do not share any of their reliability numbers, however, they do advise that the interview data was analyzed through thematic analysis and the use of Nvivo software, both of which are useful tools for theming the data. Nevertheless, measures of validity and reliability are clearly missing.
The researchers did accomplish their objective to make a contribution to the identification of those practices that improve project performance. Their list of most useful practices identified from the survey were largely similar to the set identified from the interviewees and to the existing literature. The authors conclude by advising that the quantitative results will continue to be analyzed to find if the most useful practices ae dependent upon organizational context.
While the researchers do show a high level of transparency with respect to their research methodology, they are not forthcoming in terms of their reliability numbers and this is viewed as a weakness of the study.
References
Fernandes, G., Ward, S., & Araújo, M. (2014). Identifying useful project management practices:
A mixed methodology approach. International Journal of Information Systems and Project Management, 1(4), 5 – 21.
