How to write the case study methodology chapter of your MA/PhD thesis. The following discussion explains the case study methodology – more specifically, the qualitative exploratory case study methodology – as used in a uOttawa MA in Communication thesis (2015) and a uOttawa PhD in DTI thesis (2020).
- Introduction
- The case study methodology
- Methodology rationale
- Sampling strategy and sampling criteria
- Data collection and analysis
- Coding and the analytic strategy
- Reliability and validity
- Data validation protocols
- Chapter conclusion
You may also be interested in The subject matter expert interview request email: How it’s done.
Introduction
The methodology chapter begins with the introduction section. Off the top, state the research question(s) of the thesis or research project, the proper (formal) name of the research method, and the theoretical framework. Next, give an outline of the discussion to come. Use the past tense or the present perfect tense for the methodology chapter, as required by APA Style guidelines. For example,
This thesis addressed the research questions: 1) What constitutes ethical hacking teaching practices, 2) What constitutes hacking skills, 3) What is the risk to society of teaching students hacking skills, and 4) How to mitigate the risk? This thesis followed the qualitative exploratory case study method (Creswell, 2003, 2007; Stake, 1995; Yin, 1994, 2003) focusing on two Canadian universities as case studies. The thesis applied Bunge’s (1975) technoethical pragmatic value theory (TEI) in conjunction with Karl Weick’s (1969, 1979, 1995) sensemaking model as a sociotechnical theory of society (STEI-KW) to address the research questions (see Table 8: RQs, Data Collection, and Theoretical Frameworks). This chapter first explained the research design and its suitability for addressing the thesis research questions (methodology rationale). Next, the discussion explained the sampling strategy and sampling criteria along with the rationale for the selection of the case studies. Data collection and analysis procedures were then discussed, followed by an explanation of the coding and analytic strategy. The methodology reliability and validity and the data validation protocols were then discussed. Finally, the conclusion section summarized the key points covered in this chapter.
The case study methodology
Research design must address three concerns: Knowledge claims or theoretical perspectives, strategies of inquiry, and methods of data collection and analysis (Creswell, 2003, 2007). Qualitative research takes place in a natural setting which enables the researcher to develop a level of detail about the place or individual and to be involved in the experiences of the participants (Creswell, 2003; Rossman & Rallis, 1998). Throughout the “qualitative research process, the researchers keep focus on learning the meaning that the participants hold about the problem or issue, not the meaning that the researchers bring to the research or writers from the literature” (Creswell, 2013, p. 47). A pragmatic knowledge claim to qualitative research is pluralistic and problem-centred, and is concerned with consequences of actions and real-world practice (Creswell, 2003). Two approaches are suitable for pragmatic research: Experimental and case study (Yin, 2003). Case studies allow researchers to explore a program, event, activity, process, or individuals in depth (Creswell, 2003). A case study deals with contextual variables and relies on multiple sources of evidence. It can be thought of as a comprehensive method, covering the logic of design, and data collection and analysis techniques. A case study,
is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. (Yin, 2003, p. 13)
Case analysis is a particularly appropriate method for the investigation of systems explanations of organizational functioning (Miller, 2009). It aligns with STEI and with Weick’s sensemaking model in their systemic and pragmatic theoretical orientation regarding data collection and analysis, including triangulation via data derived from multiple stakeholder perspectives and the centrality of context and personal experience as sources of knowledge. The qualitative case study methodology is well suited for capturing the unique complexities of a single case (Stake, 1995), when the study focus is on operational links rather than on frequencies or incidences, when little control over events is expected, and when the focus of the study is on contemporary phenomena within a real-life context (Yin, 1994).
The qualitative exploratory case study methodology is particularly appropriate when there is a scarcity in the literature on the subject (Stebbins, 2011). Case study research is suitable to “either develop an in-depth understanding of a single case or explore an issue or problem using the case as a specific illustration” (Creswell, 2013, p. 97). In a single instrumental case study (Stake, 1995), the researcher “focuses on an issue or concern, and then selects one bounded case to illustrate this issue” (cited in Creswell, 2013, p. 99). The thesis followed the instrumental case study approach.
Methodology rationale
The case study (Stake, 1995; Yin, 1989) qualitative research method examines in depth purposeful samples to study a phenomenon such as ethical hacking. It exemplifies the researcher preference for depth, detail, and context, often working with smaller and more focused samples in comparison to the large samples of primary interest to statistical researchers. Exploratory research is suitable for problems or phenomena that are in the formative stages to help clarify primary issues surrounding the problem or to establish priorities, clarify trends or map a field and develop operational definitions (Shields & Rangarjan, 2013).
Qualitative research is iterative and adjustments and changes are a natural part of qualitative work. The qualitative researcher uses reasoning that is multifaceted and iterative “with a cycling back and forth from data collection and analysis to problem reformulation and back” (Creswell, 2003, p. 183). Qualitative research involves “an emergent and evolving design rather than tightly prefigured design” (Creswell, 2013, p. 46). The research process for qualitative researchers is emergent. This means that the initial plan for research cannot be tightly prescribed, and that all phases of the process may change or shift after the researchers enter the field and begin to collect data. (Creswell, 2013, p. 47)
Sampling strategy and sampling criteria
The interview participants were recruited by email with the aid of a formal recruitment letter and a consent agreement to participate in the research. In-depth, semi-structured interviews were conducted within a set time (1 hour) at various university campus locations or by phone. The interview sessions were audio recorded and the relevant parts transcribed for accuracy. Further, hand-written notes were taken during the interviews. The sampling criteria for the interview participants were as follows: Each of the interviewed university experts in information security penetration testing and industry practitioners (those with experience practicing ethical hacking or hiring ethical hackers) had a minimum of one-year experience in information security testing teaching and practice in higher education and in the IT industry respectively; or participants had at least one year experience in information security or IT policy or policy analysis in academia. Interview participants were sought out for their expert knowledge in 1) scholarly research in ethical hacking education and practice; 2) current practices and trends in ethical hacking education and practice; and 3) organizational communication practices in ethical hacking education or industry experience in practicing ethical hacking or mentoring or training or hiring ethical hackers. The participating universities were chosen because the needed expert knowledge was found there.
Data collection and analysis
The theoretical framework (STEI-KW) guided data collection and analysis. Structural and behavioral analysis of society as a social system points to society as a liberal open society founded on the ideals of the Scientific Revolution and the Enlightenment. Open hacking technologies are the focus of the study. This is justified on two accounts, 1) since sociotechnology is understood as social technologies or social technological ontologies, open society is ontologically co-extensive with open technology (emphasizing the nature of technology as a social construct); hacking skills are ontologically coextensive with hacking technologies, hacking methodologies, hacking values, and so on; and 2) since the key structural property of a ST society as theorized in the thesis is its open nature, studying digital hacking technologies at the intersection of society and technology means focusing on open hacking skills and open hacking technologies. Put differently, according to the STS approach, hacking technologies as socially constructed are understood to mirror the society that produces them (society is theorized as open, scientific, and knowledge making), hence the focus on open hacking technologies.
Data collection and analysis consisted of systematic literature reviews (Jesson, Matheson, & Lacey, 2011; Okoli & Schabram, 2010), organizational document reviews of two Canadian universities, and in-depth interviews with 14 interview participants (in addition to one participant who contributed via email) comprised of university experts and industry practitioners of ethical hacking and policy experts. Numerous secondary resources were consulted including governmental and business/industry resources, policy reports, industry white papers, and many websites. For research questions 1-2: Systematic literature reviews and organizational document reviews were conducted comprised of about 50 pages of organizational documents available on public web pages of two Canadian universities. For research questions 3-4: Narrative literature reviews were conducted augmented with input from RQ1 and RQ2 SLRs. For research questions 1-4: In-depth interviews (Jackson, Gillis, & Verberg, 2011) were conducted with university experts, industry practitioners, and policy experts. RQs 1 and 2 are addressed in the Findings chapter. RQs 3 and 4 are addressed in the Advanced Analysis chapter. RQ3 is addressed under technology assessment and Teaching vs Practice (the case studies), while RQ4 is addressed under the risk mitigation discussion (recommendations). (SLRs for RQ1 and RQ2 have informed RQ3 and RQ4. Rather than conducting SLRs for RQ3 and RQ4, the researcher opted to focus on extant government and business/industry research/reports of clear and direct relevance to the thesis, e.g., CSE’s, 2018, cyber threat assessment report, and Kool et al.’s, 2017, state of the art research on social digitization, its key technologies, and potential impacts on society.)
A systematic literature review was conducted for RQ1 What constitutes ethical hacking teaching practices? Four key themes emerged: Professional ethical hacking is legal, Ethical hackers are trustworthy, What do ethical hackers do? and An identity and legitimacy crisis. A systematic literature review was conducted for RQ2 What constitutes hacking skills? Three key themes emerged: Steps of the penetration testing process, Open source penetration testing methodologies, and The penetration test report. Contribution to knowledge of RQ2 was delineated by the theoretical framework and focused on open/open source technologies. Further, RQ2 (hacking skills/knowledge) was subordinate to RQ1 (i.e., “who are ethical hackers and what do they do” included a synthesis of a foundational framework/profile for professional ethical hacking practitioners–the meanings, ethics, values, skills/knowledge, roles and responsibilities, and practices).
A narrative literature review was conducted for RQ3 (risk assessment) focusing on pragmatic technology assessment using STEI-DMG, which is concerned with the ethics of teaching students hacking skills or the ethics of using hacking technologies in ethical hacking teaching practices in higher education (weighing opportunities against threats) invoking the precautionary principle (the risk of not teaching students hacking skills). Are the teaching practices in tune with societal needs and incorporate the interests/values of key societal sectors/stakeholder groups? Open coding was performed during a first pass through the data for what constitutes ethical hacking teaching practices. Coding coalesced around three main themes that are discussed within the broader Teaching vs Practice cybersecurity skill gap context: Teaching ethical hacking skillset, Pedagogy as Communication, and Technology Assessment: An Integrative Approach.
A narrative literature review was conducted for RQ4 (risk mitigation) focusing on S&T innovation initiatives. Applying EDP-STEI-KW to advise ethical design of ethical hacking teaching practices pointed to (recommendations) the role of OSINT Analyst, a novel cybersecurity role synthesized to meet the needs of society, and the foundation framework of a body of knowledge for the role. Applying EDP-STEI-KW to advise ethical governance of using digital hacking technologies in higher education and in broader society pointed to (recommendations) the professionalization of ethical hacking as an occupation/the licensing of professional ethical hacking practitioners, and to a public policy initiative comprised of a networked centre of excellence of ethical hacking communities of practice as a research and governance approach and the policy innovation decision making framework of SSP-DMG.
Organizational documentation consisted of 50 webpages concerning program course requirements and course descriptions of undergraduate courses in CS, CE, and SE programs (3 credit) courses taught in English for 2019-2020 at the two participating higher education institutions. Online course descriptions at the two participating research institutions were surveyed for technical and social hacking skills focusing on network penetration testing high-level concepts. Courses not directly teaching computer network skills were excluded from the analysis (courses with “security” or “secure” in their title were retained for examination given their direct relevance). Finally, the study focused on courses cross-referenced between the two participating universities. Program requirements for CS/CE/SE majors were examined for required courses in technical hacking skills and social hacking skills, the latter includes ethics and social science. Further, two courses were closely examined: The syllabus for a computer systems security course and the syllabus for a professional practice course for insights into communication practices (pedagogy as communication) and insights into what constitutes professional practice. See Organizational Document Review. Finally, bachelor degree programs in CS/CE/SE disciplines were surveyed for inclusion of security majors/specializations.
Organizational Document Review
In-depth, semi-structured interviews were conducted with 14 interview participants (in addition to one participant, PPT15, who contributed via email) between December 7, 2018 to April 15, 2019: Four ethical hacking university experts, four ethical hacking industry practitioners, and six policy experts (see Interview Participants by Area of Expertise).
Interview Participants by Area of Expertise
In-depth interviews are typically done “to solicit people’s descriptions and explanations of events taking place in their own environment” (Eid, 2011, p.10). Advantages of conducting in-depth interviews include more researcher control over the line of questioning, and the ability to obtain historical and primary information (Creswell, 2003). In-depth interviews allow researchers to collect the respondents’ perceptions of their world. Interview quotations are used to illustrate key analytical points. Combining in-depth interviews with a document review enables the capturing of explicit as well as tacit knowledge surrounding organizational practices.
Coding and the analytic strategy
Data analysis involves systematically organizing, integrating, and examining data, searching for patterns and relationships in the details. The “recursive process of analysis begins immediately with the first data-collection episode and continues throughout the study” (Jackson, Gillies, & Verberg, 2011, p. 242). “To analyze, we connect particular data to concepts, advance generalizations, and identify broad trends or themes. Analysis allows us to improve understanding, expand theory, and advance knowledge” (Neuman, 2011, p. 341). After coding, concept building, and emergence of key themes, analytic strategies are applied for the analysis of the data–strategies that link data to theory. In qualitative research, coding or “concept formation is an integral part of data analysis and begins during data collection”–conceptualization is a way to organize and make sense of data. The research questions provide a guide but the data analysis process often leads to new questions. Theory is used to interpret the findings (Neuman, 2011, p. 344). Data analysis means making conceptual connections of the data or searching for patterns in the data. “Once you identify a pattern, you need to interpret it in terms of a social theory or the setting in which it occurred. This allows you to move from the particular description from a historical event or social setting to a more general interpretation” (Neuman, 2011, p. 351).
Data coding was performed against the theoretical propositions (Yin, 1994) of STEI-KW. The illustrative pattern matching method (Neuman, 2010) was applied as the analytic strategy. The illustrative method anchors or illustrates theoretical concepts with empirical evidence. It applies theory to a concrete social setting and organizes data based on theory. “Preexisting theory can provide conceptual empty boxes that you fill with the empirical evidence” (Neuman, 2011, p. 353). In the pattern matching variation of this analytic strategy, patterns or concepts identified in the case studies are matched to those derived from theory. Open coding was performed during a first pass through the data for the ethics, values, meanings, skills/knowledge, roles and responsibilities, and practices of professional ethical hackers and ethical hacking. The interviews were transcribed first, and two coding tables were created–Table 9: Hacking Skills Coding Table (Network Penetration Testing) and Table 10: Professional Ethical Hackers Coding Table. Open coding themes from the interviews, the literature reviews, and organizational documents were extracted and incorporated in the coding tables.
Reliability and validity
Reliability and validity are concepts that address the truthfulness, credibility, or believability of findings (Neuman, 2010). Reliability refers to the replicability of a researcher’s results–the extent to which another researcher can make similar observations under identical or very similar conditions (Creswell, 2003; Neuman, 2010; Stake, 1995; Stebbins, 2002; Yin, 1994). Reliability means dependability or consistency (Neuman, 2010). Researchers must be consistent in how they make observations; for example, through the use of explicit interview questions and research procedures (Neuman, 2010; Yin, 1994). Validity in exploratory research (credibility or trustworthiness) refers to whether a researcher can gain an accurate impression of a group, a process, or an activity, and how so (Stebbins, 2002). Validity suggests truthfulness. It refers to “how well an idea ‘fits’ with actual reality”; or “how well we measure social reality using our constructs about it” (Neuman, 2011, p. 175). Qualitative researchers are more interested in achieving authenticity than in realizing a single version of Truth (Neuman, 2010). Authenticity means, “offering a fair, honest, and balanced account of social life from the viewpoint of the people who live it every day” (Neuman, 2011, p. 181). Reliability requires clarity about the followed procedures of data collection, analysis, and interpretation to ensure consistency. Hence researchers are encouraged to develop a case study protocol, keep an organized case study database, and maintain a chain of evidence (Yin, 1994). Reliability also requires clarity on the logic linking the data to the research propositions or questions, the operational measures used for the concepts or theories, and the criteria used to interpret the data (Yin, 1994). The thesis enhanced the reliability of the research methodology by providing details about the participant recruitment process, the data collection methods (the interviewing process and interview questions, as well as documentation gathering), and data analysis.
Saturation is a popular strategy for the trustworthiness of findings. Data saturation or information redundancy is the point at which no new themes or codes emerge from the data. The researcher did not find it helpful to “operationalize” the concept of saturation to determine a priori the number of interview participants that would be sufficient to achieve coding reliability that somehow faithfully reflects the facts out there–as this presumes the researcher is not an active agent who interacts with the data subjectively and intersubjectively to construct knowledge that reflects “facts” inextricably mixed with values and interests. I agree with Braun and Clarke (2019) that while the concept of data/thematic saturation is “coherent with the neo-positivist, discovery-oriented, meaning excavation project of coding reliability types of (Thematic Analysis),” it is not consistent with the values and assumptions of reflexive TA. I agree with them that researchers using reflexive thematic analysis ought to “dwell with uncertainty and recognise that meaning is generated through interpretation of, not excavated from, data, and therefore judgements about ‘how many’ data items, and when to stop data collection, are inescapably situated and subjective, and cannot be determined (wholly) in advance of analysis.” The researcher’s approach to TA/to capture patterns of meaning across datasets was reflexive, probably a mix of following a deductive way where “coding and theme development are directed by existing concepts (STEI-KW guided data collection and analysis) and, more importantly, following a constructivist way. A constructivist approach puts emphasis on sociocultural context and on personal experience as sources of knowledge. For the researcher, saturation as a milestone in data collection and analysis has to do more with self-awareness than with correspondence to facts or reality. The researcher does not believe they went out there and discovered the facts; rather, the researcher interacted with the data and interpreted it based on the researcher’s experiences in life and the broader social totality that shapes the researcher’s views and values.
The theoretical framework STEI-KW guided data collection and analysis. Further, the researcher’s past work and experience on the topic of ethical hacking helped them identify key themes. Systemism (Bunge, 1979) instructs that the proper study of society is “the study of the socially relevant features of the individual as well as the research into the properties and changes of society as a whole” (p. 14) and hence pointed the researcher to the need to understand the professional attributes of ethical hacking practitioners. The researcher went into the interviews searching for insights about the socially relevant features or professional attributes of ethical hackers that can serve as a basis for sketching out a professional practice profile–the meanings, ethics, values, skill/knowledge, roles and responsibilities, and practices, as open coding elements. Further, the researcher went into the data collection interviews looking for “ST hacking skills”–that is, for technical hacking skills and social hacking skills as two broad categories or themes when discussing ethical hacking technology use/teaching practices.
Data validation protocols
Method validation protocols included: 1) Triangulation of measure (Neuman, 2011) or triangulation of data (Yin, 1994): Different sources of data and different measures (perspectives) of ethical hacking practices were used in order to increase the validity of the study; 2) triangulation of method (Stake, 1995): Three data collection methods were used—in-depth interviews with subject matter experts and stakeholder groups, organizational document reviews, and systematic literature reviews; 3) triangulation of observers (Neuman, 2011) or member checking (Stake, 1995): Participants were consulted on the findings (the interview transcripts) so as to counter researcher bias (perception and interpretation) and to ensure the accuracy of quotes; and 4) triangulation of theory (STEI-KW): Two complementary theoretical lenses, STEI and the KW, were used to situate organizational ethical hacking practices within the broader industry and social contexts.
Chapter conclusion
This chapter first addressed the methodological justification for the thesis. It then explained the research design. This was followed by a statement about the rationale for the selection of the research site and sampling strategy. Data collection and analysis procedures were then discussed. An explanation of the implemented data validation protocols followed. Finally, the methodology reliability and validity protocols were discussed.
Related content
Ethical Hacking Sociotechnology
How to choose a PhD external examiner
How to do a systematic literature review and a technoethical assessment of a technology
Literature review types with examples
STS journals – technology and society journals
The subject matter expert interview request email: How it’s done
