Conducting rigorous qualitative research

Qualitative research often gets criticised for being subjective and prone to bias, and those concerns are probably not unfounded. There are, however, ways in which we can improve qualitative research. Here are some methods:

Member checking

In member checking, the researchers return the data (transcripts etc.) and/or the results to the participants to allow them to assess the trustworthiness and ask for input. Member checking has its basis in epistemological foundationalism, which assumes that this method is neutral and can control for bias, allowing for the sorting of the more trustworthy information from the less trustworthy information. However, since there is no "true reality" in qualitative studies to compare to, it is impossible for researchers to truly objectively sort trustworthy from untrustworthy information. Similarly, there is no evidence that member checking enhances credibility or trustworthiness.

There are also practical problems with member checking. For instance, the participant and researcher may disagree with or contradict each other, though this is rarely reported in papers. In case of a disagreement, participants may be inclined to simply agree with the researcher, due to the power imbalance between participant and researcher. Participants may skim over the data, be unable to understand the data, or simply not remember what they said when the data were initially generated, making them unable to offer truly useful insight. Furthermore, political leanings and personal interests may also influence the member checking process, particularly if the topic studied is a sensitive one (e.g. illicit drug use).

Fortunately, all is not lost. Member reflection may not be perfect for verifying results, but it may help to generate additional data. When participants re-read the data, they may generate useful insights. Another positive aspect of member checking is that it may help to ensure that descriptions of the participants do not accidentally reveal who they are.

Calculating inter-rater/inter-coder reliability

This method is exactly what it says on the tin: trying to calculate some kind of inter-rater reliability metric. As simple and objective as it sounds, calculating inter-rater reliability comes with its own disadvantages. Firstly, different coders may perform unitisation (identify blocks of text for coding) differently to one another, making it difficult to compare coders. Secondly, if coders then combine to discuss their differences in results, power differentials etc. between coders may skew the final answers. Thirdly, it is difficult to determine a satisfactory level for inter-coder reliability.

One way to get around problems such as unitisation differences is to use a set of guidelines. Coders can develop a framework and firm coding rules prior to coding. The downside of this is that it may constrain creativity and limit the identification of unanticipated knowledge.

Another way to get around the problems with inter-rater reliability is to use the "critical friends" technique. In this technique, there is critical dialogue between people. Researchers voice interpretations to others who listen and offer feedback. The goal here is to create a "sounding board" or reflexivity, rather than total agreement.

Yet another way to get around the problems with inter-rater reliability is to simply ignore reliability altogether. After all, you will never be able to reproduce the exact same interview.

Universal criteria

Another possible method for improving the quality of qualitative research is to try and adhere to a set of criteria, such as the universal criteria proposed by Tracy (2010). These universal criteria consist of eight main hallmarks: worthy topic, rigor, sincerity, credibility, resonance, significant contribution, ethics and meaningful coherence. Instead of using fixed criteria, an alternative approach is a relativist one: using criteria from open-ended lists so that the criteria are appropriate for the study design.

References

Smith, B, McGannon, KR 2017, 'Developing rigor in qualitative research: problems and opportunities within sport and exercise psychology', International Review of Sport and Exercise Psychology, DOI: 10.1080/1750984X.2017.1317357

Tracy, SJ 2010, 'Qualitative quality: Eight "big tent" criteria for excellent qualitative research', Qualitative Inquiry, vol. 16, no. 10, pp. 837-851.