‘Horses for Courses’: The Differences Between Quantitative and Qualitative Approaches to Research

Abstract

Some clinicians may feel dissociated from, and intimidated by the ever-increasing emphasis on research. However, with an understanding of some of the basic principles and key terms, research can feel less daunting. It is the aim of this article, the second in a series of three focusing on understanding research, to introduce clinicians to the different approaches to research, to improve understanding of what the approaches mean, and to highlight when a particular approach may be appropriate. Furthermore, the article will provide an explanation of some of the common terms used within clinical research. This should aid the clinician in applying good, simple, scientific principles to evaluating clinical research evidence. (J Am Podiatr Med Assoc 92(3): 159-169, 2002)

Introduction

In the current health care environment, there is an increasing need for clinicians to be familiar with the philosophy of evidence-based practice. Best practice will result if clinical practice is based on the outcomes of high quality clinical research which is in turn, tailored to the individual characteristics of each patient (Dowie, 1996; Ellrodt et al, 1997). However, clinicians may sometimes feel daunted or intimidated by the ever-increasing volume of research, and seemingly constant change in research practices. As a result it can be difficult to know how to start developing an understanding of the research process. It is the aim of this article to foster a basic understanding of the research processes through an exploration of some of the underlying principles of good research.

The Barriers to Research: Some Common Misconceptions

One of the major barriers to clinician involvement in the research process, either as participants or as discerning consumers, is the presentation of the theories and ‘rules’ associated with the research process (Payne, 1999). While daunting to the newcomer, this should, in fact, be expected when confronting any new area of knowledge. For example, consider the volume of information one has to digest when confronted by the choices available for selecting appropriate wound dressings. Initially, there naturally needs to be an investment in learning some of the language and baseline knowledge. Only then are we able to make a decision as to what may be the most appropriate wound care approach or product for a particular patient. The field of research is no different in that it appears daunting to the novice, but when we apply ourselves to learning the basic principles and terminology, we can make the new found knowledge work for providing optimal patient care. It is as much our responsibility to acquire the knowledge necessary to synthesise research on behalf of our patients, as it is to acquire the clinical knowledge that underpins our work.

It may also be helpful immediately to allay the myth that research is primarily about manipulating numbers. Research does not inherently equate to statistics, and mathematical genius need not be a pre-requisite to understanding research. Research is concerned with the appropriate approach to generate, test or validate knowledge using credible techniques (Higgs, 1997). As such, statistical analysis is only a small (albeit sometimes important) component of the overall process. Rest assured that even the best clinical researchers have their limitations when it comes to understanding statistics.

Another tension arises in the apparent conflict between ‘pure’ research and clinical practice. Usually in the clinical setting, there is only one subject (the patient in front of you), who presents with a multifactorial case composed of many intrinsically complex and linked variables. Conversely, in most types of research, there will be many subjects, and these are usually deliberately and artificially selected to emphasise the role of only one variable. While the contrast between the two situations is clear, the objective of both is clearly to provide effective and efficient patient/client management. The clinical research methods outlined in this paper are specifically directed toward this role, and exist to bridge the gap between clinical practice and abstract research.

The first step in bridging the gap is to identify outcomes or measures that are applicable to both situations. For example, in evaluating the effect of foot orthoses, the ‘pure’ lab researcher may deem it appropriate to investigate the effect of orthoses on pronation velocity, a measure considered highly appropriate for determining the mechanical properties of the therapy. However in a clinical setting, a patient and their practitioner will generally be more concerned about symptom relief than some esoteric biomechanical phenomenon. The real life application of evaluating a treatment in terms of its effect on symptoms is clearly more applicable to the clinician and researcher alike, and is today being reflected more in many areas of clinical research. It is a good example of where clinician and researcher have shared goals.

Finally, clinicians may perceive the skills associated with conducting or interpreting research as having few similarities to the skills of the clinician. In fact, choosing the correct research for a problem is closely analogous to choosing the best approach to a clinical case, and as such uses very similar skills. The process of clinical decision making is a complex one, governed by three key factors as presented in Figure 1: evidence, patient and clinician factors, and constraints. These same basic features are also involved in choosing the appropriate research approach. In this instance, evidence or an appropriate knowledge of the area equates to an understanding of the basic principles of the different types of research; the individual characteristics are the type of information you wish to gather or the outcomes you wish to measure; and the constraints are financial support, time, access to equipment, and access to participants etc (see Figure 2).

Using Research to Ensure Best Practice

Few active practitioners in busy practices have either the inclination, or more particularly the time, to become active participants in the research process. While some will become involved, for the vast majority research is considered an activity which is not central to their day-to-day activities, and as such can be left to others. This can, however, lead to the misconception that because the research output is coming from others, there is no need for the practitioner to understand the workings of this research. This is definitely not the case, and even the practitioner with no interest in active participation in the research process has some professional responsibility to be equipped with the skills necessary to make informed decisions regarding the latest techniques and practices published in the professional literature. It is important to be a research sensitive clinician, even if one is not a research active clinician.

Why Good Science is Important to Clinical Practice

Scientific thinking is not a new phenomenon, indeed the development of scientific method can be traced back over many centuries (Bland, 1995; Polgar and Thomas, 1995; Spilker, 1996; Silver, 1997). The roots of modern, western scientific thinking first appeared in the 16th century. This period was characterised by a shift in the explanation of complex phenomena through speculative theorising—or acceptance of ‘divine will’—towards a more sceptical viewpoint where complex phenomena demanded empirical explanation (Silver, 1997). The emergence of new models of explanation such as Newton’s laws of motion, and Galileo’s astronomical models, marked a shift towards empiricism that has held true to this day.

It can be useful to be reminded of this derivation of the principles of good science, as it encourages the practitioner who may consider him/herself either a non-scientist or a passive scientist, to reflect that the founding tenets underlying our professional lives are, in fact, rooted in a great tradition. The purpose of the rules governing the ‘scientific method’ is not to make science mysterious or inaccessible, but to give us a validated framework for a logical process in developing our understanding of our clinical practices.

The basic tenets of the scientific method are observation, description and measurement, and in some cases, control. The astute reader will have already noticed therefore, that science embraces both the qualitative and quantitative approaches. If we take a clinical example such as the link between smoking and lung cancer, we can see a clear progression in our understanding of the association over the last half-century. The progression leads us from a point where cigarette smoking was actually promoted as health giving, to the current state of knowledge where the links between smoking and lung cancer and heart disease are considered all but irrefutable (Davis, 2000). In the course of developing this current level of understanding, it is adherence to the principles of good science that has led generations of researchers, logically, to an understanding of the process. First, it was noted qualitatively, that there was the possibility of a link through describing populations and measuring the incidence of disease. Then empirical, lab-based research examined the effects on tissues; quantitative clinical trials in animals and humans identified the disease manifestations; and finally, treatments and behaviours have been identified which can be shown (both quantitatively and qualitatively) to contribute to some degree of control over the sequelae of smoking.

The Two Philosophical Approaches: Qualitative and Quantitative Research

This, the broadest classification of research approach, is usually based on the nature of the clinical question and the type of questions to be addressed by the project. Qualitative research attempts to verify or generate a theory gleaned from gathering broad descriptive information in a natural setting. As such, it is often also referred to as naturalistic research. Quantitative research is different in that the aim is to answer a specific research question by showing statistical evidence under a strict set of guidelines. Quantitative approaches include typical experimental research, which we see often in the medical literature (Bailey, 1997). The broad differences between qualitative and quantitative approaches are presented in Table 1.

It is important to note that qualitative and quantitative methods are not necessarily mutually exclusive, nor are they inherently contradictory in nature. Many techniques, such as interviews and surveys may approach a problem from either a qualitative or a quantitative perspective or in some cases, a mix of the two. Indeed, a research question may be asked primarily in a qualitative manner to identify issues key to an individual, and then tested on a group in a quantitative manner to ascertain whether this proposition applies to a larger population. As such, it is important to appreciate each approach, and to identify their respective strengths and limitations. Neither a qualitative nor a quantitative approach is inherently superior, but one may be clearly more suited to certain circumstances. Once again, an understanding of the applications and limitations of each of the major research methods is essential for the participating researcher but is also important to the reader of the resulting published research.

As was outlined previously, qualitative research can be broadly defined as a type of investigation in which there is attention to the social or environmental context that frames the research question. In qualitative research, it is common for the initial research question to be broad, or for it to be defined or refined by results of incoming data. Quantitative research is sometimes erroneously described as that involving numbers, and qualitative research that which does not. In reality this distinction is over-simplistic, as there are examples of both types of research successfully breaking with this convention.

It is unfortunate that qualitative research has gained a sometimes poor reputation among members of the scientific community. In part, this is based on the labeling of much poor quantitative research as ‘merely qualitative’ or ‘too qualitative’. This need not be the case. Within any experimental model there can exist both rigorous and less rigorous research, and the same will hold true for quantitative and qualitative methods. It follows therefore, that any good qualitative research must follow some basic formalised and set criteria. Dwyer (1996), quoted in Higgs and Adams (1997) identifies three key criteria to identify high quality, qualitative research. These are presented as a checklist in Table 2.

Qualitative research is based on generating knowledge. It is often referred to as gathering contextually ‘rich’ knowledge, in that the information gathered will inform in detail, but without necessarily attempting to generalise to a population. The interpretation of knowledge is in turn, based on different philosophies of how knowledge is generated. These different philosophies are described in Table 3.

The techniques used to evaluate the information based on the philosophy of knowledge are wide and varied. The most common techniques include structured observation, interview techniques and survey methods.

Whereas qualitative research usually focuses on understanding a problem in its broader context (sometimes called the inductive approach), the more quantitative, deductive approach involves a process of breaking a problem or phenomenon into its component parts and then analysing the parts to predict the function of the whole. Within the conventional scientific paradigm the deductive model tends to predominate. It is arguable as to whether this is for better or for worse.

The process of deduction has been defined as follows (after Spilker, 1996).

• Theories are formulated, derived from the inductive/qualitative process as outlined above, or from simple observation. (The theories are usually based on the classification of factors within a phenomenon, and the later refinement of these observations into testable hypotheses.)
• The hypotheses are then subjected to testing under controlled conditions (the crux of the scientific process) and are duly either verified or falsified.
• Where a hypothesis is verified through experimentation the theory builds.
• Where hypotheses are found to be insupportable in experimentation, the theory is either revised and re-evaluated, or if no consistency can be found it may be discarded entirely.

Approaches in Practice

Unless we suggest that podiatry has no roots in science, then it follows that we need to subject our practice to the rigours of the scientific process in some form, either qualitatively or quantitatively. This can be a hard road to follow, but a good scientific clinical research profile is probably the key to developing the credibility of the profession—and there are no short cuts.

The subsequent pages focus mainly on the quantitative approach, however it is important to reiterate that many of the qualitative or inductive techniques may be equally applicable, or even more applicable in certain circumstances. Sometimes the boundaries are purely artificial and become blurred, hence surveys, interviews and structured observation can all contain a quantitative element, just as epidemiology can be qualitative in some applications.

Structured observation. Structured observation occurs where an individual is observed in a formal manner, and behaviour or other outcomes are recorded. The fundamental principle of structured observation is that the research does not intervene or manipulate the conditions of the study. All information is gathered from observing and analysing the situation. Observational studies are advantageous in that they are generally easier, faster and less expensive than experimental studies. However, as the researcher has less control over the conditions, there are limitations in how broadly the conclusions can be applied.

Interview techniques. Several forms of interview may be applied to an investigation. These range from one-on-one interviewing to a large group approach. The advantages of the interview are obvious: rapport is developed between the researcher and the participant, and questions or responses may be clarified or followed up. Interviews may be face-to-face or over the phone, with face-to-face interviews offering the additional advantage of allowing the researcher to observe non-verbal cues and sense confusion or lack of comprehension on behalf of the participant. The limitations of interviewing are the time required to set up and conduct the interviews, transcription, and the difficulty in interpretation of the information.

Interviews may be structured, where a formal sequence of events and wording of questions are followed; or unstructured, where the procedure is less prejudged and questions asked will in part depend on the responses from earlier questions. In unstructured interviews, the onus of responsibility lays with the researcher to search for the questions that address the research objective.

Unstructured interviews are often used in pilot studies to develop parameters for a study (Bailey, 1997). Such interviews allow the researcher to explore topics which may not have been considered prior to the development of a questionnaire but which transpire to be relevant to the sample of respondents. Unstructured interviews are also useful when exploring difficult or sensitive topics as they allow for personal reflection and individual responses.

There are several structured group interview techniques, including Functional Analysis Workshops (Folch-Lyon and Trost, 1981; Morgan, 1992; Murphy et al, 1992) and Nominal Group Techniques (McKenna, 1992). Each type of interview has a specific aim in questioning a group of participants who have been sampled from a particular population. Clearly, the selection of each requires careful consideration.

Survey methods. Surveys are an inexpensive and relatively fast way of gathering information from a large population, where information is sought either through written or verbal questioning. Surveys must be developed with consideration to the objective of the study and must be unambiguous, non-directive (or non-leading) and relevant. Also with any survey methods, it is crucial that the population is clearly defined and appropriately sampled.

Clearly there are significant problems with any research that relies solely on self-reporting. Participants must understand the question, have the appropriate responses available to them and answer honestly. In analysing any questionnaire, the interpretation and analysis of the information collected must be undertaken judiciously and conclusions must be drawn with care (Bailey, 1997). Survey technique is one area where the methodology has become highly developed, but is also an area where many novices believe that questions can be simply made up on the spot. As with all forms of research, the ‘rules’ associated with survey techniques exist for good reason. Often a short time spent with a primer text would help most people avoid making elementary and avoidable mistakes when planning a questionnaire.

Epidemiological Studies. Epidemiological studies usually form the first stage in developing a new area of theory and research. This type of study provides baseline information about the natural history of the disease process, and the distribution of the disease or disorder through a population in which we might be considering an intervention (Polgar and Thomas, 1995). This description then provides a benchmark against which we can start to measure the effects of some of our interventions. So fundamental is the role of good epidemiology that the questions addressed in an early epidemiological study are sometimes called the ‘Cardinal Questions’ (Abramson, 1990), reflecting their role in underpinning the majority of the subsequent research process. Good epidemiological data is lacking in many areas of podiatry and more work in this area would be a tremendous asset to the podiatry profession.

Epidemiological surveys may be simply descriptive or they may attempt to provide a deeper, analytical insight into the nature of a problem. The types of epidemiological study can be broadly summarised under one of three headings: descriptive, analytical or longitudinal epidemiological surveys.

(a) Descriptive epidemiological surveys themselves come in two forms, cross-sectional and longitudinal (Abramson, 1990). In cross-sectional studies, a snapshot of a disorder is taken to identify features such as incidence and prevalence. In a longitudinal survey a sample of a population with a disorder is followed over a period of time in order to describe its changing features and natural history.

(b) Analytical epidemiological surveys are used to compare the features of two groups or populations in order to describe features of commonality or difference (Abramson, 1990; Polgar and Thomas, 1995). These are usually cross-sectional or repeated cross-sectional in nature.

(c) Longitudinal epidemiological surveys, whether descriptive or analytical, can be undertaken either prospectively or retrospectively, that is either by identifying a cohort in advance and following it for a period of time, or identifying differentiating features in a group and tracing the history back using available information.

Evaluative clinical research. Logically, evidence-based practice will be most valuable when the research which is intended to inform practice is undertaken specifically with clinical outcomes in mind. The area of evaluative clinical research has grown enormously in the last ten years and includes a variety of approaches from case reports to randomised clinical trials. The majority of evaluative or experimental clinical approaches involve a series of common stages (Polgar and Thomas, 1995). These include:

• Identifying the trend to be explored
• Identifying a population in which to test for an effect
• Selecting an appropriate sample and assigning them to groups
• Administering an intervention
• Measuring the effects of the intervention.

Within this framework there is some flexibility, and there are a variety of designs which can be employed in putting together an experimental study. It is important for the prospective researcher to have utilised the most appropriate design for a given situation. However, it is the reader, not the researcher who holds the key. The knowledgeable reader should carefully appraise the appropriateness of the method chosen and should be prepared to draw their own, informed conclusions. The degree to which they refute, or concur with the researcher’s choice should affect the degree of faith they will place in the results. The natural extension of this is that such informed consumerism will rightly affect the extent to which the results are permitted to influence future practice.

In the previous reprint in this series (JAPMA Vol 92, No 2, February 2002), Keenan and Redmond proposed a hierarchy of evidence adapted from Guyatt et al (1995). The rationale for this proposed hierarchy is developed from weakest to strongest, in the subsequent pages.

Case reports. Case reports can vary from a rudimentary report on the findings with an unusual patient to a prelude to an experimental study. Case reports are usually illustrative, providing an example of a presentation or a management protocol and descriptively outlining the vagaries of the case. They are usually anecdotal rather than experimental in design. Case reports provide an important first step in bringing new ideas or approaches to the attention of the profession, and are a great way to become involved in publishing for the first time.

It should be remembered however, that the role of case reports in informing changes in practice is limited by the lack of formal evaluation involved in the process. The goal of a case report is to encourage recognition of similar experiences not to provide definitive evidence of the effectiveness of a particular management approach. The discerning practitioner should not usually consider changing their practice solely on the information presented in a case report.

Cross-sectional surveys. As previously mentioned, cross-sectional surveys describe a method for gaining a picture of a population at a specified time. Such methods are useful in establishing disease frequency and distribution, and while cause and effect cannot be ascertained using cross-sectional designs they yield important information about the association between exposures and disease (Torrence, 1997). Although they are limited, cross-sectional surveys are generally quickly completed, cost efficient and easy to administer.

Single Subject Experimental Design (SCED). SCED is similar to other scientific research methodologies whereby a series of measures (eg of pain) are taken over time. It too requires the same attention to logical design and control as other methodologies. However, SCED is usually reserved for conditions that are relatively rare or difficult to research. Further, SCED encourages assessment of individual responses to treatment, a point which is generally overlooked in larger trials (Ottenbacher, 1986). In addition, SCED may be used to pilot treatments on a few patients, thus focusing the research question and protocol, before moving on to large, expensive trials.

Usually data is collected by taking measures during different phases: for example, measurement is taken during a baseline phase where no treatment is offered, then treatment is introduced and further measurement made. There are many different routes after these initial phases including returning to the baseline condition or changing to a new treatment to compare the effect of different treatments (Kazdin, 1982). A multiple baseline design can also be used either across subjects or across conditions (Portney and Watkins, 1993). While the data collected is usually graphed for visual analysis, researchers can also use some statistical analysis to supplement the visual inspection methods (Kazdin, 1982; Ottenbacher, 1986).

SCED has been criticised in the past because the results cannot be used to draw conclusions that can be generalised to the population as a whole. Its main purpose as a methodology, however, is to assess the effect(s) of an intervention on an individual. To gain ‘generaliseability’ the researcher must conduct the research on many individuals. However, if many individuals are suitable to be studied, it would be more appropriate to use a different methodology, such as a randomised controlled trial.

Case-control designs. Case-control studies, either retrospective or cross-sectional, compare a sample group with a group known to have been spared the intervention or condition under investigation. This type of design is employed where the sample is either small, as may be the case with rare conditions, or where the sample would not be available for the duration of a full prospective study. Case-control designs are particularly useful for identifying trends when the effects of an intervention may be difficult to predict in advance. They often appear attractive to the novice researcher because they can be cheaper and easier to implement than some other methods. However, this form of study is for a number of reasons methodologically less than ideal. In particular the lack of randomisation leads to the potential for the introduction of biasing factors such as selectivity.

An example of a case-control study would be to retrospectively determine the effect of having worn foot orthoses (for a different condition) on subsequent incidence of arthritis of the knee, or heel spur formation. In such a study, the incidence of arthritis or heel spur in a group of long-term orthoses wearers could be evaluated against the incidence found in a group of non-orthoses wearers. Due to the lack of randomisation, the cases and controls must be carefully matched to ensure that the one variable under investigation is the sole, or at least the primary factor responsible for the difference between the case and its control. The data yielded from case-control studies should be treated with some caution (Abramson, 1990; Bland, 1995; Greenhalgh, 1997).

Cohort studies. Cohort studies can be described as an evaluative study that follows the same group of subjects over time. These are used prospectively to assess the natural history of a disease process. In a cohort study a sample is identified as having a suspected causal factor and then followed over a prolonged period in order to determine whether they go on to develop symptoms or a disease process. This contrasts with most types of design where the study is undertaken on groups of subjects who have already demonstrated the disease process.

Cohort studies are often used to describe the incidence of disorders that may be relatively uncommon in a normal population, and so may involve large numbers of participants. This, coupled with the long duration of most studies can cause problems relating to drop outs and non-compliance. In podiatry a number of cohort studies are currently needed in order to clarify the natural history of disorders such as metatarsus adductus and to provide information about the underlying causative factors for pathologies such as cavoid and planus foot types (Bland, 1995; Greenhalgh, 1997).

Randomised Controlled Trials (RCTs) (after Bland, 1995; Polgar and Thomas, 1995; Spilker, 1996; Greenhalgh, 1997). The underlying premise of the RCT is the employment of randomisation procedures to objectively assign subjects either to active treatment, or to control states of no treatment or a placebo treatment in a prospective study. The randomisation process, if properly conducted, results in the minimisation of bias and makes the RCT a good, robust method for evaluating clinical treatments. As such, a well-executed RCT is generally regarded as being the gold standard for experimental design. Variations on the RCT include multi-centre RCTs, where the data is collated over a number of different locations or institutions to exclude local effects; and the so called ‘mega trials’ where very large numbers of cases (typically many thousands) are evaluated in order to produce high quality data on the effects of a treatment.

Control groups. The utilisation of a control group is probably the single most important factor affecting the ‘believability’ of a study. This is because difference between the active treatment and the control is the real key to how effective a treatment is. If the two groups are appropriately similar in every way (because of a good randomisation protocol) except for the treatment, then there is a good chance that any difference is solely because of the treatment—rather than factors such as resolution over time.

If the control group is made up of participants receiving either no treatment at all, or a disguised dummy (placebo) treatment, then the trial is known as a ‘causal’ or ‘explanatory’ trial. In this kind of trial all of the effect is expected to be due to the active treatment. Sometimes however, usually for ethical reasons, a treatment under investigation will be evaluated against an existing benchmark treatment of known efficacy. These are known as ‘pragmatic trials’. In this case, the key is whether the new treatment performs better or worse than the existing benchmark. In either case it is essential that the allocation to the treatment or control groups should be properly randomised so as to avoid bias.

Note: A common short-cut, involving the comparison of the results of a group following treatment with the results obtained from the same group before treatment does not constitute ‘control’. In a case such as this, at least some of the difference between the two sets of results may have occurred for reasons other than the treatment, eg straightforward disease progress over the time of the study. The results of studies employing this type of method must be treated with caution—they are not of the same standard as a proper RCT.

Randomisation. Randomisation is useful because it removes the potential for bias in allocating the participants to the various treatment or control groups in a study. Another useful effect is that in a large enough sample, factors that could interfere with the data (confounders) should be roughly equally distributed between all of the treatment and control groups. It is not then necessary to eliminate all confounding factors because although they may add ‘noise’ to the data, they are less likely to systematically skew results one way or the other.

Randomisation requires a larger sample than some techniques (such as explicitly matched pairs) because the technique relies on the weight of numbers to ensure the likelihood of the groups being similar in make-up. In the case of randomised studies, the larger the sample, the greater the likelihood that the various confounders will cancel out. Hence randomised studies usually require at least 30 participants in each treatment or control group to ensure a good chance of the groups being relatively homogenous. Studies based on groups of less than 30 should be viewed with some scepticism, while studies based on hundreds or thousands of participants will more usually be considered important. The concepts of sample size and statistical ‘power’ will be developed in the third paper in this series.

Meta-analyses. Meta-analysis is a method of systematically combining or pooling the results from multiple RCTs where methods and the study samples are compatible with one another. The data from a number of trials may be pooled in raw form to create a large and more powerful data set, or the previously interpreted results from the trials may be averaged. The resulting information thus represents a synthesis of the output of a number of independent RCTs, and can enhance the credibility of the overall analysis. In addition to the data presentation, the author of a meta-analysis usually makes some attempt to summarise the features of the individual studies included in the analysis. This often aids the inexperienced reader in understanding the process of critically evaluating the research (Spilker, 1996; Greenhalgh, 1997).

The meta-analyst will also attempt to present data from different studies in a consistent manner. Again this can help the inexperienced reader in identifying what information is considered most relevant in the often confusing mass of data presented in scientific papers. There are however, some disadvantages to using meta-analysis, these include: non-comparable or poorly comparable data in the individual studies; publication and selection bias in the identification of the component articles; data quality; and methodological flaws in the component studies.

Systematic reviews (after Droogan and Song, 1996; Greenhalgh, 1997; Hunt and McKibbon, 1997). Prior to the advent of techniques such as the systematic review, summaries and reviews (now called ‘illustrative’ reviews) could often be rather hit and miss. The literature was often incompletely reviewed and where papers were discussed, this could sometimes occur in a somewhat journalistic manner, for example describing the overall features of an area of research, rather than evaluating the research against specific criteria. Of particular concern is the tendency toward so called ‘exclusion bias’, where the reviewer overemphasises papers supporting his or her ‘story’ and de-emphasises difficult or contradictory literature.

The techniques employed in systematic reviews have been developed in order to address these criticisms. Systematic reviews are summary reports based on a rigorous and predetermined methodological approach to identifying and reviewing all relevant literature in an area. They are based on an explicit statement of objectives, materials, methods, sources and resources as criteria against which the credibility of a paper or series of papers can be judged. Thus the role of the systematic reviewer is to identify the best available evidence and to present the information in such a way as to inform the practitioner of the current state of the art. This is an important role, because the substantial recent increase in publication in the medical literature—running at more than two million biomedical articles per year, according to Droogan and Song (1996)—and the move towards evidence-based health care have led to huge demands on practitioners to keep up with recent developments.

Where there is an abundance of good trial data, as is the case in many branches of medicine, the convention is to include in a systematic review, only information from good quality RCTs—with almost all other types of study being excluded. This approach is seen by some as perhaps being too hard-line, particularly where RCTs may not be the most appropriate method for an area of evaluation. In the case of a profession like podiatry where there are painfully few RCTs in any area, excluding all non-RCTs from a review could be seen as ‘throwing the baby out with the bath water’. In this case it is usual to specify lesser, but equally strict criteria as to what ‘evidence’ will be included in the review. The credibility of the particular method is then used to assign a weighting to the applicability of the information therein.

Systematic reviews are complicated and resource intensive to produce and are the subject of a series of methodological developments in their own right. One singular advantage of systematic reviews is that they are usually pathology or symptom based, and so offer tremendous potential for podiatry, a profession where there are areas in which we have a great deal more to offer than most other disciplines may wish to believe.

Issues Underpinning All Methods of Clinical Evaluation

Accurate Quantification Needs Accurate Measurements

One of the most pressing issues for clinicians is that of the need for appropriate and relevant clinical assessments. This is especially true in the quantitative evaluation of practice, either in basic self-audit, or in clinical research. If we are to rely on measures of any kind in our practice or in our research then we should expect that the measures meet certain criteria relating to the accuracy, reliability and validity of the information that they yield. Measures that are not accurate or reliable or valid must be a questionable basis for quantification of results. In clinical practice, a vast range of different measurements may be used to inform our decision-making. Where these measures are derived from the substantial body of mainstream medicine and health care, these are often relatively well validated. Where the measures are derived from smaller or less research-mature disciplines such as podiatry, problems with validation and applicability may arise. As we are probably all too aware, several measurements which are frequently used by podiatrists in the area of clinical biomechanics have been recently challenged (Elveru et al, 1988; Freeman, 1990; Menz, 1995; Menz and Keenan, 1997). This has caused considerable debate within the profession, concerned at such profound undermining of the founding principles of clinical biomechanics. Uncomfortable though it may be however, it is vital to acknowledge that if truly informed decisions are to be made on the basis of clinical measurements, then those measures must first be demonstrated to be accurate, reliable, and valid. The following section further examines the concepts of reliability and validity in attempt to outline the main problems.

Reliability

Very few measurements are perfectly reliable and as such, there will be some error associated with all measurements. However, it is important if we are using these instruments in clinical practice that we understand where the error may be coming from. The most common classification for errors in measurement refers to systematic and random error (Portney and Watkins, 1993).

Systematic errors are predictable errors of measurement, where a technique consistently over or under estimates the true values. If these are predicable enough they can usually be accounted for in a mathematical adjustment. Random errors, however, are not as predictable and are due to chance and inherent variability such as the slight variations in position of the measuring instrument, the line of sight, and environmental factors. Random error affects measurement in an unpredictable manner and it is quantification of this error that forms the basis of most reliability studies.

Random error leading to concerns over the reliability of a measurement technique may arise from either the testing procedures or from the testers themselves. Inter-tester reliability indicates the agreement or consistency between repeated measurements of the same variable, as obtained by two or more testers. It is important to establish whether the inter-tester reliability is adequate as it will dictate the degree to which different practitioners will be talking about the same thing when discussing a measure.

Intra-tester reliability refers to the degree of agreement or consistency, between repeated measures of the same variable by one clinician. Clearly, if intra-tester reliability is poor (one person cannot repeat the measure accurately), the inter-tester reliability (the degree to which that person will agree with someone else) is likely to be even worse. The discerning reader should check that the reliability of measures employed in a study is reported and that the degree of reliability appears appropriate for the purposes of that particular measure.

Validity

Validity is the extent to which an instrument measures what it is intended to measure (Abramson, 1990; Portney and Watkins, 1993). For a measurement to be valid it must already have been demonstrated to be reliable, ie reliability is part of the overall validity, but only one part. There are several aspects to the validity of a measure, these are presented in Table 4. Evaluating the validity of the measures to be employed in a study can be difficult and time-consuming and is therefore often short-cut by the enthusiastic researcher who wants to get down to the business of investigating treatment effects. Discerning readers should carefully and critically assess for themselves the extent to which a researcher has employed well-validated techniques in any clinical study. Where the researcher cannot point to the validation of the measure in a previous paper, the results of their own attempts should be considered a minimum requirement. One area in which unvalidated or under-validated measures are often employed is in the use of surveys or questionnaires. As mentioned previously, questionnaire and survey design is a science in its own right for good reasons. Beware of invalidated measures that do not meet the requirements of such design.

Conclusion

The single greatest step in becoming a discerning consumer of research is the development of the skills required to critically evaluate what is presented in journals and to decide whether to allow it to inform or alter one’s future practice. The fact that a paper has been published in a journal does not vouch entirely for its worth, assessment of that worth is entirely (and quite rightly) up to the well informed clinician. For the decision to be made from an informed perspective, the clinician needs at least a basic understanding of the research process and principles, and this understanding comes only from learning and application. In this paper we have outlined some of the basics but the reader is encouraged to make every effort to upgrade this area of their skills in much the same way as they would maintain and seek to continually upgrade their clinical skills. We would not, for a moment, advocate that all clinicians should be active researchers but we do suggest that all professionals have some responsibility to their patients to be discerning consumers of research output. With its apparently rigid, complex rules and semantics, the world of research can be alien and intimidating at first. However, with a little courage and application, all that this world has to offer can be opened up by anyone with an interest. The result of taking such time and trouble is a better equipped clinician with a healthy ability to make better informed treatment choices for and with their patients.

Figure 1. Factors that enter into clinical decisions, adapted from Mulrow et al (1997), pg 389.

Figure 1. Factors that enter into clinical decisions, adapted from Mulrow et al (1997), pg 389.

Figure 2. Factors that enter into decisions for research approaches.

Figure 2. Factors that enter into decisions for research approaches.

Table 1. General assumptions of qualitative and quantitative research (based on Bailey, 1977).

Table 1. General assumptions of qualitative and quantitative research (based on Bailey, 1977).

Table 2. Criteria for quality qualitative research (adapted from Higgs and Adams, 1997).

Table 2. Criteria for quality qualitative research (adapted from Higgs and Adams, 1997).

Table 3. Philosophies of approaches to knowledge generation for qualitative research (based on Higgs, 1997).

Table 3. Philosophies of approaches to knowledge generation for qualitative research (based on Higgs, 1997).

Table 4. Types of measurement validity (adapted from Portney and Watkins, 1993).

Table 4. Types of measurement validity (adapted from Portney and Watkins, 1993).