An Overview of Quantitative Research in Compostion & TESOL
Quantitative Analysis, like many other experimental type analyses were once considered out of the domain of disciplines such as Rhetoric and Linguistics. In fact, it was common for such analysis to be prevalent in the scientific fields and be considered useful only in scientific research. Today, Quantitative Analysis is probably as ubiquitous in language studies as it has been in the scientific field. Specifically, knowledge regarding second language acquisition, or human overall language development, for example, requires much more than qualitative, subjective investigation of data. [And] As the field of English and TESOL become more extensive globally, a demand for qualitative language research seem to be crucial. Language practitioners seek an all-powerful remedy for the plethora of problems and puzzles they face regarding language learning, acquisition, and use. Studies qualitatively conducted seem to be of major interest these days.
Practitioners of language development aren’t the only ones urging for more quantitative analyses in language studies. A remarkable amount of linguistics and composition researchers are pushing for such studies, claiming that quantitative studies will solidify the field of linguistics (language study) and give it more credence in the scientific emporium. Grant Henning in "Quantitative Methods In Language Acquisition Research" urges researchers to use quantitative methods in language acquisition research so that we’d have a valid basis for making it belong to the realm of scientific inquiry. A general overview of the fundamental characteristics of quantitative research will enable readers to judge for themselves.
I. Some Definitions of Quantitative Research
"the kind of research that involves the tallying, manipulation, or systematic aggregation of quantities of data." Grant Henning "Quantitative Methods In Language Acquisition Research"
"an inquiry into a social or human problem based on testing a theory composed of variables, measured with numbers, and analyzed with statistical procedures, in order to determine whether the predictive generalizations of the theory hold true." John W. Creswell Research Design: Qualitative and Quantitative Approaches
"the collection of numerical data in order to explain, predict and/or control phenomena of interest." L.R. Gay Educational Research: Competencies for Analysis and Application
Of the three definitions here Creswell’s definition appears to be more comprehensive than both Henning’s and Gay’s. Nevertheless, a common proposition carried throughout all of these definitions is that Quantitative research is indeed based on much of what it’s nomenclature indicates: quantities. For the sake of this overview, I will provide a summary of Creswell’s discussion of Quantitative Research. In Research Design: Qualitative and Quantitative Approaches, Creswell provides novice researchers with some useful information regarding conducting qualitative research. Topics ranging from choosing a research paradigm to formulating questions are covered.
According to Creswell, Quantitative methods consist of two types: experiments and Surveys. "Experiments include true experiments with the random assignment of subjects to treatment conditions and quasi experiments that use nonrandomized designs." [And] Surveys include cross-sectional and longitudinal studies using questionnaires or structured interviews for data collection with the intent for generalizing from a sample to a population." Creswell provides the following reasons for selecting a quantitative paradigm:
II. Reasons for Selecting a Quantitative Paradigm
Researcher’s World View
A researcher’s comfort with the ontological, epistemological, axiological, rhetorical, and methodological assumptions of the quantitative paradigm
Training And Experience Of The Researcher
Technical writing skills; computer statistical skills; library skills
Researcher’s Psychological Attributes
Comfort with rules and guidelines for conducting research; low tolerance for ambiguity; time for a study of short duration
Nature Of The Problem
Previously studied by other researchers so that body of literature exists; known variables; existing theories
Audience For The Study
Individuals accustomed to/supportive of quantitative studies
Creswell provides a format for a quantitative study, stating that "it conforms to standards easily identified in journal articles and research studies. The form generally follows the model for an introduction, a literature review, methods, results, and discussion. In planning a quantitative study and designing a dissertation proposal, consider the following three-part format to sketch the overall plan:"
III. Example 1. Quantitative Format
—"A quantitative introduction typically is written from the third person point of view. This impersonal view removes the writer from the picture and helps create a sense of objectivity and distance between the researcher and that being researched. (a quantitative epistemological stance) The following are areas researchers should cover in the introduction of their study
Context (statement of the Problem)
Purpose of the Study
Research Questions or Objectives or Hypothesis
Definition of Terms
Delimitations and Limitations of the Study
Review of the Literature—According to Creswell, Literature in a research study accomplishes several purposes. (a) it shares with the reader the result of other studies that are closely related to the study being reported. (b) it relates a study to the larger, ongoing dialogue in the literature about a topic, filling in gaps and extending prior studies. (c) it provides a framework for establishing the importance of the study, as well as a benchmark for comparing the results of a study with other findings. In a Quantitative study the literature is used to "provide direction for the research questions or hypotheses…it is often used to introduce a problem in the introduction….it is used deductively as a framework for the research questions or hypotheses."
Methods—In designing a quantitative method for a research study, Creswell introduces some significant steps for novice researchers who are using either an experimental research plan or a survey research plan. The following are areas researchers should cover in the Methods section of their study.
Sample, Population, or Subjects
Instrumentation and materials
Variables in the Study
Components of a survey Method Plan
Survey Design—"a survey design provides a quantitative or numeric description of some fraction of the population---the sample—through the data collection process of asking questions of people."
A checklist of questions for designing a survey method
The survey design
review the purpose of a survey and the rationale for its selection as a design in the proposed study
discuss the purpose of survey research.
specify the characteristics of the population, and the sampling procedure
describe the population in the study
identify whether the sampling design for this population is single stage or multistage
identify how individuals will be selected
discuss whether this randomly selected population will be stratified so that specific characteristics are represented in the sample and the sample reflects the true characteristics of the population
identify the characteristics used in stratifying the randomly selected population
indicate the procedure for selecting the random sample from lists or the sampling frame
indicate the number of people in the sample and how this number was determined.
identify the survey instrument to be used in the study
If one plans to use an existing instrument, describe the established validity and reliability of items and scales on the instrument
include sample items so that readers can see the actual items used
indicate the major content sections in the instrument, such as the cover letter
discuss plans for pilot testing or field testing the survey ad provide a rationale for this procedure
for mailed survey, identify steps to be taken in administering and following up the survey to obtain a high response rate.
Variables in the Study
Plan to include a table and discussion that cross-references the variables, the questions or hypotheses, and specific survey items.
Recommend that data analysis be presented in a series of steps
Components of an Experimental Plan
The components of an experimental method discussion follows a standard form; subjects, materials, procedures, and measures.
Describe whether the subjects will be selected randomly or conveniently
Indicate whether the subjects will be assigned randomly to groups in the experiments
Indicate whether the subjects will be matched, as well as randomly assigned to treatment groups
Tell the reader about the number of subjects in each group and the procedure for determining the size of each group.
Identify the independent variables, called treatment conditions or factors in an experiment
Identify the dependent variable or variables to be used in the study
Instrumentation and Materials
Describe the instrument or instruments to be used in the experiment
Discuss thoroughly the materials used to create the treatment conditions
Discuss the overall type of design
Other distinctions can be made about the overall experimental design in terms of the specific use of independent variables
Provide a diagram or a figure to illustrate the specific research design to be used
Discuss the step-by-step approach for the procedure in the experiment
Describe the descriptive statistics calculated for observations and measures at the pretest or posttest stages of experimental designs.
Describe the inferential statistics used to test the hypotheses in the study.
For single-subject research designs, use line graphs for baseline and treatment observations for abscissa units of time and the ordinate target behavior.
III. Appendices: Instruments
Gay provides a similarly comprehensive review of some of the methods involved in Quantitative research; which stating them will be mere redundancy. Gay explains that "experimental research is the only type of research that can truly test hypothesis concerning cause-and-effect relationships. It represents the most valid approach to the solution of educational problems, both practical and theoretical….experimental research is both the most demanding and the most productive type of research…" However, like Henning, Gay discussed what he called "some threats." Although Gay’s work mostly focuses on educational research, I believe that it may benefit composition and linguistics researchers to heed those threats.
Regarding the use of quantitative research in language acquisition research, Henning claims "the second most common flaw, is the tendency for quantitative research studies in applied linguistics to be labeled experiments when they are not." Gay’s work somewhat explains how to avoid some of the potential problems that Henning speaks of. Gay claims that "any uncontrolled extraneous variables affecting performance on the dependent variable are threats to the validity of an experiment." Moreover, Gay looks at some important topics that researchers should heed to ascertain the validity of their research.
Threats to Internal Validity
History—refers to the occurrence of any event that is not part of the experimental treatment but which may affect performance on the dependent variable.
Maturation—refers to physical or mental changes that may occur within the subjects over a period of time. These changes may affect the subjects’ performance on the measure of the dependent variable.
Testing—refers to improved scores on a posttest resulting from subjects having taken a pretest.
Instrumentation—refers to unreliability, or lack of consistency, in measuring instruments which may result in invalid assessment of performance.
Statistical Regression—occurs when subjects are selected on the basis of their extreme scores and refers to the tendency of subjects who score highest on a pretest to score lower on a posttest, and of subjects who score lowest on a pretest to score higher on a posttest.
Differential Selection of Subjects—occurs when already formed groups are used and refers to the fact that the groups may be different before the study even begins, and this initial difference may at least partially account for posttest differences.
Mortality—is more likely to occur in longer studies and refers to the fact that subjects who drop out of a study may share a characteristic such that their absence has a significant effect on the results of the study.
Selection-Maturation Interaction, etc.—the "etc." means that selection may also interact with factors such as history and testing, although selection-maturation interaction is more common. What this means is that if already formed groups are used, one group may profit more (or less) from a treatment or have an initial advantage (or disadvantage) because of maturation, history, or testing factors.
Threats to External Validity
Threats affecting "to whom," to what persons, results can be generalized, are referred to as problems of population validity.
Threats affecting "to what" to what environments (setting, dependent variables, and so forth) results can be generalized, are referred to as problems of ecological validity.
Main External Validity threats
pretest-treatment interaction—subjects respond or react differently to a treatment because they have been pretested.
Multiple-Treatment Interference—the same subject receive more than one treatment---or when subjects who have already participated in the study are selected for inclusion in another, theoretically unrelated study.
Selection-Treatment Interaction—interaction of personological variables and treatment effects
Specificity of Variables—interaction of history and treatment effects; interaction of time of measurement and treatment effects.
Experimenter Effects—experimenter personal-attributes effect; experimenter bias effect
Reactive Arrangements—Hawthorne effect, John Henry effect, Placebo effect, Novelty effect.
Regarding Rhetoric and Linguistics study, Henning suggests the following research paradigms to help researchers avoid some mistakes easily made in quantitative studies. He suggests Spearman’s Rho, "in correlational studies in which randomization is not possible and it is not known whether there was a normal distribution in the population from which the sample was drawn; Chi Square Methodology to test independence or goodness of fit with small samples; Path Analytic methods to help language acquisition researchers to test competing theories; Latent Trait model to investigate parameters of reading; Multiple T tests to determine which of an array of linguistic phenomena elicited were most effective in differentiating between two or more groups of individuals;
Factor analysis for aggregating data into mutually exclusive or correlated variables called factors.
Henning, Grant. "Quantitative Methods in Language Acquisition Research" TESOL Quarterly, Vol. 20, No. 4 December 1980; p 701-707.
Creswell, John W. (1994) Research Design: Qualitative and Quantitative Approaches; Sage Publications; California.
Gay, L.R. (1996) Educational Research; Competencies for Analysis and Application; 5th ed.; Prentice Hall; New Jersey. Pp. 390-400.