This in an excerpt from my 2026 book Efficacious Educational Technology. It is available under a Creative Commons license. My rationale if that theories are rules we should follow when designing educational technology systems.
The “gold standard” of research used to develop theories is the double-blind random experiment. When researchers conduct experiments, they review existing theory and define a gap in the theory. This gap is clarified in a research question and a method for answering it is approved. Experiments find a sample of the population being randomly selected, and the individuals in the sample are divided into the treatment group and the control group. The research method comprises a strategy for ethically gathering and analyzing the data, including the statistical methods used to address the research question. The double-blind nature means the individual conducting the analysis is unaware of which group was the treatment group and which was the control group.
When we conduct experiments, we collect quantitative data, which is analyzed with statistics to test hypotheses. While experiments can be conducted in some educational settings, in many cases, educational researchers seek other sources of quantitative data to answer their research questions. They also use descriptive statistics to document trends and to describe populations. In addition to collecting data with surveys or questionnaires, qualitative data can also be collected with observations. For example, trained observers can record how many events of interest occur in a classroom, and with digital data and cloud computing used in so many educational settings, data collected through those systems can be studied as well.
Educational researchers also collect qualitative data to answer interesting questions in educational technology. Qualitative data are recorded as text and can include individual or focus group surveys; they can also emerge from document analysis such as meeting notes or recording transcripts. Whereas quantitative data are analyzed using descriptive or inferential statistics, qualitative data are usually read and coded by multiple readers. These codes are used to either describe and detail the themes the researchers have agreed are relevant, or the coded data are used to write profiles of relevant users.
Once researchers have asked a question, developed a hypothesis, gathered and analyzed the data, they submit it for publication. Other researchers review the article, and once their concerns have been resolved, it is accepted for publication. Before the theory becomes established, it is replicated in similar and different settings. After more studies have validated the theory, it may become established. This is when practitioners should begin using it.
It is important for practitioners to wait until a theory is established before they begin to rely on it to guide decisions. A common phenomenon in recent decades has been the replication problem. New “discoveries” are made in research; in many cases the findings are new and may get attention because they are novel. The new findings, however, are not replicated. There may be a problem with data collection or analysis; the findings may have been the result of error.