The validity of an experiment is directly affected by its construction and execution, hence attention to experimental design is crucial. The process of experimental design involves many elements, and often in a reiterative fashion. It can be described in terms of a framework, inductive space, system, set of experiments, and model building (see Glass, below). Implicit to these elements are controls, whose purpose is to negate error and bias. Positive and negative controls come up in virtually every discussion of an experimental protocol, but one must consider as well controls that pertain to the system, assumptions, and experimentalist. Statistics plays an enormous role is both design and interpretation, as so much of the logic used in asking experimental questions is inductive, i.e. relating to predictability and generalizability.
Experimental design uses critical thinking to address a specific scientific question or hypothesis. From the standpoint of a competency, it is in essence a form of problem-solving that focuses one’s talents and skills on pushing the envelope of knowledge. Its exact formulation differs depending on the questions being asked, so that it evolves with time and experience. The question or the tools used may be quite specific, but the skill is generalizable.
The following concepts are relevant to experimental design.
Premise. ‘Premise’ in the context of science is defined as the body of research, both what it is and where it leaves off, that provides the basis for the research question. It is not formally a component of experimental design, but it sets the stage. It is, as a competency, the ability to consider the general strengths and weaknesses of prior research before proposing additional research. It encompasses the rigor of previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources.
System. David Glass in Experimental Design for Biologists begins to frame a working experimental design with the development of a ‘system’ that can enable the appropriate experiments to be performed. The system is very much akin to a “Materials and Methods” section of a paper and in many ways represents the resources and methodology one has at her or his disposal to answer the question posed. The system is very much rooted in concepts of validation, which are discussed below, as they relate to resources and methodology. The system is not a competency, of course, but its management is. Management is covered in a separate section of this website. It is important to point out here, however, that the focus of the experimental question and hence system integrates the separate skills of management at both logistical and strategic levels.
Statistics. Statistics is addressed as a competency in a separate section of the website. Statistical concepts most relevant to experimental design are randomization, replication, and blocking. Randomization provides a mechanism for assigning treatments to experimental units that is free from intentional or unintentional biases; replication allows us to assess the degree of existing variation in a response variable among experimental units treated alike; and blocking refers to grouping similar experimental units together and assigning the treatments of interest to the experimental units within such groups (Nettleton, below). Sample size and variation are important determinants of the power of a statistical test.
Controls. A control is defined as a person, group, event, etc., used as a constant and unchanging standard of comparison in experimentation (Dictionary.com). There are many types of controls, for example negative, positive, and assumption controls (see Glass for detailed discussions of these and others). Their deployment is a skill that goes well beyond knowing how they are defined, extending for example teasing out potentially obfuscating variables and how precisely to deal them. The adept use of controls is an art form.
Relevant biological variables. As noted, controls are performed to account for variables. But this presumes one is aware of all the variables that are relevant. NIH is currently concerned in animal and human studies with sex as a biological variable, but it suggests that others need to be considered as well, for example genetic background, age, body mass index, room temperature and lighting. Consideration of these and other variables from the standpoint of NIH enables “reviewers and the scientific community to assess the internal validity of a study – whether the findings hold up after accounting for confounding and selection biases – and the external validity a study – whether the findings, even if internally valid, apply to the ‘real world’” (Mike Lauer, Consideration of Relevant Biological Variables in NIH Grant Applications).
Reagent, methods, and equipment validation. Science is exacting – nothing can be taken for granted. This includes reagents, methods, and equipment. Examples of where projects can go seriously wrong at the level of reagents include misidentification of cell lines, taking at face-value the properties of antibodies, and assuming the genetic backgrounds of knock-out mice. The validity of methods is generally assured through the use of proper controls, but it’s the rare lab in which favored protocols haven’t mutated slowly with time. The need to review and (re)validate such protocols is important. Equipment performance is not necessarily stable with time, hence the need to recalibrate micropipettes, cell incubator CO2 tensions, water bath temperature settings, etc. At one level, having a firm understanding of reagents, protocols, and equipment provides for a productive set of experiments and efficiency in troubleshooting. At another and more important level, it is central to assuring an outcome that is both valid and reproducible.
Robust and unbiased results. NIH expects anyone submitting a research or training proposal to describe in the experimental design how he or she will achieve robust and unbiased results. Robust results are those that can be reproduced under well-controlled and reported experimental conditions. Key elements include appropriate statistical design, control, consideration of relevant biological variables and authentication of key biological or chemical resources, all of which are discussed above.
Resources
Experimental Design for Biologists, David J. Glass, 2nd Edition. This book covers the philosophy of experimentation, system validation, experimental constructs, and model-building in a highly readable and contemporaneous fashion.
At the Bench: A Laboratory Navigator, Kathy Barker, 2005. This book, which is provided to all BGS students upon matriculation, contains a chapter on how to set up and experiment and a large number of chapters on experimental techniques.
Nettleton, D. A Discussion of Statistical Methods for Design and Analysis of Microarray Experiments for Plant Scientists. The Plant Cell 18: 2112-2121, 2006.
NIH’s website on rigor and reproducibility. https://www.nih.gov/research-training/rigor-reproducibility
‘Experimental design and transparency’ represents a component of BGS required training. You can view coverage of the topic in a module that provides background, policies, and case studies at the ‘Required Training’ website of BGS (https://www.med.upenn.edu/bgs-rcr-exdes/).