CITEULIKE: 1268816 | REFERENCE: BibTex, Endnote, RefMan | PDF

Bartneck, C. (2007). Quality Criteria for Design and Science. Proceedings of the CHI 2007 Workshop: Exploring Design as a Research Activity, San Jose.

Quality Criteria for Design and Science

Christoph Bartneck

Department of Industrial Design
Eindhoven University of Technology
Den Dolech 2, 5600MB Eindhoven, NL


Abstract - This paper discusses the quality criteria that are used in design and science to evaluate the value of the produced knowledge and artifacts. The scientific criteria discussed are: generizability, falsifiability, truth, novelty, parsimony, precision, accuracy and efficiency. Their design counterparts are also discussed. This comparison may help to guide the design method into a more scientific direction. If design is to become a useful research method, then its resulting knowledge must achieve the same or better quality than the traditional scientific method.

Keywords: design, science, quality, criteria


If design is to become a scientific method then its results must be measured by the quality criteria for scientific knowledge. If the resulting knowledge is of the same or even better quality compared to the knowledge resulting from the traditional scientific method, then the application of the design method can be justified. A fair discussion of the scientific method is available (Chalmers, 1999).

Design methods are not yet optimized for the creation of scientific knowledge and therefore generally produce knowledge that is of lesser scientific quality. Instead, the design methods are optimized to create artifacts. It appears worthwhile to compare the quality criteria used for scientific knowledge and designed artifacts. It may provide insights into what direction the design method has to evolve.

Qualilty Criteria

The generizability of scientific knowledge is one of the most important criteria. It describes the degree to which general statements can be derived from a particular statement. The more general statements can be derived the better the particular statement. Newton's law of gravity was not only able to describe the behavior of Newton's inspirational falling apple but also all other apples, fruits, organic material and inorganic material. Even the motion of the stars could be described by it. His law is therefore of high value. If a statement, on the other hand, depends on the researcher him/herself then its generizability is low. If I state the "bugs are awful" then this may only hold true for people that share my paranoia about small creatures with many legs. Objectivity is therefore a good method to increase the generizability of a statement.

Designers know a similar concept: universality. It describes the degree to which general problems can be solved by a particular solution. The more universal a solution is the better. A hammer, for example, is more universal than a pair of horseshoe pliers and hence more valuable.

Falsifiability is another important criterion that is known to both, scientists and designers. Originally proposed by Karl Popper (2002), falsifiability describes the property of statements that they must admit of logical and empirical counterexamples. The latter refers to the condition that it must be possible, at least in principle, to make an observation that would show the statement to be wrong, even if that observation is not actually made. The statement "all swans are white" is in principle falsifiable by observing a black swan. The higher the number of logical and empirical counterexamples a statement withstood the higher its value.

The usage of falsifiability in design is very similar. A solution must admit of logical and empirical counterexamples. If a certain device, for example, is intended to continuously increase one's karma then its function is impossible to falsify. Such a device could not be considered a design. Falsifiability is plays a less importance role in design in comparison to science, since it often deals with concrete and well-defined problems. The effects of a solution are usually easy to observe.

Truth is a key criterion in science and it also plays and important role in design. However, multiple definitions of truth exist. The Wikipedia lists the following eight theories of truth: correspondence, coherence, constructivist, consensus, pragmatic, performative, semantic and Kripke's theory. The correspondence and coherence theories are probably the most acknowledged and hence this study focuses on them. In the coherence theory, truth is primarily a property of whole system of statements. The truth of a single statement can only be derived from its accordance with all the other statements. If a new statement contradicts an existing statement then both statements need to be reconsidered. In the pervious used examples of swans, one of the statements must be false. Either not all swans are white or the particular swan is not black. The similar concept in design is known as compatibility. If a new component is introduced to an existing system then it should not prevent any existing component form operating correctly. The installation of new software on a computer can lead to such incompatibilities in which previous functions cease to function.

The correspondence theory of truth deals with the relationship between statements and reality. If theories correspond to observations in reality then they are considered to be true. This direction in the relationship between truth and reality is usually attributed to science. The other direction can be attributed to design. If an artifact corresponds to theory then it is considered true. Our understanding of the physical world makes it difficult to invent artifacts that could not fully be explained by existing theories of physics. Many attempts have been made to invent a perpetuum mobile and even patents have been filed, but no working model has been build. By now, the United States Patent and Trademark Office (USPTO) has made an official policy of refusing to grant patents for perpetual motion machines without a working model:

With the exception of cases involving perpetual motion, a model is not ordinarily required by the Office to demonstrate the operability of a device. - 608.03 Models, Exhibits, Specimens [R-3]

However, often solutions have been used without full theoretical understanding. The Bayer Company patented aspirin already in 1899 and successfully marketed it ever since. Its pain relieving effect was only understood in 1971 by John Robert Vane, who received the Nobel Prize in Physiology or Medicine in 1982 for his discovery.

Another important quality criterion for scientific knowledge is novelty. Rediscovering Newton's laws has little value. But newness in itself is not sufficient. A novel scientific theory does not only need to be different from existing theories, it also has to explain more than existing theories. Galileo's theories extended Aristotle's, Newton's law extended Galileo's and Einstein's extended Newton's. The same principle is known in design as innovation. Novelty, in its pure newness definition, is even a requirement for patents. Moreover artifacts are not only expected to work differently, but also better. Modern PCs are currently powerful enough to even completely simulate older computers, such as the Comodore 64, using the VICE emulator. Modern PCs can do everything that older ones can, and more.

The criterion of parsimony, also known as Occam's razor, is the preference for the least complex statement to explain a fact. A good example can be found in the field of Astronomy. The Copernican model is said to have been chosen over the Ptolemaic due to its greater simplicity. The Ptolemaic model, in order to explain the apparent retrograde motion of Mercury relative to Venus, posited the existence of epicycles within the orbit of Mercury. The Copernican model (as expanded by Kepler) was able to account for this motion by displacing the Earth from the center of the solar system and replacing it with the sun as the orbital focus of planetary motions while simultaneously replacing the circular orbits of the Ptolemaic model with elliptical ones. In addition the Copernican model excluded any mention of the crystalline spheres that the planets were thought to be embedded in according the Ptolemaic model. In a single stroke the Copernican model reduced by a factor of two the ontology of Astronomy.

In design, simplicity plays a similar role. Simplicity is the preference for the least complex solution to achieve a given goal. Just 20 years ago, a complete photochemical process was necessary to print a photo, which involved various toxic chemicals and sophisticated machines. These days, everybody can print his own pictures with standard inkjet printers.

Lastly, the scientific criteria of precision, accuracy and efficiency are discussed, including their counterparts in design: reliability, effectiveness and efficiency. Precision refers to the degree to which a statement or theory predicts the exact same facts while accuracy refers to the degree to which a statement or theory predicts the facts it is intended to predict. The analogy of bullets shot at a target is useful to explain the difference between these two related concepts and at the same time show the similarity between design and science criteria. In design, the concepts are known as reliability and effectiveness.

In this analogy, a gun firing at a target (design) parallels a theory predicting observations (science). The effectiveness of the gun describes the closeness of the bullets to the center of the target (see Figure 1 left). Bullets that strike closer to the center are considered more effective. In parallel, the closer the observations occur compared to the theories prediction, the more accurate the theory is.

Figure 1

Figure 1: high effectiveness but low reliability (left), high reliability but low effectiveness (middle) and high reliability and high effectiveness (right).

To continue the analogy, the reliability of the gun refers to the spread of the bullets. The closer the bullets strike to each other the higher the reliability (see Figure 1 middle). In science, the closer the observations are to each other, the more precise the theory is. The bullets do not necessarily need to be close to the center for this. The bullets (or observations) can be reliable (precise) without being effective (accurate). However, for bullets (and observations) to be perfectly effective (accurate), they also need to be reliable (precise) (see Figure 1 right).
Efficiency then refers to the resources expended in relation to the precision and accuracy of the observations predicted, in case of science, and to resources expended in relation to the effectiveness and reliability of goals achieved.


This comparison of quality criteria used in design and science illustrates that the often perceived gap between them does not exist in principle. Pirsig (1995) attested, they are just two different complementary ways of looking at the same thing. At the most immediate level, they have never been separated. Both disciplines are creative; designers primarily create artifacts and scientists primarily knowledge. This similarity may mislead people to believe that design is a science. Design traditionally focuses on the creation of artifacts, not knowledge. One possible way to improve the generizability of the knowledge produced by the design method is to make the method objective. An objective design method would make the resulting artifacts and knowledge independent of the designers involved. An example of such an objective method is the rational problem method that heavily relies on mathematics for decision making (Alexander, 1964; Simon, 1996; Vincenti, 1990). Such a method could, as Pitt claimed (2001), lead to knowledge that is far more reliable, secure and trustworthy than scientific knowledge.


Alexander, C. (1964). Notes on the synthesis of form. Cambridge,: Harvard University Press. | view at

Chalmers, A. F. (1999). What is this thing called science? (3rd ed.). Indianapolis: Hackett. | view at

Pirsig, R. M. (1995). Subjects, Objects, Data & Values. Proceedings of the Einstein meets Magritte, Vrije University Brussels.

Pitt, J., C. (2001). What Engineers Know. Techne, 5(3), 17-30.

Popper, K. R. (2002). The logic of scientific discovery. London ; New York: Routledge. | view at

Simon, H. A. (1996). The sciences of the artificial (3rd ed.). Cambridge, Mass.: MIT Press. | view at

Vincenti, W. G. (1990). What engineers know and how they know it : analytical studies from aeronautical history. Baltimore: Johns Hopkins University Press. | view at

This is the print version | last updated February 5, 2008 | All Publications