Title: How to Judge Scientific Research Articles Article by: H. P. P. (Hennie) Lötter HOW TO JUDGE SCIENTIFIC RESEARCH ARTICLES Abstract How should scientists judge the quality of research articles? In this article I present general criteria for judging the scientific value of a research report submitted for publication. These criteria can improve the quality of research articles and produce fair referee reports that are scientifically justifiable. My view is based on four fundamental rules that guide all good science. These rules ought to determine whether scientific research reports merit publication in scientific journals. The rules for good science also structure this article. They are as follows. [1] Scientists must use specialised problem-solving; [2] scientists must refer to previous work; [3] scientists must justify their findings; and [4] scientists must convince their scientific community. HOW TO JUDGE SCIENTIFIC RESEARCH ARTICLES (Word count 4497) Most academics are asked to act as referees for scientific journals. That means they have to judge whether articles submitted for publication are good enough to be published in the journal concerned. The criteria used for judging the merit of a scientific research report submitted for publication are not always explicitly articulated. As a result academics are often bad referees who do not judge other people's research according to appropriate criteria. A philosopher recently described some referee reports that he has seen as not having the aim "merely to point out the faults of the work under consideration, but rather to wound the author" (Garcia 1999: 155). Having had access to many referee reports from non-philosophers as well, Garcia thinks philosophers are some of the "nastiest, if not the nastiest, referees of all" (Garcia 1999: 155). Is there a way of finding a broad consensus on general criteria that can be used to judge the scientific value of a research report submitted for publication? Can such shared criteria improve the quality of research articles and lead to referee reports that are less nasty and do not intend to wound? In this article I want to propose a broad framework of ideas that can be used for determining the merit of a scientific research report. This framework rests on a conception of how science works that I call "scientific research as problem-solving." Several philosophers of science emphasise the problem-solving nature of science. According to Thomas Kuhn (1970), so-called normal science consists of "puzzle solving," as intricate instrumental, conceptual, mathematical, and empirical problems must be solved. The challenge of fitting all the right pieces of the puzzle functions as an important driving force in scientific research. Even in revolutionary (extraordinary) science, scientists present new comprehensive theories that solve the problems that existing theories have encountered and failed to deal with appropriately over a long time. Karl Popper (1981) thought that the result of stringent critical examination and rigorous testing of existing scientific contributions is the presentation of imaginative, new scientific theories, results, methods, and techniques. These new proposals aim to correct problems, deal with challenges, and alleviate pressures on existing accepted scientific results. Robert Dunbar (1995) depicts science as specialised, refined methods of problem-solving that build on universally human (and animal) abilities of problem-solving through trial and error. 2 A simplified definition of science could go as follows. Science is the human quest to know and understand the worlds around us and within us. All good science operate according to four fundamental rules, that are specified and interpreted by scientists according to the nature of their own discipline (see Rossouw 1993: 95–97). In the rest of this article I will discuss these four rules as basis for judging whether scientific research reports merit publication in scientific journals. The four rules for good science are: [1] scientists must use specialised problem-solving; [2] scientists must refer to previous work; [3] scientists must justify their findings; and [4] scientists must convince their scientific community. 1. Use specialised problem-solving In a conception of science that judges the main characteristic of scientific research as problem-solving, the first rule of good science ought to be that scientists must use specialised problem-solving in their research. The implications of this rule for scientific research reports are as follows. Scientific research reports [1] must deal with a genuine research problem, [2] the researcher must have a clear purpose in dealing with the problem, [3] the research problem must be unpacked with adequate understanding and in sufficient detail, [4] the researcher must formulate a tentative solution to guide the research process; and [5] an appropriate method must be selected or designed for dealing with the problem. These issues will now be discussed in more detail. 1.1 What is a research problem? Many scientists think that problems that they find interesting, or that they feel they must know more of, are suitable candidates for scientific research problems. Other scientists believe that problems that have practical consequences, like pollution of the environment, or problems that might lead to important applications, are good research problems. Many good research problems originate in people's curiosity, in practical consequences, or possible useful applications. However, that is not enough. Good research problems are ones that scientists [1] can demonstrate are relevant to one or more research communities and [2] can convince their colleagues to judge as significant. Scientists judge research problems as significant if they deal with [1] issues that are not yet known at all or not known in sufficient detail or [2] issues that will solve some of the theoretical, empirical, technical, or instrumental problems faced by their disciplines. 1.2 What is the purpose of dealing with the problem? Good research reports must have a clearly defined purpose for dealing with an acceptable research problem. A general purpose for research is to reach a better understanding of the problem. A better understanding could mean different things. A scientist might want to give a first time description of a phenomenon that is not yet known. Often scientists aim to give improved descriptions of phenomena, i.e. more detailed and accurate descriptions made possible by the use of improved theories or measuring equipment. Besides better descriptions, scientists might have many other purposes in dealing with research problems. Some of the more common ones are as follows. Scientists might want to offer explanations for phenomena or behaviour currently not well enough understood. Where adequate explanations do exist, scientists might want to make predictions of what might happen in future and measure the accuracy of those predictions. Adequate explanations of phenomena and behaviour are part of developing a theory to provide a more comprehensive understanding of the issues at hand. To develop, correct, modify, and refine theories are legitimate purposes for scientists to pursue. On the basis of well-developed theories scientists would often suggest solutions for problems or interventions to improve problematic situations. A further legitimate purpose for scientific research is to evaluate such interventions and proposed solutions. In the highly specialised world of contemporary science, research articles often deal with a minute subsection of one of the purposes mentioned above. The link of the research problem with its broader context must, however, be clear. 3 1.3 Unpack the problem It is not enough to have a legitimate research problem and a clear purpose for studying it. Good research reports show their understanding of their research problems through an adequate unpacking of the problem. What it means to unpack a research problem could be different from one discipline to the next. In all cases though, scientists must demonstrate the significance of a clear and detailed understanding of the problem they are studying. What are general requirements for unpacking a research problem? An important requirement would be to analyse a problem into its component parts so as to know which factors are involved. This kind of analysis can be complex and could sometimes be the main purpose of a research project. At times it is necessary to trace the history of a research problem, to find out how the problem was understood by scientists in the past. In their analyses of research problems, scientists must acknowledge what they do not yet know, or do not know in sufficient detail. The reason why that missing knowledge is important must also be stated. Part of unpacking a research problem is to be clear on exactly what it is that is being studied. The key concepts that are used in the research must be clearly defined and how they relate to the world must be made explicit. The research problem and the key concepts needed to formulate the problem must be linked to broader theoretical frameworks currently dominant in the relevant scientific disciplines. In this way the significance of the problem as scientific research will become evident. 1.4 Formulate a tentative solution After unpacking the problem, scientists must formulate a hypothesis, a claim, or an aim to guide their research. A hypothesis is a proposition in testable form about the relation between two or more variables, purporting to provide a tentative correlation, or explanation. A claim is a proposition about a concept, events, phenomena, arguments, etc. that can be supported or rejected through arguments and evidence. An aim is a statement of what they intend to deliver as outcome of their research process. Hypotheses, aims, or claims function as tentative solutions to the research problem under investigation that will be critically evaluated in the light of available evidence and arguments. 1.5 Find a method for dealing with the problem If the problem is well understood and the aim of the research clearly defined, then a scientist must find the right method for dealing with the research problem. As a general rule the research problem must determine the appropriate research method. This rule presupposes that the researcher is skilled in various methods that can be used as needed. Researchers are, however, not always multi-skilled in different research methods. In such cases the general rule can be modified to the idea that the method determines the problem. This rule applies to cases where the researcher is skilled in one method only, for example. The range and possibilities of the method's applicability thus limit the kind of research problems that the researcher can investigate. Researchers must beware of forcing their favourite method onto research problems that might better be studied through other methods. For this reason researchers must be aware of the strengths and weaknesses of research methods available to them. A comparison between researchers and carpenters is instructive (cf. Lötter 1995: 52–54). Like carpenters, researchers have a set of tools available that can fulfil different functions. Like carpenters, researchers must know the possibilities and limitations of their tools and use them appropriately. Furthermore, like the tools of carpenters, the tools of researchers are continually modified, improved, and sometimes drastically changed. Researchers must thus know about modifications to their tools and the effects 4 thereof on their research. They must be aware of new applications for their tools and how those tools might be combined to do complex or intricate functions. 2. Refer to the work of others By now it might be obvious that no one can do science on their own, without knowing what other scientists have done and still are doing. Good science is produced by scientists who do not have tunnel vision, staying only within the narrow confines of their own particular research problem or within the strict boundaries of their own discipline. What does this imply? 2.1 Refer to previous work on this problem and related ones A prerequisite for producing good scientific research reports is that scientists must study relevant scientific literature to know how their problem and similar ones have been dealt with in previous research (Mouton 1996: 119–124). Being knowledgeable about previous research enables scientists to embed their research projects into the body of currently accepted knowledge on their problem and related ones. Through awareness of such knowledge scientists can locate, plot, and position themselves within a kind of "virtual reality" comprised of state of the art knowledge dealing with their problems and related ones. To take note of research on related problems might enable scientists to imaginatively transpose or adapt ideas and solutions to their own research problems. To be knowledgeable about relevant literature is not enough. Scientists must be able to make a coherent argument about the literature. In such an argument – however briefly stated – scientists must be able to indicate how reliable and valid current knowledge is. They must point to current shortcomings in available knowledge and hint how they are going to overcome those shortcomings and add to, or modify, existing knowledge. An adequate definition of a research project makes scientific sense when the project is thus positioned with reference to currently available research results. 2.2 Refer to work in other disciplines too Some scientists believe in the so-called "First Commandment of Academia" which goes as follows: "Thou shalt not transgress thy disciplinary boundary" (Kline, 1995: 5). Unfortunately for them most examples of scientific progress overstep this boundary! Many important advances result from the way that scientists appropriate methods, techniques, and results from other scientific disciplines. The illuminating role that statistical analysis play in many human sciences is a good example of the value that other scientific disciplines can contribute to renewal and creativity if scientists are willing to learn from them. Co-operation between scientists across disciplinary boundaries can be explained further. Many scientific disciplines and fields share interests as their objects of research are similar or have significant overlaps. Such disciplines and fields form groups, or clusters that investigate similar, or analogous phenomena. As there are continual shifts within the disciplines themselves, the relationships between disciplines have the character of temporary alliances. Between these disciplines and fields there are continuous transfers of methods, techniques, theories, and results used for reinforcement. Scientists usefully appropriate methods, results, theories, and techniques from others to improve the scientific accuracy, reliability, and credibility of their research. In the process of appropriation scientists typically adopt something from their colleagues elsewhere, then they adapt those intellectual products to suit their purposes, and eventually they become adept at using the appropriated intellectual instruments as if they were their own. The appropriation of laser technology, statistical formulas, and participant observation are good examples. 5 Why do scientists cross disciplinary boundaries in search of intellectual products for appropriation into their disciplines and fields? They do so with the hope that they can widen their own horizons and expand the frontiers of science through their own research. This practice of borrowing between disciplines are found in most sciences and it generates new differentiations and specialisations within disciplines. Boundaries between disciplines are shifted and new combinations or fusions between disciplines and fields are formed. To note related or relevant research in other disciplines that might improve the quality of one's own research project can transform an ordinary, bland research project into a path-breaking, cutting-edge research project, if done with the necessary skill and insight. However, this does not imply that research within the strict confines of a discipline is second-rate or not capable of creative contributions. 3. Justify everything Researchers are confronted with constantly changing research tools – methods, techniques, theories, and research results are constantly modified or renewed. Besides being careful in selecting appropriate methods, scientists must be able to justify their research problem, their choice of method or combination of methods, as well as their application of those methods. This need to justify the research problem and methodology requires that scientists must know the state of the art concerning both widely accepted results and research methodology in their discipline. The choice of method is done in the context of currently accepted standards and results in a discipline. Justification is extremely important in science. Not only the choice of method, but also the understanding of a method and its use in a research project need justification. Reasons must also be given for the way the currently accepted literature are interpreted and why certain authors or sections are judged relevant to a research project. The evidence or grounds used in a research project as support for conclusions need to be justified to be accurate and reliable, as well as precise and appropriate to the field under investigation. Similarly, a scientist must give reasons why the evidence or grounds can be taken as representative and relevant to the research problem. Evidence or grounds need appropriate analysis and adequate interpretation in order to derive convincing results. The analysis and interpretation must therefore be defended in terms of current state of the art practices as well. A discussion of research results must provide two justifications. One is to demonstrate that the conclusions are consistent with the analysis of evidence or arguments. This implies that all parts of the research project cohere and reinforce one another. The discussion of research results usually contains a comparison of the results with existing results to indicate whether the new results are similar, analogous, or contradictory to the currently accepted results. The upshot of the comparison needs explanation and justification as well. Reasons for the same or different results must be given so as to convince fellow scientists that the research results are meaningful and significant. 4. Convince the scientific community By now it ought to be clear that science is a communal human activity. No one can practise science in isolation from what other scientists have done and are currently doing. Not only must a scientist's work be informed by the work of others, but a scientist's research must be directed at fellow scientists for accreditation. Research only qualify as scientific once fellow scientists recognise it as such. If the communities of scientists in various disciplines and fields play such an important role in certifying the quality and significance of research, how are they to be convinced? Before this question can be answered, the way scientific communities work must be explained. 6 4.1 How does this community work? Two aspects of the nature of scientific communities are relevant for understanding how scientific research articles are judged. One aspect concerns decision-making and the other the continual changes in scientific communities. The power of decision-making is widely distributed in scientific communities. There is no central control over science, as decision-making powers are found in scattered locations. Decision makers who determine what good science is, are found in editorial boards and among referees of scientific journals, referees and consultants of publishers, examiners of postgraduate research, organisers of conferences, appointment committees, award committees, and ordinary researchers as consumers of already published research. The communities of scientists in these various roles determine the nature and standards of science. Thus, what is considered to be viable research projects and what becomes accepted results and theories are determined through millions of smaller decisions made by scientists in different roles and various contexts (see Lötter 1999). Although many errors of judgement are made, those errors of judgement have a good chance of being corrected. The judgements are made by many people independently of one another and the same research results can be judged at more than one site of decision-making. Errors of judgement can thus be minimised or eliminated. A characteristic of science is that scientific communities continually redefine their disciplines. How does this happen? At the frontiers of scientific research one finds the boundary between order and chaos, as in other complex systems. At this boundary one finds both continuity with previous work, as well as changes of all degrees. Both stability and continual flux of methods and results co-exist uneasily as a result of a struggle for dominance. The frontiers of research are characterised by divisions between scientists and struggles for priority and influence. Sufficient consensus exists for scientists to understand one another and they have overlapping beliefs about the important developments and achievements in their field and related ones. However, new ideas erode the status quo. New ideas result from newly proposed theories and results. New ideas lead to creative tensions between the already accepted research results and the newly proposed ones. Significant contributions in science are those that are judged to reinforce, modify, renew, change, or destroy what are currently accepted. Research results that merely duplicate or repeat what is already known and accepted are not significant. Significant contributions to a scientific discipline are well-supported, thoroughly justified contributions that improve and extend our knowledge through renewing, revising, or changing the current state of the art (see Rouse 1990). Thus, scientific research leads to continual transformation within disciplines. To be a scientist requires the ability to adapt to an environment of continually changing theories, results, methods, techniques, and fellow scientists. Scientists are therefore involved in a process of co-evolution – they are evolving in response to their evolving environment (see Lötter 1999). The environment evolves as a result of the way scientists rethink the contents, methods, and aims of their disciplines. The degree of revision and reconstitution of a specific discipline as a result of new contributions varies. At times radical change occurs, while at other times gradual change results from small modifications. The question is whether a research article makes at least a modest contribution toward such change. 4.2 How does a scientist convince them? 7 What must scientists do to get their research accepted as significant contributions to their discipline? The easy answer is that they must present well-founded arguments employed in dialogue with currently accepted research results. These arguments must be directed at fellow scientists in their field or discipline who are the decision makers determining the nature of good science. Sometimes research is directed at even smaller specialised groups within a discipline or field. In the human sciences such "subgroups" are sometimes paradigm-based, meaning that they operate in a discipline split by opposing broad theoretical frameworks. Members of opposing camps often do not recognise the validity of one another's research. What do scientists use to convince the decision makers? In general they use well-constructed research reports. There is no doubt that science depends on communication through solid argumentation. Good communication is the duty of researchers – if they want others to take their research seriously, they must ensure that they communicate well. Good communication further enhances the impact of a researcher's proposed research results. What are the requirements of communicative research reports? Requirements differ from discipline to discipline and often even from journal to journal. Despite such differences, some general requirements for research reports can be specified. Most research reports ought to be reconstructions of the research process. Thus, important aspects regarding the research process need to be discussed. This might require details about the gathering of information and the choice and use of methods. A description and justification of the procedures of analysis and interpretation might be required. Often an evaluation of findings is needed as well. The overall shape of the report is important as well. In general a report must express an argument in support of a particular position, view, interpretation, or description. There must be an appropriate relation between the evidence presented and the claim made in the argument. Furthermore, the points or sections in the report must add up to a coherent whole that supports or rejects the main claim, hypothesis, or argument (see Booth et al 1995: 85–110). Readers must be able to detect the principle of organisation of the information presented in the report. For this reason a research report must have an identifiable structure clearly expressed in important sentences. The problem discussed and the proposed solution offered must have strong links. The introduction of a research report must announce the main theme(s) and readers must be able to track those themes clearly through the report. A report is mostly divided into sections and they must have introductions stating clearly what is going to follow. Sections must hang together to form a coherent whole (Booth et al 1995: 201–212). Scientists must use good language in research reports that are understandable and comply with current use in the discipline. The style and grammar of report must be acceptable. Research reports must comply with technical guidelines of the journal or publisher in question, e.g. punctuation, citation form, etc. 5. Conclusion Is it possible to avoid mean and nasty referee reports? Is it possible to improve the quality of research reports so that they become significant contributions to the continuing growth and development of a scientific discipline? In both cases the answer is yes. Mean and nasty referee reports can easily be avoided by making the well-founded assumption that all scientists wish to contribute to their discipline's growth. Their wish can be judged impartially in terms of the current state of the art methods, techniques, results, and theories in their discipline. Many aspects of research reports can be judged in this collective light – the research problem, the choice of methods, ways of analysis and interpretation, and the discussion of research results. A referee must also judge whether the tools of the trade have been used competently. The research report must comply with general guidelines for research reports and 8 those guidelines also enable fairly impartial judgement. If referees too are required to give acceptable reasons for their decisions and to argue their judgements, arbitrary judgements based on personal prejudice can be avoided. Scientists producing research reports can enhance the quality of their work through a thorough knowledge and sharp awareness of the current state of the art methods, techniques, results, and theories in their discipline. Competent and creative use of state of the art tools in studying important problems, with results embodied in communicative research reports, ought to improve research articles. Bibliography Booth, William; Colomb, Gregory G.; Williams, Joseph M. 1995. The Craft of Research. Chicago: The University of Chicago Press. Dunbar, Robin. 1995. The trouble with science. London : Faber and Faber. Garcia, Jorge J. E. 1999. Philosophy in American Public Life: De Facto and De Jure, Proceedings and Addresses of The American Philosophical Association, 72, (5), 149–158. Kline, Stephen Jay. 1995. Conceptual foundations for multidisciplinary thinking. Stanford : Stanford University Press. Kuhn, Thomas. 1970. The structure of scientific revolutions. (Second Edition, Enlarged). Chicago : The University of Chicago Press. Lötter, H. P. P. (Hennie). 1999. "The complexity of science," Koers, Vol. 64, No. 4, pp.–22. Lötter, H. P. P. (Hennie). 1995. "Kwalitatiewe of kwantitatiewe navorsing? 'n Wetenskapsfilosofiese perspektief," Acta Academica, Vol. 27, No. 2, pp. 28–56. Mouton, J. 1996. Understanding social research. Pretoria: J. L. van Schaik. Popper, Karl. 1981. The rationality of scientific revolutions. (In Hacking, Ian. ed., Scientific revolutions. Oxford: Oxford University Press, pp. 80–106.) Rossouw, H. W. 1993. Universiteit, wetenskap en kultuur. Kaapstad : Tafelberg Uitgewers. Rouse, Joseph. 1990. The narrative reconstruction of science. Inquiry, 33 (2), 179–196.