DOI: 10.1109/ICDS.2010.14 | CITEULIKE: 6709246 | REFERENCE: BibTex, Endnote, RefMan | PDF PDF Version

Bartneck, C. (2010). The All-In Publication Policy. Proceedings of the Fourth International Conference on Digital Society (ICDS 2010), St. Maarten pp. 37-40.

The All-In Publication Policy

Christoph Bartneck

Department of Industrial Design
Eindhoven University of Technology
Den Dolech 2, 5600MB Eindhoven, NL
christoph@bartneck.de

 

Abstract - The productivity of scientists and the quality of their papers differ enormously. Still, all papers written get published eventually and the impact factor of the publication channel is not correlated to the citations that individual papers receive. Hence it does not matter where to publish papers. Based on these two conjectures, I conclude that all papers should be published. The review process should focus on feedback that helps authors to improve their manuscripts. But we should no longer waste effort to a selection procedure. This All-In policy would decrease the number of published papers and would refocus the attention of the authors on the quality of their papers and not their quantity.

Keywords: publication process; peer review; impact factor; quality;


Introduction

From the ice-age to the web-age
there is but one concern,
I have just discovered:
some scientists are bigger than others.
Some scientists' papers are bigger than other scientists' papers.

(lyrics adapted from Morrissey & Johnny Marr, 1986)

The productivity of scientists and the citations that papers receive are extremely unevenly distributed. For the CHI conference, 18 percent of papers receive approximately 80 percent of the citations [1]. A similar distribution can be expected of the ACHI conference. This mirrors the general observation that scientific productivity and citedness are heavily skewed [2]. One might then try to select only the very best papers into a unit, such as a journal or an elite conference. In the past, this was even necessary due to the limited number of pages available in a journal. The selection process is usually done through the peer-review process. The accepted papers are published, receive citations and the journals or conferences pride themselves with the resulting impact factor that signifies their status. Authors then send their best papers to journals or conferences with a high-impact factor with the assumption that their paper would receive more attention from the scientific community, but also from their direct colleagues and superiors.

This is of course a very nave view of the scientific publication process, waiting to be scrutinized. I shall try not to disappoint you. The scientific publication process and in particular the peer review process has been criticized before (for a good review see [34]), but I am not completely convinced that the consequences of the arguments brought forward have reached the full awareness of the scientific community. We still try to write as many papers as we can, submit them to journals and conference and experience joy upon acceptance. We suffer from the sheer endless numbers of papers to read, the burden of reviewing, and the frustration of rejections. This can be particularly frustrating in a multidisciplinary field, such as human-computer interaction, because the criteria upon which submission are being evaluated are not necessarily shared by the diverse set of reviewers [5], which can even result in shouting matches [6]. I will therefore present two major challenges to the current publication process and discuss their logical consequences before presenting possible improvements.

Conjecture A: It does not matter where you publish your articles

It has been convincingly shown that there is no correlation between the impact factor of a journal or conference and the citations its papers receive [7]. The citations that papers receive within the very same journal are heavily skewed. A few papers receive many citations and most papers receive very little citations. It is a myth that publishing your article in a high-impact factor journal will automatically result in high citations of your own work (free ride theory). But why do have some journals have a higher impact factors than others? It is because people continue to believe in this myth and sent their best work to the assumed best journals. But again, the papers make the journal and not the other way around. Publishing an article in high impact journals or conferences might give you the recognition of other believers, but certainly not more citations.

Conjecture B: Every article written gets published

I asked many colleagues and friends if they ever gave up on a paper they had written. Meaning that they would not re-submit it after an initial rejection. Nobody did. All papers that were written got published eventually. This is not surprising since publication channels exist that have such a relaxed review process that even complete nonsense papers get accepted [89]. Why waste the writing effort when there is always a publication channel open that accepts the work?

Conclusion: All papers shall be published

If all papers get accepted eventually and if it does not matter where papers are published then it makes no sense to waste effort on a selection procedure. In the computer age, page limitations do not exist anyway. This does not mean the author should not be given the opportunity to improve their paper based on constructive feedback and it does also not mean that author may not retract their manuscript if desired. It only means that every submitted manuscript should not be denied publication. What would be the consequences of such an all-in publication policy?

One might think that authors would publish even more papers. Already today we are suffering from such an onslaught of scientific publications [10], that is also reflected in the grows of literature in the field of human computer interaction [1], to a degree that hardly anybody can keep an overview of the field and automatic systems are being developed for it [11]. Would the abandoning of the selection procedure not worsen the situation? I think not! If it is no challenge to publish many papers, then productivity becomes a meaningless dimension upon which to judge scientists. If everybody can publish hundreds or even thousands of papers easily then it makes no sense to use this as a promotion criteria. It would become a meaningless indicator for staff appraisal and the distribution of funding. Researchers would be able to decrease their scientific output and focus on the quality of their contributions. The focus would move away from quantity and towards quality, a move that we all should welcome.

Another consequence of the all-in publication policy would be that journals would stop existing since the selection procedure is the reason of their existence to start with. I will pick up the discussion about consequences for publication channels towards the end of this paper. First, I would like to discuss what an all-in publication policy would not change.

Lets start with an easy aspect of the publication process that should remain: the presentation. Already today, the layout and language editing is often done by the authors themselves. Putting effort into the presentation makes the papers easier to understand and this effort should continue. Free layout programs, such as LaTeX, help authors to typeset their manuscripts and many commercial editors correct language issues. However, the language editor is usually of little help when it comes to editing the core content of the paper.

Improving the content of a paper is a difficult task and the peer review process is partially intended to help. However, the process has its own set of problems and its weaknesses have been discussed in length [1213].

One of the major suggestions on how to improve the peer review process is to separate feedback from judgment [14]. Cycles of editing a manuscript based on feedback received from peers certainly helps to improve manuscripts. However, writing a good review takes considerable effort and this effort is currently not appropriately rewarded. Two approaches have been proposed to overcome this problem. First, market based systems have been proposed [15] in which reviewers gain points, which helps them when they themselves want to publish a papers. Such market-based system would require the scientific community to agree on a system and value structure. This might be difficult to achieve.

A second approach is open peer commentary, in which the reviews themselves are considered a publication. Associating ones own name to the review of a paper certainly increases the commitment that reviewers would feel towards their review and they receive the opportunity to shine through the quality of their review. But the feedback should not be limited to only a handful of invited reviewers. The whole scientific community should have the opportunity to give feedback. Today, discussing a paper in once own paper does this, but the publication delay makes these kinds of discussions unattractive in comparison to a direct discussion. Still, discussing one owns work in relation to previous work is an essential ingredient for the functioning of science and therefore a system of referring to work must remain. Various standards on how to cite work exist, such as the APA style [16], and the recently introduced Document Object Identifier (DOI) makes the lookup of articles even easier (http://www.doi.org/).

The access to the articles must be guaranteed over a long period of time. Putting an article on ones own web page is in principle already sufficient to publish it and to give it a unique address that can be cited. But the web is a very dynamic environment and people change their web pages, move to new organizations or retire. We still need archives that last longer than the careers of individual researchers. We still need archives. In the past, commercial publisher generated value by reproducing the distributing scientific literature. The universities build the archives of the printer journals. This distribution of tasks is undergoing major changes. Reproduction and distribution has become trivial with the arrival of the internet and the commercial publishers are struggling with justifying their inflated prices [17]. In the digital age, they started to maintain their own archive, selling only the access rights to the libraries. The absolute bizarre role of the commercial publishers has been criticized in length [18] and it should not be the focus of this paper since there are already viable alternatives.

The first alternative is non-commercial institutional publishers, such as the ACM or IEEE. They at least are directed by the research community itself and are not setup to maximize profit from publishing scientific literature. They are intended to serve their respective community.

But there is an even better alternative: our own organizations. We ourselves have to change. In my university, the budget of the library comes directly from the central university administration. My department is not charged a penny when I publish an article in a journal, much to the joy of my superiors. However, it is an illusion to believe that publishing in journals comes for free. Our libraries have to pay the bill, because they purchase the subscriptions to the journals. But since their budget is handled independently from the each individual department, it receives less attention. I am not certain if many other universities have similar financial models, but I assume that the situation may be to some degree the same. The solution is of course to stop poring money into journal subscriptions and instead investing it into open access publishing. Meaning that the researcher pays a certain amount up front and then the access to the articles is for free to everyone. Reports indicate that millions of Euros can be saved by adopting an open access publishing strategy [1920].

All of the activities described so far can be done online. Nobody needs to travel to write, give feedback, and publish the papers on the internet. Again, we could save millions of Euros by no longer attending conferences. Do we really need conferences?

The changing role of conferences

Conferences have an obvious disadvantage as a publication channel: authors have to attend. I do not have precise numbers as to how much money is annually spend for travelling and registration fees, but given the thousands of conferences each year, I imagine that it is not trivial, an estimation that is shared by others [21]. The human computer interaction community as well as the computer science community value conferences, but this does not hold true for all branches of science. In some areas, only short position papers get submitted to conferences as the basis of a discussion. These papers are not considered as full publication, this status is reserved for journals.

What then drives all those HCI researcher to travel around the globe? The paper presentations are certainly not it, since one can just read the papers. I can imagine two reasons that we cannot be proud of and one that we can. In the we cannot be proud of category we have the fact that conference trips are an academic reward structure. You plan and conduct a study and as a reward you may travel to a conference, occasionally in exotic locations. One can consider this as a form of professional tourism. Another frightening explanation is that the attention of the community is in short supply and that at least the people in the audience will have noticed the study. Otherwise nobody might notice the existence of the papers at all.

But there are also advantages to the conference publication channel. First, there it is a rigid and fast process. The deadlines for submission, review and publication are fixed. The process is usually much quicker than the journal review process that can easily stretch for one year between the initial submission and the final publication. Second, it allows researchers to meet and talk about their work, in particular the informal aspects, in face-to-face communication. We can build networks, hunt for jobs and discuss important results. This is the true strength of conferences. But the conferences are not yet optimized to nurture this exchange of ideas. The best way to discuss your study with experts in the field is not a paper presentation, but a workshop. Only in these small, dedicated group meetings time is available for in depth discussion. The five to ten minutes question and answer sessions after paper presentations is not match for it. Conference should therefore only consist of workshops.

Conclusion

I propose that all scientific papers should be accepted. Providing feedback that helps the authors to improve their manuscript is desirable and authors should be allowed to go through several cycles of revisions, but in the end every paper shall be accepted. An open peer commentary policy, in which the reviews for a certain paper are considered as publications would motivate reviewers to spend effort on the review. But the review process should not end with the publication. The scientific community should be able to comment directly on scientific contributions.

The all-in publication process would dramatically shorten the publication process and decrease the scientific outcome. It would no longer be a challenge to publish many papers and therefore this evaluation criterion would become meaningless. Instead, authors could focus on the quality of their papers and vivid discussion with their peers. Following this line of thoughts, conference should also refocus on providing a better platform for academic debates. I propose that conference should only consist of workshops and possibly some social events. Conference should be about meeting peers and the discussion of science. Paper presentations are inferior to reading the paper in terms of efficient communication. The amount of people that check their emails during paper presentations clearly indicate that that this form of scientific communication has become inadequate.

But not only the conference publication channel needs to evolve, the commercial and institutional publisher need to adapt the open access policy. The ACM half-heartedly proceeds in this direction by allowing authors to post their papers on their personal web pages and in their institutional repositories. The assumption is that this unstructured form of publishing would not endanger the ACMs digital library, since no central index exists. The Google Scholar service is becoming exactly that: a central index of scientific literature. All the papers published on personal web pages are being indexed. This shows that there is no halfway solution. I hope that the ACM and IEEE will be brave enough to change their publication model quickly. The Public Library of Science is an excellent example of how an open access model can work. Springer already introduced the Open Choice option, but the 2000USD price tag shows that they still underestimate the open access competition. It will be painful shrinking process for the commercial publishers, similar to the shrinking process that the music industry is going through. But in the end, even the music industry had to abandon the digital rights management shackles and so will the scientific publishers.

The changes proposed in this paper are radical and involve changes in the social structure of the scientific community. Promotion criteria for scientific staff, publication policies, funding criteria and communication patterns need to adapt. Theacademic publishing system is complex, involving researchers, careers, funding agencies, publishers and politicians. Changing such a system is difficult and we are limited to those parameters that we can control. The review process is in the hand of the researchers and hence it can be the field in which we initiate innovation.

Some people argue that science only progresses through the death of its professors. I expect that many senior professors and policy creators would oppose the changes proposed, but I still believe that change is possible. But being born in Germany, I know that even the most guarded walls can break down. Mikhail Gorbachev announced shortly before the fall of the Berlin wall that:

Трудности подстерегают тех, кто не реагирует на жизнь
(“Danger awaits only those who do not react to life.”)

Lets embrace the changes to the scientific publication process that the information age made possible.

References

  1. C. Bartneck and J. Hu, “Scientometric analysis of the chi proceedings,” in Conference on Human Factors in Computing Systems (CHI2009). Boston: ACM, 2009, pp. 699–708.
  2. O. S. Per, “The skewness of science,” Journal of the American Society for Information Science, vol. 43, no. 9, pp. 628–638, 1992.
  3. M. Enserink, “Scientific publishing: Peer review and quality: A dubious connection?” Science, vol. 293, no. 5538, pp. 2187a–2188, 2001.
  4. D. F. Horrobin, “The philosophical basis of peer review and the suppression of innovation,” Journal of the American Medical Association, vol. 263, no. 10, pp. 1438–1441, 1990.
  5. C. Bartneck, “What is good? - a comparison between the quality criteria used in design and science,” in Conference on Human Factors in Computing Systems (CHI2008). Florence: ACM, 2008, pp. 2485–2492.
  6. J. Arnowitz and E. Dykstra-Erickson, “Chi and the practitioner dilemma,” Interactions, vol. 12, no. 4, pp. 5–9, 2005.
  7. P. O. Seglen, “Why the impact factor of journals should not be used for evaluating research,” BMJ, vol. 314, no. 7079, pp. 497–, 1997.
  8. N. Scientist, “Randomly-generated ’scientific paper’ accepted,” New Scientist, vol. 2496, p. 6, 2005.
  9. A. D. Sokal, “Transgressing the boundaries: An afterword,” Philosophy and Literature, vol. 20, no. 2, pp. 338–346, 1996.
  10. D. A. King, “The scientific impact of nations,” Nature, vol. 430, no. 6997, pp. 311–316, 2004.
  11. K. D. Bollacker, S. Lawrence, and C. L. Giles, “Discovering relevant scientific literature on the web,” IEEE Intelligent Systems, vol. 15, no. 2, pp. 42–47, 2000.
  12. T. Jefferson, E. Wager, and F. Davidoff, “Measuring the quality of editorial peer review,” Journal of the American Medical Association, vol. 287, no. 21, pp. 2786–2790, 2002.
  13. D. P. Peters and S. J. Ceci, “Peer-review practices of psychological journals: The fate of published articles, submitted again,” Behavioral and Brain Sciences, vol. 5, no. 2, pp. 187–195, 1982.
  14. D. Kaplan, “How to fix peer review,” The Scientist, vol. 19, no. 1, p. 8, 2005.
  15. J. Crowcroft, S. Keshav, and N. McKeown, “Scaling the academic publication process to internet scale,” Communications of the ACM, vol. 52, no. 1, pp. 27–30, 2009.
  16. A. P. Association., Concise rules of APA style, 1st ed. Washington, DC: American Psychological Association, 2005.
  17. T. Delamothe and R. Smith, “Open access publishing takes off,” BMJ, vol. 328, no. 7430, pp. 1–3, 2004.
  18. J. Willinsky, The access principle : the case for open access to research and scholarship. Cambridge, Mass.: MIT Press, 2006.
  19. J. Houghton, J. d. Jonge, and M. v. Oploo, “Costs and benefits of research communication: The dutch situation,” SURFfoundation, Tech. Rep., 2009.
  20. J. Houghton, C. Steele, and P. Sheehan, “Research communication costs in australia, emerging opportunities and benefits,” Victoria University, Tech. Rep., 2006.
  21. L. Fortnow, “Time for computer science to grow up,” Communications of the ACM, vol. 52, no. 8, pp. 33–35, 2009.

This is a pre-print version | last updated February 22, 2010 | All Publications