Qualitätssicherungsprozess für Veröffentlichungen der Forschungsgruppen SDQ und ARE

Interne Begutachtung (von Mitarbeitern der Gruppe): All papers must be internally reviewed as follows before being submitted to workshops, conferences, journals etc.
- Reviewer: Ask colleagues who can likely give good feedback because they are familiar with your paper's topic. For conferences with a broader topic range (e.g. ICSE) or a topic range that does not fit your paper well, also ask colleagues who are less familiar with your paper, because they can give better feedback on the understandability.
- Paper State: Das Papier sollte zur internen begutachtung fertiggestellt und von den Autoren selbst gründlich gelesen und korrigiert sein. Zur Not kann ein Papier zu diesem Zeitpunkt auch noch unvollständig sein. Fehlende oder noch nicht finale Teile sollten dann entsprechend markiert sein, um den Gutachtern Arbeit zu ersparen.
- Frühe Abgabe: Einen Papier-Entwurf erst 2 Tage vor Abgabetermin der Konferenz zur Begutachtung vorgelegt zu bekommen, ist einerseits nicht höflich gegenüber den Kollegen, die selbst genug anderes zu tun haben und sich nicht kurzfristig Zeit freischaufeln können. Andererseits kann man in einem solchen Fall weder qualifizierte Kommentare erwarten noch sinnvoll die dann doch gemachten Kommentare einarbeiten und das Papier verbessern. Daher sollte Konferenzpapiere mindestens 2 Wochen vor Abgabetermin der Konferenz zur internen Begutachtung vorgelegt werden. Dann haben die Mitarbeiter eine Woche Zeit zur Begutachtung und dem Autor bleibt eine Woche zur Überarbeitung des Papiers anhand der Kommentare der Gutachter.
- Umgang mit Kommentaren: Erstmal sollte man sich klar machen, ob man den Kommentar verstanden hat, ansonsten nochmal nachfragen. Man muss nicht jeden Kommentar unkritisch übernehmen, jedoch sollte man sich auch klar machen, dass die Kommentare nicht nur (ignorierbare) Nörgeleien der Gutachter sind, sondern auf Probleme hinweisen. Es ist besser, diese Probleme im internen Begutachtungsprozess auszuräumen, als bei einer Konferenz abgelehnt zu werden. Widersprechen sich Kommentare von verschiedenen Gutachtern oder sind deren Kommentare nur schwierig in Einklang zu bringen, sollte man dies mit den Gutachtern diskutieren.
- Stand der Reviewfassung: Meist sollten nur fast fertige Versionen ins interne Review gegeben werden, um den Kollegen unnötigen Aufwand zu ersparen. Paper-Ideen sollten eher in Forschungstreffen o.ä. diskutiert werden, anstatt unfertige Paper-Skelette ins Review zu geben. Ein unfertiges Paper zum Review zu geben macht höchstens Sinn, wenn man zu fertigen Abschnitten konkrete Fragestellungen an die Kollegen hat.
Externe Begutachtung (vom Programmkommittee der Konferenz)
- Kein Debuggen durch Gutachter: Man sollte keinesfalls die Gutachter als Fehlerkorrekturinstanz missbrauchen und ein unfertiges oder mit Rechtschreib- oder Grammatikfehlern versehenes Papier einreichen. Daher vor der Einreichung gründlichen die formalen Aspekte des Papiers überprüfen.
- Umgang mit Kommentaren: Grundsätzlich sollten die Kommentare von externen Gutachter sehr ernst genommen werden und in der Gruppe diskutiert werden. Bei Kommentaren von anonymen Gutachtern kann man natürlich keine Verständnisfragen stellen, jedoch sollte man unklare Kommentare zumindest in der Gruppe der Mitarbeiter besprechen.

Double Anonymous

Many conferences ask for double anonymous (double blind) submissions. Read the instructions of the conference carefully.

If citing own previous work, use the third person instead of making the reference anonymous. It does not matter whether reviewers can guess your identity as long as they do not know for sure.

Provide reasonably anonymised replication packages, e.g. using „anonymous GitHub“.

Writing Reviews

This is a collection of advice for reviewing scientific articles from different conferences. These criteria help you to prepare your article for reviewing, but also help you to write high-quality reviews yourself. Please check whether the conference or journal that you are submitting to or reviewing for has specific instructions what to cover in reviews.

Before writing a review, check the call for papers for the workshop/conference to see what the expected types of submissions are.

Structuring your review

A useful structure for a scientific review is the following

Summary: Give a brief summary of what the content and the claimed contribution are
Evaluation: Provide the main review. It can be structured as follows (but you can also change it, depending on the kind of paper). Focus on the main points that were relevant for the decision of how to "grade" the paper (depending on the conference, the sclae could be "accept", "weak accept", "weak reject", "reject" or similar).
- Outline of argumentation: A paragraph that summarizes your evaluation and guides the reader (i.e. the authors) through the following, more detailed, points of criticism and praise
- Start positive: What is good about the paper?
- Points of criticism or praise: In one paragraph each (or more), you can present and discuss each point of criticism and praise. Next to points related to the specific content of the paper, please also discuss the following aspects:
- Contribution: Is the claimed contribution convincing and does the paper advance the state of the art?
- Related Work: Is the related work adequatly discussed? Please briefly search for other related work yourself (Google Scholar), to see whether something obvious is missing. Is the claimed contribution clearly different (and beneficial!) compared to what has already been published on the topic?
- Presentation: Is the paper well written and easy to understand?
- Validation (for conference papers): How the the contribution of the paper validated? Is it convincing? Are there threats to validity that are not (well) discussed? Is the experiment setup (or more general evaluation scenario or setup) appropriate?
Minor comments: Separate minor comments, such as typos and grammatical problems, from the main evaluation above.

Some conferences ask reviewers for a short bullet-point list of the paper’s key strengths and weaknesses, usually preceeding the main review. These lists should usually be short, both in the sense that only those strengths and weaknesses shall be listed that influence the overall evaluation of the paper (accept or reject) and in the sense that each bullet point should just mention the strength or weakness in usually one sentence maximum, if not only a few words. Overall, this list shall summarize the main strengths and weaknesses also discussed in the main review.

ICSE 2018 evaluation criteria for submissions to the technical track

"Research track submissions will be evaluated based on the following criteria:

Soundness: Are all claimed contributions supported by the rigorous application of appropriate research methods? The claims should be scoped to what can be supported, and limitations should be discussed.
Significance: Are contributions evaluated for their importance and impact with respect to the existing body of knowledge? The authors are expected to explicitly argue for the relevance and usefulness of theresearch and discuss the novelty of the claimed contributions through a comparison with pertinent related work.
Novelty: Is there sufficient originality in the contribution, and is it clearly and correctly explained with respect to the state of the art?
Replicability: Is there sufficient information in the paper for the results tobe independently replicated? The evaluation of submissions will take into account the extent to which sufficient information is available to support the full or partial independent replication of the claimed findings.
Presentation Quality: Are results clearly presented? Submissions are expected to meet high standards of presentation, including adequate use of the English language, absence of major ambiguity, clearly readable figures and tables, and respect of the formatting instructions provided below."

from the call for papers for the ICSE 2018 technical paper track

ICPE 2017 reviewer guidelines

"For your review, please consider novelty, relevance, technical quality and scientific soundness of the paper. Also note that we have different categories of papers with different quality criteria:

Full Research Paper - original and significant results of theoretical, empirical, conceptual, or experimental research with mature validation
Short Research Paper - novel ideas or innovative proposals whose aim is to stimulate discussion related to experience and ideas, rather than to present mature results.

[...]

Furthermore, when writing your review, please consider the following review quality criteria:

Provide feedback to authors
- be constructive
- avoid using a harsh tone
- provide evidence for your criticism (e.g. referring to paragraphs in the paper or by providing references)
Help the decision of the PC
- provide concise praise and criticism
- discuss points relevant to the decision
- explain why your praise and criticism holds"

Wiley

Wiley recommendation for journal reviewers at https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html

Wissenschaftliches Schreiben