Review instructions

Your goal as an artifact reviewer is to ensure that the quality of the artifact matches the content of the article and the minimum requirements expected to obtain each badge. To carry out this activity with excellence, before evaluating the artifacts (i) read the article; and (ii) define the main contributions of the work. These two steps make the evaluation process for each badge simpler.

Note: Please note that the review period is relatively short. We recommend starting your revisions as soon as you receive your assignment.

Steps for evaluating an artifact

The review process can be performed in an environment of your choice, as long as it meets the minimum requirements of the expected execution environment for the artifact. We recommend running the artifact (when applicable) in a virtual environment, as it brings convenience to reviewers and ensures that components present on your local machine do not harm the evaluation process (a clean installation in a new environment can reduce unforeseen events).

All additional resources required to run the artifact (cloud infrastructure, SSH keys, etc.) must be present in the appendix describing the artifact.

The artifact under evaluation is related to an article being evaluated by the SF and WTICG committees. A CTA reviewer's focus is on the artifact rather than reviewing the article. However, if any issues are found, they should be reported to the artifact assessment coordinators.

Note: Please remember that all artifacts, analyzes and discussions are confidential.

Review Process

The review process is divided into two stages. Firstly, you must select works whose themes are familiar to you for the subsequent review process. Choose papers as soon as possible so that the committee is aware of each member's reviews. The work is presented in an online spreadsheet that will be shared with CTA members. Each CTA member will choose 3 or more works to evaluate (number to be confirmed by coordinators later).

Once you have selected your work, you can now start making revisions. To guide this second stage, a set of instructions with key points of the artifact evaluation process was defined by the committee, as set out in the definitions of requirements for obtaining each badge. For the artifact to be eligible to receive a badge, the respective requirements must be met.

For the work/artifact to be eligible to receive the badge, the respective requirements must be met:

Available Artifacts (SeloD)

Code and/or data is expected to be available in a stable repository (such as GitHub or GitLab). In this repository, a README.md is expected with minimal documentation (but clear enough for understanding), describing the objective of the artifact(s) (s), with the respective title and summary of the article.

Functional Artifacts (SeloF)

It is expected that the program code(s) of the artifact(s) can be executed and reviewers can observe its functionalities. To obtain this badge, it is important that additional information is present in the repository's README.md, such as:

  1. list of dependencies;
  2. list of dependencies/languages/environment versions;
  3. description of the execution environment;
  4. installation and execution instructions;
  5. one or more execution examples.

Sustainable Artifacts (SeloS)

It is expected that the program code(s) of the artifact(s) are modularized, organized, intelligible and easy to understand **. To obtain the badge you must:

  1. there is documentation of the code(s) (describing files, functions, APIs, etc.);
  2. there is minimum readability in the code(s) and other artifacts;
  3. it is possible for reviewers to identify the main claims of the article in the artifact(s).

Reproducible Experiments (SeloR)

It is expected that the reviewer will be able to reproduce the main claims presented in the article. To obtain this badge you must have:

  1. instruction to execute the code(s) in order to reproduce and confirm the main claims of the article (e.g., results of the main graphs/tables);
  2. description of the process of carrying out the experiments to reach the result(s) of the article;
  3. description of specific technical details of the environment (when applicable) such as, for example, details of the infrastructure used in Amazon Cloud or Google Cloud. Eventually, include access keys and other information that allows the experiment to be reproduced.

Submission of Reviews

For each artifact, you must produce a brief review text justifying the reason for assigning or denying the artifact a badge. The review text should only be written after the evaluation process has been completed.

To facilitate the evaluation process, complete the comments for each badge using the following template that takes into account the requirements of each badge.

Examples:

(case 1, badge assigned) SeloD: publicly available artifact(s) (comments)
+ available in a stable repository
+ sufficient documentation
+ title
+ summary
I recommend awarding the badge.

(case 2, badge not assigned) SeloD: publicly available artifact(s) (comments)
+ available in a stable repository
- sufficient documentation
+ title
- summary
I cannot recommend the attribution of the badge, because the artifact does not have documentation considered sufficient to understand and reproduce its characteristics and because the summary of the article was not provided.

(case 3, IF THE BADGE HAS NOT BEEN REQUESTED) SeloD: publicly available artifact(s) (comments)
The badge was not requested for this category.

Use the signs (-) and (+) to demonstrate which of the criteria the artifact does not satisfy (- sign) or satisfies (+ sign). It is also important to include a conclusion in the review, explaining the motivations for not awarding the badge when the artifact does not meet the minimum expected criteria.

Note Try to write your review in a precise, impersonal and polite way, considering that it will be available to the authors at a later stage of the process.