Write a Blog >>
PPoPP 2021
Sat 27 February - Wed 3 March 2021

PPoPP is the premier forum for leading work on all aspects of parallel programming, including theoretical foundations, techniques, languages, compilers, runtime systems, tools, and practical experience. In the context of the symposium, “parallel programming” encompasses work on concurrent and parallel systems (multicore, multi-threaded, heterogeneous, clustered, and distributed systems; grids; datacenters; clouds; and large scale machines). Given the rise of parallel architectures in the consumer market (desktops, laptops, and mobile devices) and data centers, PPoPP is particularly interested in work that addresses new parallel workloads and issues that arise out of extreme-scale applications or cloud platforms, as well as techniques and tools that improve the productivity of parallel programming or work towards improved synergy with such emerging architectures.

Artifact Evaluation

Artifact evaluation submission site

Due Time: 11:59pm 11/26/2020 (AOE) 11:59pm 11/28/2020 (AOE)

Call for Artifacts

Authors of accepted PPoPP 2021 papers/posters are invited to formally submit their supporting materials to the Artifact Evaluation (AE) process. The Artifact Evaluation Committee attempts to reproduce experiments (in broad strokes) and assess if submitted artifact supports the claims made in the paper/poster. The submission is voluntary and does not influence the final decision regarding paper/poster acceptance.

We invite every author of an accepted PPoPP paper/poster to consider submitting an artifact. It is good for the community as a whole. At PPoPP, we follow ACM's artifact reviewing and badging policy. ACM describes a research artifact as follows:

"By "artifact" we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results."

Submission Site

The submission site is located at https://ppopp21ae.hotcrp.com/.

Evaluation Process

Artifact evaluation is single-blind. Please take precautions (e.g. turning off analytics, logging) to help prevent accidentally learning the identities of reviewers. Each submitted artifact is evaluated by at least two members of the artifact evaluation committee.

During the process, authors and evaluators are allowed to anonymously communicate with each other to overcome technical difficulties. Ideally, we hope to see all submitted artifacts to successfully pass the artifact evaluation.

The evaluators are asked to evaluate the artifact based on the following criteria, that are defined byACM.

ACM recommends awarding three different types of badges to communicate how the artifact has been evaluated. A single paper can receive up to three badges — one badge of each type.



The green Artifacts Available badge indicates that an artifact is publicly accessible in an archival repository. For this badge to be awarded the paper does not have to be independently evaluated. ACM requires that a qualified archival repository is used, for example Zenodo, figshare, Dryad. Personal webpages, GitHub repositories or alike are not sufficient as it can be changed after the submission deadline!
The red Artifacts Evaluated badges indicate that a research artifact has been successfully completed an independent audit. A reviewer has verified that the artifact is documented, complete, consistent, exercisable, and include appropriate evidence of verification and validation. Two levels are distinguished:

The lighter red Artifacts Evaluated — Functional badge indicates a basic level of functionality. The darker red Artifacts Evaluated — Reusable badge indicates a higher quality artifact which significantly exceeds minimal functionality so that reuse and repurposing is facilitated.

Artifacts need not be made publicly available to be considered for one of these badges. However, they do need to be made available to reviewers.
The blue Results Validated badges indicate that the main results of the paper have been successfully obtained by an independent reviewer. Two levels are distinguished:

The darker blue Results Reproduced badge indicates that the main results of the paper have been successfully obtained using the provided artifact. The lighter blue Results Replicated badge indicates that the main results of the paper have been independently obtained without using the author-provided research artifact.

Artifacts need not be made publicly available to be considered for one of these badges. However, they do need to be made available to reviewers.



At PPoPP the artifact evaluation committee awards for each successfully evaluated paper one of the two red Artifacts Evaluated badges as well as the darker blue Results Reproduced badge. We do not award the lighter blue Results Replicated badge in this artifact evaluation process. The green Artifact Available badge does not require the formal audit and, therefore, is awarded directly by the publisher — if the authors provide a link to the deposited artifact.

Note that the variation of empirical and numerical results is tolerated. In fact, it is often unavoidable in computer systems research - see "how to report and compare empirical results?" in AE FAQ on ctuning.org!

Packaging and Instructions

Your submission should consist of three pieces:

  1. The submission version of your paper/poster.
  2. A README file (PDF or plaintext format) that explains your artifact (details below).
  3. The artifact itself, packaged as a single archive file. Artifacts less than 600MB can be directly uploaded to the hotCRP submission site; for archives larger than 600MB, please provide a URL pointing to the artifact; the URL must protect the anonymity of the reviewers. Please use a widely available compressed archive format such as ZIP (.zip), tar and gzip (.tgz), or tar and bzip2 (.tbz2). Ensure the file has the suffix indicating its format. Those seeking the "Available" badge must additionally follow the appropriate instructions recommended by ACM on uploading the archive to a publicly available, immutable location to receive the badge.

The README file should consist of two parts:

  1. a Getting Started Guide and
  2. Step-by-Step Instructions for how you propose to evaluate your artifact (with appropriate connections to the relevant sections of your paper);

The Getting Started Guide should contain setup instructions (including, for example, a pointer to the VM player software, its version, passwords if needed, etc.) and basic testing of your artifact that you expect a reviewer to be able to complete in 30 minutes. Reviewers will follow all the steps in the guide during an initial kick-the-tires phase. The Getting Started Guide should be as simple as possible, and yet it should stress the key elements of your artifact. Anyone who has followed the Getting Started Guide should have no technical difficulties with the rest of your artifact. In this step, you may want to include a single high-level "runme.sh" script that automatically compiles your artifact, runs it (printing some interesting events to the console), collects data (e.g., performance data), and produces files such as graphs or charts similar to the ones used in your paper.

The Step by Step Instructions explain how to reproduce any experiments or other activities that support the conclusions in your paper. Write this for readers who have a deep interest in your work and are studying it to improve it or compare against it. If your artifact runs for more than a few minutes, point this out and explain how to run it on smaller inputs.

Where appropriate, include descriptions of and links to files (included in the archive) that represent expected outputs (e.g., the speedup comparison chart expected to be generated by your tool on the given inputs); if there are warnings that are safe to be ignored, explain which ones they are.

The artifact's documentation should include the following:

  • A list of claims from the paper supported by the artifact, and how/why.
  • A list of claims from the paper not supported by the artifact, and how/why. Example: Performance claims cannot be reproduced in VM, authors are not allowed to redistribute specific benchmarks, etc. Artifact reviewers can then center their reviews / evaluation around these specific claims.

If you are seeking a "reusable" badge, your documentation should include which aspects of the artifact you suggest the reviewer exercise in a different setting. For example, you may want to point out which script to modify so that the reviewer may be able to run your tool on a benchmark not used in the paper. You may want the reviewer to suggest where to edit a script to change the number of CPU cores used for evaluation.

Submission Guidelines

1. Carefully think which badges you want.

In your hotCRP submission, be upfront about which badge(s) you are seeking.

  1. If making your code public is all you want to do, seek only the "available" badge. The reviewers will not exercise the artifact for its functionality or validate the claims.
  2. If you do not plan on making the artifact available to the public, do not seek the "available" badge.
  3. If you only plan to reproduce the claims without making your artifact Documented, Consistent, Complete, and exercisable, do not seek the "functional" badge.

2. Minimize the artifact setup overhead

A well-packaged artifact is easily usable by the reviewers, saving them time and frustration, and more clearly conveying the value of your work during evaluation. A great way to package an artifact is as a Docker image or in a virtual machine that runs "out of the box" with very little system-specific configuration. Using a virtual machine provides a way to make an easily reproducible environment — it is less susceptible to bit rot. It also helps the AEC have confidence that errors or other problems cannot cause harm to their machines.

Giving AE reviewers remote access to your machines with preinstalled (proprietary) software is also possible.

The submission of an artifact is not the same as making it public. AEC members will be instructed that they may not publicize any part of your artifact during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, and similar items in your artifact.

After preparing your artifact, download and test it on at least one fresh machine where you did not prepare the artifact; this will help you fix missing dependencies, if any.

3. Carefully think your artifact working on a reviewer's machine

The reviewers will not have access to any special hardware or software outside of their own research needs provided by their university or research team. There are more tips preparing a submission available on the ctuning website.

If you have an unusual experimental setup that requires specific hardware (i.e., custom hardware, oscilloscopes for measurements …) or proprietary software please contact the artifact evaluation chairs before the submission.

Discussion with Reviewers

Throughout the review period, reviews will be submitted to HotCRP and will be (approximately) continuously visible to authors. AEC reviewers will be able to continuously interact (anonymously) with authors for clarifications, system-specific patches, and other logistics to help ensure that the artifact can be evaluated. The goal of continuous interaction is to prevent rejecting artifacts for "wrong library version" types of problems.

For questions, please contact AE co-chairs, Milind Chabbi (milind@uber.com) and Xu Liu (xliu88@ncsu.edu).

Call for Papers

PPoPP 2021: 26th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming

Seoul, S. Korea. (collocated with CC-2021, HPCA-2021 and CGO-2021) Dates: Feb 27-Mar 3, 2021.

Submission URL: https://ppopp21.hotcrp.com


Important dates:

  • Paper registration and abstract submission: August 6, 2020
  • Full paper submission: August 13, 2020
  • Early notification for papers not passing first stage: October 10, 2020
  • Author response period: October 30–November 2, 2020
  • Author Notification: November 16, 2020
  • Artifact submission to AE committee (tentative): November 26, 2020
  • Artifact notification by AE committee (tentative): December 26, 2020
  • Final paper due: January 2, 2021

All deadlines are at midnight anywhere on earth (AoE), and are firm.


Scope:

PPoPP is the premier forum for leading work on all aspects of parallel programming, including theoretical foundations, techniques, languages, compilers, runtime systems, tools, and practical experience. In the context of the symposium, “parallel programming” encompasses work on concurrent and parallel systems (multicore, multi-threaded, heterogeneous, clustered, and distributed systems; grids; data centers; clouds; and large scale machines). Given the rise of parallel architectures in the consumer market (desktops, laptops, and mobile devices) and data centers, PPoPP is particularly interested in work that addresses new parallel workloads and issues that arise out of extreme-scale applications or cloud platforms, as well as techniques and tools that improve the productivity of parallel programming or work towards improved synergy with such emerging architectures.

Specific topics of interest include (but are not limited to): - Compilers and runtime systems for parallel and heterogeneous systems - Concurrent data structures - Development, analysis, or management tools - Fault tolerance for parallel systems - Formal analysis and verification - High-performance / scientific computing - Libraries - Middleware for parallel systems - Parallel algorithms - Parallel applications and frameworks - Parallel programming for deep memory hierarchies including nonvolatile memory - Parallel programming languages - Parallel programming theory and models - Parallelism in non-scientific workloads: web, search, analytics, cloud, machine learning - Performance analysis, debugging and optimization - Programming tools for parallel and heterogeneous systems - Software engineering for parallel programs - Software for heterogeneous architectures - Software productivity for parallel programming - Synchronization and concurrency control

Papers should report on original research relevant to parallel programming and should contain enough background materials to make them accessible to the entire parallel programming research community. Papers describing experience should indicate how they illustrate general principles or lead to new insights. PPoPP submissions will be evaluated based on their technical merit and accessibility. Submissions should clearly motivate the importance of the problem being addressed, compare to the existing body of work on the topic, and explicitly and precisely state the paper’s key contributions and results towards addressing the problem. Submissions should strive to be accessible both to a broad audience and to experts in the area. Authors of papers that do not pass the first round of reviewing will receive a notification so that they can start working as early as possible on revising their papers and resubmitting them to other conferences or journals.


Paper Submission:

Conference submission site: https://ppopp21.hotcrp.com.

All submissions must be made electronically through the conference web site and include an abstract (100–400 words), author contact information, the full list of authors and their affiliations. Full paper submissions must be in PDF formatted printable on both A4 and US letter size paper.

All papers must be prepared in ACM Conference Format using the 2-column acmart format: use the SIGPLAN proceedings template acmart-sigplanproc-template.tex for Latex,and interim-layout.docx for Word. You may also want to consult the official ACM information on the Master Article Template and related tools. Important note: The Word template (interim-layout.docx) on the ACM website uses 9pt font; you need to increase it to 10pt.

Papers should contain a maximum of 10 pages of text (in a typeface no smaller than 10 point) or figures, NOT INCLUDING references. There is no page limit for references and they must include the name of all authors (not {et al.}). Appendices are not allowed, but the authors may submit supplementary material, such as proofs or source code; all supplementary material must be in PDF format. Looking at supplementary material is at the discretion of the reviewers.

Submission is double blind and authors will need to identify any potential conflicts of interest with PC and Extended Review Committee members, as defined here: http://www.sigplan.org/Resources/Policies/Review/ (ACM SIGPLAN policy).

PPoPP 2021 will employ a lightweight double-blind reviewing process. To facilitate this process, submissions should not reveal the identity of the authors in any way. Authors should leave out author names and affiliations from the body of their submission and from the supplementary material. They should also ensure that any references to authors’ own related work should be in the third person (e.g., not “We build on our previous work …” but rather “We build on the work of …”). The purpose of this process is to help the PC and external reviewers come to an initial judgment about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult. In particular, important background references should not be omitted or anonymized. In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For instance, authors may post drafts of their papers on the web or give talks on their research ideas. Authors with further questions on double-blind reviewing are encouraged to contact the Program Chair by email.

Submissions should be in PDF and printable on both US Letter and A4 paper. Papers may be resubmitted to the submission site multiple times up until the deadline, but the last version submitted before the deadline will be the version reviewed. Papers that exceed the length requirement, that deviate from the expected format, or that are submitted late will be rejected.

All submissions that are not accepted for regular presentations will be automatically considered for posters. Two-page summaries of accepted posters will be included in the conference proceedings

To allow reproducibility, we encourage authors of accepted papers to submit their papers for Artifact Evaluation (AE). The AE process begins after the acceptance notification, and is run by a separate committee whose task is to assess how the artifacts support the work described in the papers. Artifact evaluation is voluntary and will not affect paper acceptance, but will be taken into consideration when selecting papers for awards. Papers that go through the AE process successfully will receive one or several of the ACM reproducibility badges, printed on the papers themselves. More information will be posted on AE website.

Deadlines expire at midnight anywhere on earth.


Publication Date:

The titles of all accepted papers are typically announced shortly after the author notification date (late November 2020). Note, however, that this is not the official publication date. The official publication date is the date the proceedings are made available in the ACM Digital Library. ACM will make the proceedings available via the Digital Library for one month, up to 2 weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work.