Write a Blog >>
PPoPP 2021
Sat 27 February - Wed 3 March 2021
Tue 2 Mar 2021 14:12 - 14:18 - Session 7. Posters 2 Chair(s): Todd Mytkowicz

DNNs are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. When training a DNN model, the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. Traditional memory saving techniques such as data recomputation and migration either suffers from a high performance overhead or is constrained by specific interconnect technology and limited bandwidth. In this paper, we propose a novel memory-driven high performance CNN training framework that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger neural networks. Specifically, we provide theoretical analysis and then propose an improved lossy compressor and an adaptive scheme to dynamically configure the lossy compression error-bound and adjust the training batch size to further utilize the saved memory space for additional speedup. We evaluate our design against state-of-the-art solutions with four widely-adopted CNNs and the ImangeNet dataset. Results demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5$\times$ and 1.8$\times$ over the baseline training and state-of-the-art framework with compression, respectively, with little or no accuracy loss. The full paper can be referred to at https://arxiv.org/abs/2011.09017.

Conference Day
Tue 2 Mar

Displayed time zone: Eastern Time (US & Canada) change

13:30 - 14:30
Session 7. Posters 2Main Conference
Chair(s): Todd MytkowiczMicrosoft Research
13:30
6m
Talk
POSTER: In-situ Workflow Auto-tuning through Combining Component Models
Main Conference
Tong ShuSouthern Illinois University Carbondale, Yanfei GuoArgonne National Laboratory, Justin WozniakArgonne National Laboratory, Xiaoning DingNew Jersey Institute of Technology, Ian FosterArgonne Nat Lab and U.Chicago, Tahsin KurcStony Brook University
Link to publication
13:36
6m
Talk
POSTER: Simplifying Low-Level GPU Programming with GAS
Main Conference
Da YanHong Kong University of Science and Technology, Wei WangHong Kong University of Science and Technology, Xiaowen ChuHong Kong Baptist University
Link to publication
13:42
6m
Talk
POSTER: Corder: Cache-Aware Reordering for Optimizing Graph Analytics
Main Conference
YuAng ChenThe Chinese University of Hong Kong, Shenzhen, Yeh-Ching ChungThe Chinese University of Hong Kong, Shenzhen
Link to publication
13:48
6m
Talk
POSTER: DFOGraph: An I/O- and Communication-Efficient System for Distributed Fully-out-of-Core Graph Processing
Main Conference
Jiping YuTsinghua University, Wei QinTsinghua University, Xiaowei ZhuTsinghua University, Zhenbo SunTsinghua University, Jianqiang HuangTsinghua University, Xiaohan LiTsinghua University, Wenguang ChenTsinghua University
Link to publication
13:54
6m
Talk
POSTER: An Efficient Uncertain Graph Processing Framework for Heterogeneous Architectures
Main Conference
Heng ZhangInstitute of Software, Chinese Academy of Sciences; University of Sydney, Lingda LiBrookhaven National Laboratory, Donglin ZhuangUniversity of Sydney, Rui LiuUniversity of Chicago, Shuang SongFacebook Inc., Dingwen TaoWashington State University, Yanjun WuInstitute of Software, Chinese Academy of Sciences, Shuaiwen Leon SongUniversity of Sydney
Link to publication
14:00
6m
Talk
POSTER: Dynamic Scaling for Low-Precision Learning
Main Conference
Ruobing HanPeking University, Min SiArgonne National Laboratory, James W. DemmelUC Berkeley, Yang YouUC Berkeley
Link to publication
14:06
6m
Talk
POSTER: Exploring Deep Reuse in Winograd CNN Inference
Main Conference
Ruofan WuRenmin University of China, Feng ZhangRenmin University of China, Zhen ZhengAlibaba Group, Xiaoyong DuRenmin University of China, Xipeng ShenNorth Carolina State University
Link to publication
14:12
6m
Talk
POSTER: A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression
Main Conference
Sian JinWashington State University, Guanpeng LiUniversity of Iowa, Shuaiwen Leon SongUniversity of Sydney, Dingwen TaoWashington State University
Link to publication
14:18
6m
Talk
POSTER: FFT Blitz: The Tensor Cores Strike Back
Main Conference
Sultan DurraniUniversity of Illinois at Urbana-Champaign, Muhammad Saad ChughtaiGeorgia Institute of Technology, Abdul DakkakUniversity of Illinois at Urbana-Champaign, Wen-mei HwuUniversity of Illinois at Urbana-Champaign, Lawrence RauchwergerUIUC
Link to publication
14:24
6m
Break
Break
Main Conference