Write a Blog >>
PPoPP 2021
Sat 27 February - Wed 3 March 2021
Tue 2 Mar 2021 14:12 - 14:18 - Session 7. Posters 2 Chair(s): Todd Mytkowicz

DNNs are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. When training a DNN model, the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. Traditional memory saving techniques such as data recomputation and migration either suffers from a high performance overhead or is constrained by specific interconnect technology and limited bandwidth. In this paper, we propose a novel memory-driven high performance CNN training framework that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger neural networks. Specifically, we provide theoretical analysis and then propose an improved lossy compressor and an adaptive scheme to dynamically configure the lossy compression error-bound and adjust the training batch size to further utilize the saved memory space for additional speedup. We evaluate our design against state-of-the-art solutions with four widely-adopted CNNs and the ImangeNet dataset. Results demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5$\times$ and 1.8$\times$ over the baseline training and state-of-the-art framework with compression, respectively, with little or no accuracy loss. The full paper can be referred to at https://arxiv.org/abs/2011.09017.

Tue 2 Mar

Displayed time zone: Eastern Time (US & Canada) change

13:30 - 14:30
Session 7. Posters 2Main Conference
Chair(s): Todd Mytkowicz Microsoft Research
13:30
6m
Talk
POSTER: In-situ Workflow Auto-tuning through Combining Component Models
Main Conference
Tong Shu Southern Illinois University Carbondale, Yanfei Guo Argonne National Laboratory, Justin Wozniak Argonne National Laboratory, Xiaoning Ding New Jersey Institute of Technology, Ian Foster Argonne Nat Lab and U.Chicago, Tahsin Kurc Stony Brook University
Link to publication
13:36
6m
Talk
POSTER: Simplifying Low-Level GPU Programming with GAS
Main Conference
Da Yan Hong Kong University of Science and Technology, Wei Wang Hong Kong University of Science and Technology, Xiaowen Chu Hong Kong Baptist University
Link to publication
13:42
6m
Talk
POSTER: Corder: Cache-Aware Reordering for Optimizing Graph Analytics
Main Conference
YuAng Chen The Chinese University of Hong Kong, Shenzhen, Yeh-Ching Chung The Chinese University of Hong Kong, Shenzhen
Link to publication
13:48
6m
Talk
POSTER: DFOGraph: An I/O- and Communication-Efficient System for Distributed Fully-out-of-Core Graph Processing
Main Conference
Jiping Yu Tsinghua University, Wei Qin Tsinghua University, Xiaowei Zhu Tsinghua University, Zhenbo Sun Tsinghua University, Jianqiang Huang Tsinghua University, Xiaohan Li Tsinghua University, Wenguang Chen Tsinghua University
Link to publication
13:54
6m
Talk
POSTER: An Efficient Uncertain Graph Processing Framework for Heterogeneous Architectures
Main Conference
Heng Zhang Institute of Software, Chinese Academy of Sciences; University of Sydney, Lingda Li Brookhaven National Laboratory, Donglin Zhuang University of Sydney, Rui Liu University of Chicago, Shuang Song Facebook Inc., Dingwen Tao Washington State University, Yanjun Wu Institute of Software, Chinese Academy of Sciences, Shuaiwen Leon Song University of Sydney
Link to publication
14:00
6m
Talk
POSTER: Dynamic Scaling for Low-Precision Learning
Main Conference
Ruobing Han Peking University, Min Si Argonne National Laboratory, James W. Demmel UC Berkeley, Yang You UC Berkeley
Link to publication
14:06
6m
Talk
POSTER: Exploring Deep Reuse in Winograd CNN Inference
Main Conference
Ruofan Wu Renmin University of China, Feng Zhang Renmin University of China, Zhen Zheng Alibaba Group, Xiaoyong Du Renmin University of China, Xipeng Shen North Carolina State University
Link to publication
14:12
6m
Talk
POSTER: A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression
Main Conference
Sian Jin Washington State University, Guanpeng Li University of Iowa, Shuaiwen Leon Song University of Sydney, Dingwen Tao Washington State University
Link to publication
14:18
6m
Talk
POSTER: FFT Blitz: The Tensor Cores Strike Back
Main Conference
Sultan Durrani University of Illinois at Urbana-Champaign, Muhammad Saad Chughtai Georgia Institute of Technology, Abdul Dakkak University of Illinois at Urbana-Champaign, Wen-mei Hwu University of Illinois at Urbana-Champaign, Lawrence Rauchwerger UIUC
Link to publication
14:24
6m
Break
Break
Main Conference