POSTER: A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression
DNNs are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. When training a DNN model, the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. Traditional memory saving techniques such as data recomputation and migration either suffers from a high performance overhead or is constrained by specific interconnect technology and limited bandwidth. In this paper, we propose a novel memory-driven high performance CNN training framework that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger neural networks. Specifically, we provide theoretical analysis and then propose an improved lossy compressor and an adaptive scheme to dynamically configure the lossy compression error-bound and adjust the training batch size to further utilize the saved memory space for additional speedup. We evaluate our design against state-of-the-art solutions with four widely-adopted CNNs and the ImangeNet dataset. Results demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5$\times$ and 1.8$\times$ over the baseline training and state-of-the-art framework with compression, respectively, with little or no accuracy loss. The full paper can be referred to at https://arxiv.org/abs/2011.09017.
Tue 2 MarDisplayed time zone: Eastern Time (US & Canada) change
13:30 - 14:30 | |||
13:30 6mTalk | POSTER: In-situ Workflow Auto-tuning through Combining Component Models Main Conference Tong Shu Southern Illinois University Carbondale, Yanfei Guo Argonne National Laboratory, Justin Wozniak Argonne National Laboratory, Xiaoning Ding New Jersey Institute of Technology, Ian Foster Argonne Nat Lab and U.Chicago, Tahsin Kurc Stony Brook University Link to publication | ||
13:36 6mTalk | POSTER: Simplifying Low-Level GPU Programming with GAS Main Conference Da Yan Hong Kong University of Science and Technology, Wei Wang Hong Kong University of Science and Technology, Xiaowen Chu Hong Kong Baptist University Link to publication | ||
13:42 6mTalk | POSTER: Corder: Cache-Aware Reordering for Optimizing Graph Analytics Main Conference YuAng Chen The Chinese University of Hong Kong, Shenzhen, Yeh-Ching Chung The Chinese University of Hong Kong, Shenzhen Link to publication | ||
13:48 6mTalk | POSTER: DFOGraph: An I/O- and Communication-Efficient System for Distributed Fully-out-of-Core Graph Processing Main Conference Jiping Yu Tsinghua University, Wei Qin Tsinghua University, Xiaowei Zhu Tsinghua University, Zhenbo Sun Tsinghua University, Jianqiang Huang Tsinghua University, Xiaohan Li Tsinghua University, Wenguang Chen Tsinghua University Link to publication | ||
13:54 6mTalk | POSTER: An Efficient Uncertain Graph Processing Framework for Heterogeneous Architectures Main Conference Heng Zhang Institute of Software, Chinese Academy of Sciences; University of Sydney, Lingda Li Brookhaven National Laboratory, Donglin Zhuang University of Sydney, Rui Liu University of Chicago, Shuang Song Facebook Inc., Dingwen Tao Washington State University, Yanjun Wu Institute of Software, Chinese Academy of Sciences, Shuaiwen Leon Song University of Sydney Link to publication | ||
14:00 6mTalk | POSTER: Dynamic Scaling for Low-Precision Learning Main Conference Ruobing Han Peking University, Min Si Argonne National Laboratory, James W. Demmel UC Berkeley, Yang You UC Berkeley Link to publication | ||
14:06 6mTalk | POSTER: Exploring Deep Reuse in Winograd CNN Inference Main Conference Ruofan Wu Renmin University of China, Feng Zhang Renmin University of China, Zhen Zheng Alibaba Group, Xiaoyong Du Renmin University of China, Xipeng Shen North Carolina State University Link to publication | ||
14:12 6mTalk | POSTER: A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression Main Conference Sian Jin Washington State University, Guanpeng Li University of Iowa, Shuaiwen Leon Song University of Sydney, Dingwen Tao Washington State University Link to publication | ||
14:18 6mTalk | POSTER: FFT Blitz: The Tensor Cores Strike Back Main Conference Sultan Durrani University of Illinois at Urbana-Champaign, Muhammad Saad Chughtai Georgia Institute of Technology, Abdul Dakkak University of Illinois at Urbana-Champaign, Wen-mei Hwu University of Illinois at Urbana-Champaign, Lawrence Rauchwerger UIUC Link to publication | ||
14:24 6mBreak | Break Main Conference |