BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:Europe/Stockholm
X-LIC-LOCATION:Europe/Stockholm
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20190719T085743Z
LOCATION:HG E 3
DTSTART;TZID=Europe/Stockholm:20190612T133000
DTEND;TZID=Europe/Stockholm:20190612T140000
UID:submissions.pasc-conference.org_PASC19_sess126_msa263@linklings.com
SUMMARY:Recent Trends in Distributed Training - Sparsified, Compressed and
  Local SGD
DESCRIPTION:Minisymposium\nComputer Science and Applied Mathematics, Emerg
 ing Application Domains, Physics, Life Sciences\n\nRecent Trends in Distri
 buted Training - Sparsified, Compressed and Local SGD\n\nStich, Karimiredd
 y, Cordonnier, Rebjock, Patel...\n\nWe discuss recent techniques to improv
 e distributed training of machine learning models, which are also of inter
 est on HPC systems. First, SGD with sparsified or compressed gradients, su
 ch as signSGD, are key techniques offering orders-of-magnitude improvement
 s in scalability and communication requirements. We prove that such method
 s convergences at essentially the same rate as vanilla SGD, if error-feedb
 ack is applied [1,2]. In the second part of the presentation, we will disc
 uss variants of local SGD, which perform several update steps on a local m
 odel before communicating to other nodes, as opposed to classical mini-bat
 ch SGD. The scheme and its hierarchical extension offer improved overall p
 erformance and adaptivity to the underlying system resources [3].<br />Ref
 erences:<br />[1] Error Feedback Fixes SignSGD and other Gradient Compress
 ion Schemes, Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich
 , Martin Jaggi, https://arxiv.org/abs/1901.09847<br />[2] Sparsified SGD w
 ith Memory, NIPS 2018, Sebastian U. Stich, Jean-Baptiste Cordonnier, Marti
 n Jaggi. https://arxiv.org/abs/1809.07599<br />[3] Don't Use Large Mini-Ba
 tches, Use Local SGD, Tao Lin, Sebastian U. Stich, Martin Jaggi, https://a
 rxiv.org/abs/1808.07217
END:VEVENT
END:VCALENDAR