Matrix Algorithms at Scale - Alchemist and Spark - MPI Gap

Matrix Algorithms at Scale: Randomization and using Alchemist to bridge the Spark-MPI gap

Presented By Michael Mahoney, PhD
Michael Mahoney, PhD
Michael Mahoney, PhD
Statistics Professor at UC Berkeley

Michael Mahoney is at the University of California at Berkeley in the Department of Statistics and at the International Computer Science Institute (ICSI). He works on algorithmic and statistical aspects of modern large-scale data analysis. Much of his recent research has focused on large-scale machine learning, including randomized matrix algorithms and randomized numerical linear algebra, geometric network analysis tools for structure extraction in large informatics graphs, scalable implicit regularization methods, and applications in genetics, astronomy, medical imaging, social network analysis, and internet data analysis. He received him PhD from Yale University with a dissertation in computational statistical mechanics, and he has worked and taught at Yale University in the mathematics department, at Yahoo Research, and at Stanford University in the mathematics department. Among other things, he is on the national advisory committee of the Statistical and Applied Mathematical Sciences Institute (SAMSI), he was on the National Research Council's Committee on the Analysis of Massive Data, he runs the biennial MMDS Workshops on Algorithms for Modern Massive Data Sets, and he spent fall 2013 at UC Berkeley co-organizing the Simons Foundation's program on the Theoretical Foundations of Big Data Analysis.

Presentation Description

In this session, we will describe some of the underlying randomized linear algebra techniques. We'll describe Alchemist, a system for interfacing between Spark and existing MPI libraries that are designed to address this performance gap. We describe use cases from scientific data analysis that motivated the development of Alchemist and that benefit from this system. We'll also describe related work on communication-avoiding machine learning, optimization-based methods that can call these algorithms, and extending Alchemist to provide an ipython notebook <=> MPI interface.

Presentation Curriculum

Matrix Algorithms at Scale: Randomization and using Alchemist to bridge the Spark-MPI gap
30:29
Hide Content