Since 2005, the single-threaded CPU speed has stopped improving significantly; it is the numbers of cores in each machine that continue to arise. On the other hand, most of our algorithms are still single-threaded, and because so, their running time will stay about the same in the future. To develop faster algorithms, especially for those large-scale problems, it is inevitable to consider parallel computing.

In parallel computing, multiple agents (e.g., CPU cores) collaboratively solve a problem by concurrently solving their simpler subproblems. For most, the subproblems depend on each other, so the agents must regularly exchange information. In asynchronous computing, each agent can compute with the information it has, even if the latest information from other agents has not arrived. Asynchronism is extremely important to the efficiency and resilience of parallel computing. Without asynchronism, all cores have to wait for the arrival of latest information, so the speed of parallel computing is dictated by the slowest core, the most difficult subproblem, and the longest communication delay. Without asynchronism, the entire parallel computing must stop when an agent (or a network link) fails and awaits a fix, and such failures will happen more often as the system gets larger. 

Today, most algorithms are still singled-threaded, and most of the already-parallelized algorithms are synchronous. In spite of both mathematical and coding challenges, we report recently established convergence and impressive numerical results for a set of fixed point problems and optimization problems that arise in machine learning, image processing, portfolio optimization, second-order cone programming, and beyond.