The researches on machine learning systems are attracting a lot of efforts. We separate the existing prevailing ML systems into three categories: Stand-alone systems, PS-based systems and All-reduce-based systems. Hogwild! [
5] is a stand-alone machine learning system with the lock-free parallel model. HELIX [
6] optimizes the execution across iterations, including intelligently caching and reusing, or recomputing intermediates as appropriate. The stand-alone ML systems generally show poor scalability and parallelism. The existing representative PS-based systems, such as Petuum [
3], Parameter Server [
1], MXNet [
2], TensorFlow [
4], are all based on a memory implementation. FlexPS [
7] introduces a novel multi-stage abstraction to support flexible parallelism control, which can map machine learning tasks to a series of stages, and the parallelism for a stage can be set according to its workload. PS2 [
8] builds the Parameter Server on top of Spark [
9]. The pieces of literature mentioned above are orthogonal to our paper, which mainly focus on the optimizations of a memory-based implementation. All-Reduce is another famous distributed framework for machine learning. BlueConnect [
10] designs a parallelizable All-Reduce mode to make a trade-off between latency and bandwidth. However, systems like this are always multi-GPUs friendly architecture. Moreover, there are also some works to explore the adaptive adjustment among these architectures. Husky [
11] attempts to strike a better balance between high performance and low development cost. BytePS [
12] studies the load allocation problem of distributed machine learning under different heterogeneous GPU/CPU resources, which does not consider the impact of disk I/Os on load allocation. HybridGraph [
13] is also a disk-resident distributed system, however, it mainly focuses on distributed graph iterative tasks.