Context-Aware Scheduling for Apache Hadoop over Pervasive Environments

Guilherme Weigert Cassales (1) , Andrea Schwertner Charão (1) , Manuele Kirsch Pinheiro (2) , Carine Souveyet (2) , Luiz Angelo Steffenel (3)

1 LSC - Laboratório de Sistemas de Computação
2 CRI - Centre de Recherche en Informatique de Paris 1
3 CRESTIC - Centre de Recherche en Sciences et Technologies de l'Information et de la Communication - EA 3804

This article proposes to improve Apache Hadoop scheduling through the usage of context-awareness. Apache Hadoop is the most popular implementation of the MapReduce paradigm for distributed computing, but its design doesn't adapt automatically to computing nodes' context and capabilities. By introducing context-awareness into Hadoop, we intent to dynamically adapt its scheduling to the execution environment. This is a necessary feature in the context of pervasive grids, which are heterogeneous, dynamic and shared environments. The solution has been incorporated into Hadoop and evaluated through controlled experiments. The experiments demonstrate that context-awareness provides comparative performance gains, especially when part of the resources disappear during execution.

Mots clés: Context-Awareness; Map Reduce; Apache Hadoop; Task Scheduling; Pervasive Grid; Big Data

Domaines: Informatique [cs]; Informatique ubiquitaire; Informatique mobile; Calcul parallèle, distribué et partagé [cs.DC]

Guilherme Weigert Cassales, Andrea Schwertner Charão, Manuele Kirsch Pinheiro, Carine Souveyet, Luiz Angelo Steffenel. Context-Aware Scheduling for Apache Hadoop over Pervasive Environments. 6th International Conference on Ambient Systems, Networks and Technologies (ANT 2015), Jun 2015, Londres, United Kingdom. pp.202-209, ⟨10.1016/j.procs.2015.05.058⟩. ⟨hal-01196986⟩