Hybrid SORS/WiCS: Convergence of HPC, Big Data and Machine Learning Applications and Containerized Infrastructures

Download here the slides.

Abstract: The convergence of HPC, BD and ML in the computing continuum is being pursued in earnest across the academic and industry. We envision virtualization and containerization technologies can be the basis for the convergence, because they reside as bridges between applications and infrastructures and provide well-known advantages, such as the encapsulation of specific software environments, which allows for customization, portability, and reproducibility; the isolation of users from the underlying system and from other users, which allows for security and fault protection; and the agile and fine-grain resource allocation and balancing, which allows for efficient cluster utilization and failure recovery. However, challenges remain for this convergence at containerization level due to the diversity of applications and hardware heterogeneity. In this talk I will present previous and ongoing work, (1) Enable deployments and understand the performance of HPC, BD and ML applications using containers. (2) Provide an autonomic management platform for containerized HPC, BD, and ML applications. (3) Optimize container management and scheduling for containerized HPC, BD, and ML applications.

Short bio: Peini Liu received her B.S. degree in Internet of Things Engineering at Central South University, China, in 2016, and her M.S. degree in Software Engineering at National University of Defense Technology, China, in 2018. She is currently a doctoral student in Computer Architecture at Universitat Politècnica de Catalunya (UPC) and Barcelona Supercomputing Center (BSC). Her research interests include virtualization/containerization technologies, cloud-native, resource management and the convergence of HPC, Big Data and AI. Her research has been published in high-quality conferences and journals and she obtained FI grants from Generalitat de Catalunya and WISC primer scholarship from IEEE Service. Peini was funded by the BSC Mobility Grant the last year to conduct a two-month research stay in the Network and Distributed System Group at University of Oslo.