We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Magic Castle: Terraforming the Cloud for HPC

Formale Metadaten

Titel
Magic Castle: Terraforming the Cloud for HPC
Serientitel
Anzahl der Teile
490
Autor
Lizenz
CC-Namensnennung 2.0 Belgien:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Compute Canada provides HPC infrastructures and support to every academic research institution in Canada. In recent years, Compute Canada has started distributing research software to its HPC clusters using with CERN software distribution service, CVMFS. This opened the possibility for accessing the software from almost any location and therefore allow the replication of the Compute Canada experience outside of its physical infrastructure. From these new possibilities emerged an open-source software project named Magic Castle, which aims to recreate the Compute Canada user experience in public clouds. Magic Castle uses the open-source software Terraform and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. The infrastructure definition is packaged as a Terraform module that users can customize as they require. Once their cluster is deployed, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software compiled by experts with EasyBuild. Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH. Compute Canada staff has been using this software to deploy ephemeral clusters for training purposes every other week for the past two years. Magic Castle is also gaining in popularity with HPC cluster users for development, testing, and continuous integration. In this talk, we will give a live demonstration of the creation of a cluster. We will present the architecture of Magic Castle, explain infrastructure and provisioning design, and present use cases. We will conclude by describing some of the challenges experienced while developing this novel usage of Terraform.