UberCloud Releases Multi-Cloud, Hybrid-Cloud HPC Application Platform (2020/11/06)

The way enterprises run High Performance Computing (HPC) applications has changed. With Cloud providers offering improved security, better cost/performance, and seemingly endless compute capacity, more enterprises are turning to Cloud for their HPC workloads.

However, many companies are finding that replicating an existing on-premise HPC architecture in the Cloud does not lead to the desired breakthrough improvements. With this in mind, from day one, the UberCloud HPC Application Platform has been built with cloud computing in mind, resulting in highly increased productivity of the HPC engineers, significantly improving IT security, reducing cloud costs and administrative overhead to a minimum, and maintaining full control for engineers and corporate IT over their HPC cloud environment. Today, we are announcing UberCloud’s next-generation HPC Application Platform.

Building blocks of the UberCloud Platform, including HPC, Cloud, Containers, and Kubernetes, have been previously discussed on HPCwire: Kubernetes, Containers and HPC, and Kubernetes and HPC Applications in Hybrid Cloud Environments.

Key Stakeholders when Driving HPC Cloud Adoption

When we started designing the UberCloud HPC Application platform we recognized that three major stakeholders are crucial for the overall success of a company’s HPC cloud journey: HPC engineers, Enterprise IT, and the HPC IT team.

HPC application engineers are the driving force behind innovation. To excel in (and enjoy) their job they require a frictionless, self-service user portal for allocating the computational resources when they are required. They don’t necessarily need to understand how compute nodes, GPUs, storage, or fast network interconnects have to be configured. They expect to be able to allocate and shutdown fully configured HPC application environments.

Enterprise IT demands the necessary software tools and pre-configured containerized HPC applications for creating fully automated, completely tested environments. These environments must be suited to interact with HPC applications and their special requirements for resources and license servers. The platform needs to be pluggable to modern IT environments and support technologies like CI/CD pipelines and Kubernetes orchestration.

The HPC IT team (often quite independent from Enterprise IT) requires a hybrid cloud strategy for enhancing their existing on-premise HPC infrastructure with cloud resources for bursting and hybrid cloud scenarios. This team demands control on software versions and puts emphasis on the entire engineering lifecycle, from design to manufacturing.

Introducing the UberCloud HPC Application Platform

The UberCloud HPC Application Platform aims at supporting each of the three major key stakeholders during their HPC cloud adoption journey. How is that achieved?

For the HPC application engineers UberCloud provides a self-service HPC user interface where they select their application(s) along with the hardware parameters they need. With a single click the fully automated UberCloud HPC Application Platform allocates the dedicated computing infrastructure, deploys the application, and configures access for the engineer for instant productivity. Similarly, the HPC application infrastructure can be resized at any given point in time to run distributed memory simulations, parameter studies, or a design of experiments. After work is done the application and the simulation platform can be safely shut down.

Enterprise IT operations often have their own way of managing cloud-based resources. Infrastructure as Code, GitOps, and DevOps are some of the paradigms found in those organizations. The UberCloud HPC Application Platform contains a management tool which can be integrated in any kind of automation or CI/CD pipeline tool chains. UberCloud’s application platform management tool takes care of all aspects of managing containerized HPC applications using Kubernetes based container orchestrators like GKE, AKS, and EKS.

HPC IT teams require integration points for allocating cloud resources and distributing HPC jobs between their on-premise HPC clusters and dynamically allocated cloud resources. The UberCloud HPC Cloud Dispatcher provides batch job interfaces for hybrid cloud, cloud bursting, and high-throughput computing. It relies on open standards through the whole application stack to provide stable integration interfaces.

Putting UberCloud’s HPC Application Platform into Practice

Our first customer that enjoyed the benefits of the UberCloud HPC Application Platform is FLSmidth, a Danish multinational engineering company providing global cement and mineral industries with factories, machinery, services and know-how. The Proof of Concept implementation at the end of last year has been recently summarized here, and the extended case study (including a description of the hybrid cloud architecture) is freely available through This e-mail address is being protected from spambots. You need JavaScript enabled to view it. .

UberCloud Webinar with Microsoft (2020/10/26)

It is never to late to consider moving HPC workload to the cloud. Registration for this weeks webinar is still open!

Check out Wolfgang's awesome video message about the key values of UberCloud's HPC Application Platform.

https://www.linkedin.com/feed/update/urn:li:activity:6725420905389944832/

Hybrid Cloud interactive HPC Applications on Kubernetes (2020-03-19)

We just published a follow up article about our experiences at UberCloud running HPC Engineering Applications on different Clouds and on-premises using Kubernetes as middleware-stack.

https://www.hpcwire.com/2020/03/19/kubernetes-and-hpc-applications-in-hybrid-cloud-environments-part-ii/

Univa Grid Engine 8.4.1 Released (2016-07-19)

Univa Grid Engine 8.4.1 was released by Univa’s engineering team last week. This is the first update for the major 8.4 release which brings in main fixes as well as new capabilities.

The major feature introduced in 8.4 is native Docker support. Native Docker support of Univa Grid Engine means that Docker enabled hosts are automatically discovered (without any configuration needed by the admin) and the installed images are forwarded to the global scheduler of Univa Grid Engine. The user just needs to select a image name and then its job script or binary is executed on an selected execution host within the automatically created Docker container. Since the Docker containers are not anymore child processes of UGE and therefore not directly supervised by Univa Grid Engine (for accounting and resource limitation for example) a new process is injected in the container which supervises the job within the container as Univa Grid Engine does it for jobs outside of a container. Working directories and job spooling directories are automatically mounted inside the container.

Univa Grid Engine 8.4.1 adds now capabilities to launch containers with all the parameters you know from Docker run with a new qsub parameter called -xd (to select ports or devices for example).

Univa Grid Engine is a proven and a highly scalable (thousands of compute nodes) solution not just for traditional compute jobs, now also for Docker containers, which comes with enterprise features like accounting, reporting, access control, fair usage of compute resources through a set of policies which controls job priorities, simplification of administration and job submission through job classes, APIs, and quotas on container usage.

Update of the Univa Grid Engine Vagrant integration (2016-06-15)

I just updated the Vagrant integration for Univa Grid Engine so that it allows you to create an Univa Grid Engine 8.4 test cluster with three hosts just by calling vagrant up.

Just place the new Univa Grid Engine 8.4 Demo Packages ge-8.4.0-demo-bin-lx-amd64.tar.gz and ge-8.4.0-demo-common.tar.gz in the directory. You can download them from Univa

The referenced CentOS box was required to be updated to version 7.2, so if it is not yet cached locally by Vagrant a bigger download will follow. Also Docker is now going to be installed (and a config change needs to be done) on the hosts so that you easily can try the Docker integration by requesting images for your jobs. Also a simple script which downloads, compiles, and installs runc is in the directory.

For any kind of problems please create an issue at the github repository directly.

After the intallation process you can run your first Docker job in Univa Grid Engine:

• vagrant up --> this might take a while
• vagrant ssh --> you are now on the master node
• qhost --> just showing if the cluster is up and running
• docker pull golang:latest --> we need an image for our container
• cd /nfs
• warning: mounting non-fitting /etc in Docker could become dirty
• qsub -b y -l docker,docker_images="*golang:latest*" -wd /nfs -xdv /nfs:/nfs,/etc:/etc -S /bin/sh /bin/sleep 123
• docker ps --> it is launched locally because our image is only there / watch the created container
• qdel --> remove the job and along with it the container
• docker ps --> see that the container is removed
• launch now an interactive job
• qrsh -l docker,docker_images="*golang:latest*" -xdv /nfs:/nfs,/etc:/etc
• you are now in the container
• ps -ef --> see PID namespace isolation within the container