Creating Kubernetes based UberCloud HPC Application Clusters using Containers (2020-12-16)

This article was originally published at UberCloud's Blog

UberCloud provides all necessary automation for integrating cloud based self-service HPC application portals in enterprise environments. Due to the significant differences inside the IT landscape of large organizations we are continuously challenged providing the necessary flexibility within our own solution stack. Hence we continuously evaluate newly adopted tools and technologies about the readiness to interact with UberCloud’s technology.

Recent adoption of Kubernetes not just for enterprise workloads but for all sorts of applications, be it on the edge, AI, or for HPC, has strong focus. We created hundreds of Kubernetes clusters on various cloud providers hosting HPC applications like Ansys, Comsol, OpenFoam and many more. We can deploy fully configured HPC clusters which are dedicated to an engineer on GKE or AKS within minutes. We can also use EKS but the deployment time of an EKS cluster is at this point in time significantly slower as on the other platforms (around 3x times). While GKE is excellent and has been my favorite service (due to its deployment speed and its good APIs), AKS has begun in the last months to get really strong. Many features which are relevant for us (like using spot instances and placement groups) and its speed in terms of AKS cluster allocation time (now even almost one minute faster as GKE - 3:30 min. from 0 to a fully configured AKS cluster) have been implement on Azure. Great!

When managing HPC applications in dedicated Kubernetes clusters one challenge remains: How to manage fleets of clusters distributed across multiple clouds? At UberCloud we are building simple tools which takes HPC application start requests and turns it into a fully automated cluster creation and configuration job. One very popular way is to put this logic behind self-service portals where the user selects an application he/she want to use. One other way is creating those HPC applications based on events in workflows, CI/CD and gitops pipelines. Use cases are automated application testing, running automated compute tasks, cloud bursting, infrastructure as code integrations, and more. To support those tasks we’ve developed a container which turns an application and infrastructure description into a managed Kubernetes cluster independent of where the job runs and on which cloud provider and regions the cluster is created.

Due to the flexibility of containers UberCloud’s cluster creation container can be used in almost all modern environments which support containers. We are using it as a Kubernetes job and as CI/CD tasks. When the job is finished, the engineer has access to a fully configured HPC desktop including a HPC cluster attached.

Another integration we just tested is Argo. Argo is a popular workflow engine targeted and working on top of Kubernetes. We have a test installation running on GKE. As the UberCloud HPC cluster creation is fully wrapped inside a container running a single binary the configuration required to integrate it in a Argo workflow is very minimal.

After the workflow (task) is finished, the engineer get’s automatically access to the freshly created remote visualization application running on a newly allocated AKS cluster spanning two node pools having GUI based remote Linux desktop access setup.

The overall AKS cluster creation, configuration, and deployment of our services and HPC application containers just took a couple of minutes. A solution targeted for IT organizations challenged by the task of rolling out HPC applications for their engineers but required to work with modern cloud based technologies.

UberCloud Releases Multi-Cloud, Hybrid-Cloud HPC Application Platform (2020/11/06)

The way enterprises run High Performance Computing (HPC) applications has changed. With Cloud providers offering improved security, better cost/performance, and seemingly endless compute capacity, more enterprises are turning to Cloud for their HPC workloads.

However, many companies are finding that replicating an existing on-premise HPC architecture in the Cloud does not lead to the desired breakthrough improvements. With this in mind, from day one, the UberCloud HPC Application Platform has been built with cloud computing in mind, resulting in highly increased productivity of the HPC engineers, significantly improving IT security, reducing cloud costs and administrative overhead to a minimum, and maintaining full control for engineers and corporate IT over their HPC cloud environment. Today, we are announcing UberCloud’s next-generation HPC Application Platform.

Building blocks of the UberCloud Platform, including HPC, Cloud, Containers, and Kubernetes, have been previously discussed on HPCwire: Kubernetes, Containers and HPC, and Kubernetes and HPC Applications in Hybrid Cloud Environments.

Key Stakeholders when Driving HPC Cloud Adoption

When we started designing the UberCloud HPC Application platform we recognized that three major stakeholders are crucial for the overall success of a company’s HPC cloud journey: HPC engineers, Enterprise IT, and the HPC IT team.

HPC application engineers are the driving force behind innovation. To excel in (and enjoy) their job they require a frictionless, self-service user portal for allocating the computational resources when they are required. They don’t necessarily need to understand how compute nodes, GPUs, storage, or fast network interconnects have to be configured. They expect to be able to allocate and shutdown fully configured HPC application environments.

Enterprise IT demands the necessary software tools and pre-configured containerized HPC applications for creating fully automated, completely tested environments. These environments must be suited to interact with HPC applications and their special requirements for resources and license servers. The platform needs to be pluggable to modern IT environments and support technologies like CI/CD pipelines and Kubernetes orchestration.

The HPC IT team (often quite independent from Enterprise IT) requires a hybrid cloud strategy for enhancing their existing on-premise HPC infrastructure with cloud resources for bursting and hybrid cloud scenarios. This team demands control on software versions and puts emphasis on the entire engineering lifecycle, from design to manufacturing.

Introducing the UberCloud HPC Application Platform

The UberCloud HPC Application Platform aims at supporting each of the three major key stakeholders during their HPC cloud adoption journey. How is that achieved?

For the HPC application engineers UberCloud provides a self-service HPC user interface where they select their application(s) along with the hardware parameters they need. With a single click the fully automated UberCloud HPC Application Platform allocates the dedicated computing infrastructure, deploys the application, and configures access for the engineer for instant productivity. Similarly, the HPC application infrastructure can be resized at any given point in time to run distributed memory simulations, parameter studies, or a design of experiments. After work is done the application and the simulation platform can be safely shut down.

Enterprise IT operations often have their own way of managing cloud-based resources. Infrastructure as Code, GitOps, and DevOps are some of the paradigms found in those organizations. The UberCloud HPC Application Platform contains a management tool which can be integrated in any kind of automation or CI/CD pipeline tool chains. UberCloud’s application platform management tool takes care of all aspects of managing containerized HPC applications using Kubernetes based container orchestrators like GKE, AKS, and EKS.

HPC IT teams require integration points for allocating cloud resources and distributing HPC jobs between their on-premise HPC clusters and dynamically allocated cloud resources. The UberCloud HPC Cloud Dispatcher provides batch job interfaces for hybrid cloud, cloud bursting, and high-throughput computing. It relies on open standards through the whole application stack to provide stable integration interfaces.

Putting UberCloud’s HPC Application Platform into Practice

Our first customer that enjoyed the benefits of the UberCloud HPC Application Platform is FLSmidth, a Danish multinational engineering company providing global cement and mineral industries with factories, machinery, services and know-how. The Proof of Concept implementation at the end of last year has been recently summarized here, and the extended case study (including a description of the hybrid cloud architecture) is freely available through This e-mail address is being protected from spambots. You need JavaScript enabled to view it. .

UberCloud Webinar with Microsoft (2020/10/26)

It is never to late to consider moving HPC workload to the cloud. Registration for this weeks webinar is still open!

Check out Wolfgang's awesome video message about the key values of UberCloud's HPC Application Platform.

https://www.linkedin.com/feed/update/urn:li:activity:6725420905389944832/

Kubernetes Topology Manager vs. HPC Workload Manager (2020-04-15)

Looks like another gap between Kubernetes and traditional HPC scheduler is solved: Kubernetes 1.18 enables a new feature called Topology Manager.

That’s very interesting for HPC workload since the process placements on NUMA machines have a high impact for compute, IO, and memory intensive applications. The reasons are well known:

• access latency differs - well, it is NUMA (non-uniform memory access)!
• when processes move between cores and chips, caches needs to be refilled (getting cold)
• not just memory, also device access (like GPUs, network etc.) is non-uniform
• device access needs to be managed (process which runs on a core needs should access a GPU which is close to the CPU core)
• when having a distributed memory job all parts (ranks) should run with the same speed

In order to get the best performance out of NUMA hardware, Sun Grid Engine introduced around eleven years ago a feature which lets processes get pinned to cores. There are many articles in my blog about that. At Univa we heavily improved it and created a new resource type called RSMAP which combines core binding with resource selection, i.e. when requesting a GPU or something else, the process gets automatically bound to cores or NUMA nodes which are associated with the resource (that’s called RSMAP topology masks). Additionally the decision logic was moved away from the execution host (execution daemon) to the central scheduler as it turned out to be necessary for making global decisions for a job and not wasting time and resources for continued rescheduling when local resources don’t fit.

Back to Kubernetes.

The new Kubernetes Topology Manager is a component of Kubelet which also takes care about these NUMA optimizations. That it is integrated in Kubelet, which runs locally on the target host is the first thing to note.

As Topology Manager provides a host local logic (like in Grid Engine at the beginning) it can: a.) make only host local decisions and b.) lead to wasted resources when pods needs to be re-scheduled many times to find a host which fits (if there is any). That’s also described in the release notes as a known limitation.

How does the Topology Manager work?

The Topology Manager provides following allocation policies: none, best-effort, restricted, and single-numa-node. These are kubelet flags to be set.

The actual policy which is applied to the pod depends on the kubelet flag but also on the QoS class of the pod itself. The QoS class of the pod depends on the resource setting of the pod description. If cpu and memory are requested in the same way within the limits and requests section then the QoS class is Guaranteed. Other QoS classes are BestEffort and Burstable.

The kubelet calls so called Hint Providers for each container of the pod and then aligns them with the selected policy, like checking if it works well with single-numa-node policy. When set to restricted or single-numa-node it may terminate the pod when no placement if found so that the pod can be handled by external means (like rescheduling the pod). But I’m wondering if that will work when having side-car like setups inside the pod.

The sources of the Topology Manager are well arranged here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/topologymanager

In order to summarize. A bit complicated since I can’t really find the decision logic of the hint provider documented and so the source code will be the source of truth. Also, since it is a host local decision it will be pain for distributed memory jobs (like MPI jobs). Maybe they move the code one day to the scheduler, like it was done in Grid Engine. It is really good that Kubernetes takes now also care about those very specific requirements. So, it is still on the right way to catch up with all required features for traditional HPC applications. Great!

At UberCloud we closely track Kubernetes as platform for HPC workload and are really happy to see developments like the Topology Manager.

Hybrid Cloud interactive HPC Applications on Kubernetes (2020-03-19)

We just published a follow up article about our experiences at UberCloud running HPC Engineering Applications on different Clouds and on-premises using Kubernetes as middleware-stack.

https://www.hpcwire.com/2020/03/19/kubernetes-and-hpc-applications-in-hybrid-cloud-environments-part-ii/

Paper about Virtual Grid Engine (2019/12/12)

Kubernetes, Containers, and HPC at UberCloud (2019-09-20)

Long time no updates on my blog. Need to change that…

@UberCloud we published a whitepaper about our experiences running HPC workload on Kubernetes. You can download it here.

A shortened version appeared today as lead-article at HPCWire.

qsub for Kubernetes - Simplifying Batch Job Submission (2019-02-23)

Many enterprises adopting cloud application platforms like Pivotal Application Service and container orchestrators like Kubernetes for running their business applications and services. One of the advantages for doing so is to have shared resource pools instead of countless application server islands. This not just simplifies infrastructure management and improves security, it also saves much of the costly infrastructure resources. But that must not be the end of the story. Going one step further (or one step back looking why resource management systems like Borg or Tupperware have been build) you will certainly find other groups within your company being hungry for spare resources in your clusters. Data scientists, engineers, and bioinformaticians need to execute masses of batch jobs in their daily job. So why not allowing them accessing your spare Kubernetes cluster resources you are providing to your enterprise developers? With the Pivotal’s Container Service (PKS) control plane cluster creation and resizing even on-premises is just a matter of one API or command line call. With PKS the right tooling for managing a cluster which fills up your anyhow available resources is available.

One barrier for the researchers which are used to run their workloads within their HPC environment can be the different interfaces. If you worked for decades with the same HPC tooling going to Kubernetes can be a challenge. For Kubernetes you need to write declarative yaml files describing your jobs but users might already have complicated, imperative job submission scripts using command line tools like qsub, bsub, or sbatch. So why not having a similar job submission tool for Kubernetes? I started an experiment writing one using the interfaces I’ve built a while ago (the core is basically just one line of wfl using drmaa2os under the hood). After setting up GKE or an PKS 1.3 Kubernetes cluster running a batch job is just a matter of one command line call.

$ export QSUB_IMAGE=busybox:latest
$ qsub /bin/sh -c 'for i in `seq 1 100`; do echo $i; sleep 1; done'
Submitted job with ID drmaa2osh9k9v
$ kubectl logs --follow=true jobs/drmaa2osh9k9v
1
2
3
4

More details about the job submission tool for Kubernetes you can find here. Note that it is provided AS IS.

One thing to have in mind is that there is no notable job queueing and job prioritization system build into the Kubernetes scheduler. If you are submitting a batch job and the cluster is fully occupied the job submission command will block.

Kubernetes allows to hook in other schedulers. kube-batch is one of the notable activities here. Poseidon provides another alternative for the default scheduler which claims to be extremely scalable while allowing complex rule constraints. Univa’s Command provides an alternative scheduler as well. Note that these schedulers can also be used with qsub by specifying the scheduler during job submission time.

Combining Technical and Enterprise Computing (2019-01-23)

New workload types and management systems popping up once in a while. Blockchain based workload types started to run on CPUs, then on GPUs, some of them later even on ASICs. Recent success of AI systems are pushing GPUs, and in analogy to blockchains, specialized circuits to its boundaries. The interesting point here is that AI and blockchain based technologies are from general business interest, i.e. are the forces for lot’s of new business models in many industries. The automotive industry for example (re-)discovered artificial neural networks for solving many of the tasks required to let cars drive them-selfs. I had the luck joining a research team taking care of image based environment perception of a famous car maker for a short while more than 10 years ago. From my perspective the recent developments were not so clear back than - even we had already beginning of the 90s self-driving cars steered by neural networks.

The high-performance computing community has a long tradition in managing tons of compute workload, maximizing resource usage, and queueing and prioritizing workload. But business critical workloads for many enterprises are much different by nature. Stability, interconnectivity, and security are key requirements. Today the boundaries get blurred. Hyperscalers had to solve unique problems running huge amount of enterprise applications. They had to build their own systems which combined traditional batch scheduling with services. Finally Google came up with an open source system (Kubernetes) to allow companies building their own workload management system. Kubernetes solves a lot of core problems around container orchestration but many things are missing. Pivotal’s Container Service enriches Kubernetes when it comes to solve what we at Pivotal call day 2 issues. Kubernetes needs to be created, updated, and maintained. It’s not a closed box - PKS gives you choice, opinions, and best practices running a platform as product within large organizations.

But back to technical computing. What is clearly missing within Kubernetes are capabilities built into traditional HPC schedulers over decades. Batch jobs are only supported rudimentarily at the moment. There is no queuing, sophisticated job prioritization, nor first-class support for MPI workloads like we know from the HPC schedulers. Also the interfaces are completely different. Many products are built on the HPC scheduler interfaces. Also Kubernetes will not replace traditional HPC schedulers, like Hadoop’s way of doing batch scheduling (what today most people associate with batch scheduling) did not replace classic HPC schedulers, they are still around and will survive for a reason. Also the cloud let you think to make queueing obsolete - but only in a perfect world where money and resources are unlimited.

What we need in order to tackle large scale technical computing problems is a complete system architecture combining different aspects and different specialized products. There are three areas I want to look at:

• Applications
• Containers
• Batch jobs

Pivotal has the most complete solution when it comes to provide an application runtime which abstracts about pure container orchestration. The famous cf push command works with all kind of programming languages, including Java, Java/Spring, Go, Python, node…it keeps the developer focusses on its application / business logic rather than on building and wiring containers. That’s already all completely automated since years by concepts like buildpacks, service discovery etc. Additionally to that we need a container runtime for pre-defined containers. This is what Pivotal’s Container Service (PKS) is for. Finally we have the batch job part which can be your traditional HPC system, might it be Univa Grid Engine, slurm, or HTCondor.

If we draw the full picture of a modern architecture supporting large scale AI and technical computing workloads it looks like following:

Thanks to open interfaces, RESTful APIs, and software defined networking a smooth interaction is possible. The open service broker API is acting as a bridge in many of the components already.

Enough for now, back to my Macap coffee grinder and later to OOP conference here in Munich.

The Road to Pivotal Container Service - PKS (2019-01-21)

A few days ago Pivotal released version 1.3 of its solution for enterprise ready container orchestration. Unfortunately the days of release parties are gone but let me privately celebrate the advent of PKS 1.3 - right here, right now.

What has happened so far? It all started with a joint project between Google and Pivotal combining two amazing technologies: Google’s Kubernetes and Pivotal’s BOSH. I’m pretty sure most of the readers know about Kubernetes but are not so familiar with BOSH. BOSH can be described as life-cycle management tool for distributed systems (note, that this is not BOSH). Through deployment manifests and operating system images called stemcells BOSH deploys and manages distributed systems on top of infrastructure layers like vSphere, GCP, Azure, and AWS. Well, Kubernetes is a distributed system so why not deploy and manage Kubernetes with BOSH? Project kubo (Kubernetes on BOSH) was born - a joint collaboration between Google and Pivotal. Finally kubo turned into an official Cloud Foundry Foundation project called Cloud Foundry Container Runtime. CFCR in turn is a major building block of PKS.

With the release of PKS 1.3 Pivotal is supporting vSphere, GCP, Azure, and AWS as infrastructure layer. PKS delivers the same interfaces and admin / user experience independent if you need to manage lots of Kubernetes clusters at scale on-premises or in the cloud.

PKS takes care about life-cycle management of Kubernetes itself. With PKS you can manage fleets of Kubernetes clusters with a very small but highly efficient platform operations team. We at Pivotal strongly believe that is much better to run lot’s of small Kubernetes installations rather than one big. For doing so you need the right tooling but also the right methodology and mindset to do so. Infrastructure as code as well as SRE techniques are mandatory to be effective and scalable. Pivotal supports their customers in that regard by enabling them in our famous Operation Dojos.

PKS as commercial product is a joint development between Pivotal and VMware. With the acquisition of Heptio (a company founded by two of the original Kubernetes creators) VMware has first class knowledge about Kubernetes in house. But let’s go deeper in the PKS components in order to get a better understanding what value PKS provides.

As core component PKS offers a control plane which can be accessed through a simple command line tool called pks. Once PKS is installed through Pivotal’s Ops Manager the control plane is ready. In order to create a new Kubernetes cluster you just need to emit one command: pks create-cluster. What happens is that the control plane creates a new BOSH deployment rolling out a Kubernetes cluster on set of newly allocated VMs or machines. BOSH takes care about keeping all components up and running - if a VM fails it is going to be automatically repaired. Resizing the cluster, i.e. adding more worker nodes or removing worker nodes is just a matter of one single command. But that’a not all, dev and platform ops need logging and monitoring. Logging sinks are forwarding everything which happens on process level to a syslog target. Through a full integration into Wavefront monitoring dashboards can made available with just a few settings. PKS and operating system upgrades can be fully automated and executed without application downtime, Pivotal provides pre-configured CI/CD pipelines and a continuous stream of updates through Pivotal Network. Also part of PKS is VMware’s enterprise ready container registry called Harbor which was recently accepted as official CNCF project with more than 100 contributors. VMwares software defined networking NSX-T (also included in PKS) glues everything seamlessly and very secure together and provides scalable routing and load-balancing functionalities.

There is much more to say but I will stop here, making myself a coffee with my greatly engineered Bezzera, and celebrate the release of PKS 1.3! :-)

Well done!

Unikernels Still Alive? (2019-01-05)

During the widespread adoption of container technologies Unikernels have been seen for a very short while as potentiell successor technology. Docker bought Unikernel Systems in January 2016. After that it has been very quiet around the technology itself. There were some efforts for applying it to high performance computing but also in the HPC community it didn’t get popular. In Japanese compute clusters they experimented with McKernel. In Europe we have an EU funded project called Mikelangelo.

But there is some potential that Unikernels come back again. With the quick adoption of firecracker, a virtualizing technology that is able to start VMs in hundreds of milliseconds, unikernels build into VMs can be potentially orchestrated very well. Of course something which needs to be handcrafted and still put a lot of effort into but the major building blocks exist.

The very well known Unik project (which originated at Dell / EMC btw.) has added already support for firecracker. More information about how it works is described in this blog post.

Pivotal’s Function Service and Go (2018-12-15)

Since a few days Pivotal grants access to Pivotal Function Service for early adopters. All what is required for testing PFS is an account at Pivotal Network (which can be setup within minutes) and a request to the get early access for the alpha version of PFS.

PFS installation is quite simple. All what is required is a Kubernetes cluster, which can be easily created by Pivotal’s Container Service (PKS), or alternatively minikube, or GKE can be used.

There are two files to download: the packages and the pfs command line tool (mac, linux,…). Once downloaded and extracted just follow the documentation which explains each of the necessary steps. Basically you need to push the downloaded images to your container registry and then start the installation inside Kubernetes with pfs system install. It automatically installs istio and the knative components on which PFS is build on. The open source project of PFS sponsored by Pivotal is riff.

After installation PFS is ready to use. Three components are important to understand PFS: functions/services, channels for data exchange between the services, and subscriptions for connecting channels with services.

For this early PFS release functions can be written in Java/Spring, JS, or provided as scripts or executables. I went the extra mile and created functions in go which needs to be compiled in executables and registered in PFS as Command functions. Not a big deal. Nevertheless waiting for native support…:)

The first step is setting up your git repo for your functions which my case is here. Your function can be a shell script converting stdin and writing to stdout or any other binary doing the same. For Java you are implementing the Function interface converting a string into another string by an apply() method. In my go example application I’ve a simple wrapper which selects the function which should be executed by an environment variable which I provide during service creation. I’m reading from stdin and write to standard out. Stderr is ignored.

Now you can register the function creating the service reverse calling the binary f which invokes reverse():

pfs function create reverse --git-repo https://github.com/dgruber/f --artifact lxamd64/f 
--image $REGISTRY/$REGISTRY_USER/reverse --env "FUNCTION=reverse" --verbose

Afterwards I can register my other function _upper_ the same way:

pfs function create upper --git-repo https://github.com/dgruber/f --artifact lxamd64/f
--image $REGISTRY/$REGISTRY_USER/upper --env "FUNCTION=upper" —verbose

Note that I’ve setup my image registry beforehand (in my case the REGISTRY env variable is set to _gcr.io_).

In order to list the functions we can do:

pfs service list
NAME    STATUS
reverse RevisionFailed: Revision "reverse-00001" failed with message: "The target is not receiving traffic.".
upper   RevisionFailed: Revision "upper-00001" failed with message: "The target is not receiving traffic.".
pfs service list completed successfully

There is no traffic yet hence nothing is running (scales down to 0).

To test a function we can use the pfs utility with service invoke:

pfs service invoke reverse --text -- -w 'n' -d 'functional test'

curl 35.205.162.82/ -H 'Host: reverse.default.example.com' -H 'Content-Type: text/plain' -w 'n' -d 'functional test'
tset lanoitcnuf

Voila!

The next step is connecting the functions by channels and subscriptions. Channels abstract away from the underlying infrastructure like Kafka or RabbitMQ or the internal test message bus stub:

pfs channel create messages --cluster-bus stub
pfs channel create upperedmessages --cluster-bus stub
pfs subscription create --channel messages --subscriber upper --reply-to upperedmessages
pfs subscription create --channel upperedmessages --subscriber reverse

What we have done now is letting the upper function subscribe to the messages channel and send the output to upperedmessages channel to which the reverse function is subscribed.

Now we have to send a message to the incoming messages channel. This is typically done by an application (see knative serving) but can be tested by creating by accessing a shell in a cluster pod.

I’m following Dave’s recommendation to just do that within a container for testing.

$ kubectl run -it --image=tutum/curl client --restart=Never --overrides='{"apiVersion": "v1", "metadata":{"annotations": {"sidecar.istio.io/inject": "true"}}}'

And then inside the container:

root@client:/# curl -v upperedmessages-channel.default.svc.cluster.local -H "Content-Type: text/plain“ -d mytext

If no pods are running yet they will be started. Yeah!

kubectl get pods

NAME READY STATUS RESTARTS AGE client 2/2 Running 0 30m reverse-00001-deployment-86bf59b96-4pz8l 3/3 Running 0 2m reverse-00001-k9m2q 0/1 Completed 0 1d upper-00001-deployment-6974ccc47d-6lgz7 3/3 Running 0 1m upper-00001-lgtn8 0/1 Completed 0 1d

When having issues with messages or subscriptions a look into the logs stub-clusterbus-dispatcher pod can be very helpful (it runs in the knative-eventing namespace).

That’s it! Happy functioning!

The Singularity is :%s/near/here/g - Using wfl and Go drmaa2 with Singularity (2018-12-04)

The Singularity containerization software is re-written in Go. That’s a great message and caught immediately my interest :-). For a short intro into Singularity there are many sources including wikipedia or the official documentation.

A typical problem using Docker in a broader context, i.e. including integrating it in workload managers, is that it runs as a daemon. That means requests to start a Docker container end up in processes which are children of the daemon and not of the original process which initiated the container creation request. Workload managers are built around managing and supervising child processes and have reliable mechanisms for supervising processes which detach itself from the parent processes but they don’t work very well with Docker daemon. That’s why most of the serious workload managers do container orchestration using a different mechanism (like using directly runc or rkt or even handling cgroups and namespaces directly). Supervision includes resource usage measurements and limitations which sometimes exceed Dockers native capabilities. Singularity brings back the expected behavior for workload managers. But there are more reasons people use Singularity which I don’t cover in this article.

This article is about using Singularity in Go applications using wfl and drmaa2. Since I wanted to give Singularity a try by myself I started to add support for it in the drmaa2 go binding by implementing a job tracker. It was a very short journey to get the first container running as most of the necessary functionality was already implemented.

If you are Go programmer and write Singularity workflows either for testing, performance measurements, or for running simulations I would be more than happy to get feedback about future improvements.

DRMAA2 is a standard for job management which includes basic primitives like start, stop, pause, resume. That’s enough to built workflow management applications or even simpler abstractions like wfl around it.

In order to make use of the implemented DRMAA2 calls for Singularity in your Go application you need to include it along with the drmaa2interface dependency

    "github.com/dgruber/drmaa2interface"
    "github.com/dgruber/drmaa2os"

The next step is to create a DRMAA2 job session. As the underlying implementation need to make the session names and job IDs persistent you need to provide a path to a file which is used for that purposes.

    sm, err := drmaa2os.NewSingularitySessionManager(filepath.Join(os.TempDir(), "jobs.db"))
    if err != nil {
        panic(err)
    }

Note that when Singularity is not installed the call fails.

Creating a job session is like defined in the standard. As it fails when it exists (also defined in the standard) we need to do:

   js, err := sm.CreateJobSession("jobsession", "")
    if err != nil {
        js, err = sm.OpenJobSession("jobsession")
        if err != nil {
            panic(err)
        }
    }

The JobTemplate is the key for specifying the details of the job and the container.

       jt := drmaa2interface.JobTemplate{
            RemoteCommand: "/bin/sleep",
            Args:          []string{"600"},
            JobCategory:   "shub://GodloveD/lolcow",
            OutputPath:    "/dev/stdout",
            ErrorPath:     "/dev/stderr",
        }

The RemoteCommand is the application executed within the container and therefore must be accessible inside of the container. Args is the list of arguments for the file. The JobCategory is mandatory and specifies the image to be used. Here we are downloading the image from Singularity Hub (shub). In order to see some output in the shell we need to specify the OutputPath pointing to /dev/stdout. We should do the same for the ErrorPath otherwise we will miss some error messages printed by the applications running within the container.

In order to give Singularity more parameters (which are of course not part of the standard) we can use the extensions:

    jt.ExtensionList = map[string]string{
        "debug": "true",
        "pid":   "true",
    }

Please check out the drmaa2os singularity job tracker sources for a complete list of evaluated parameters (it might be the case some are missing). debug is a global parameter (before exec) and pid is an exec specific parameter. Since they are boolean parameters it does not matter if they are set to true or just an empty string. Only when set to false or FALSE they will be ignored.

This JobTemplate will result in a process managed by the Go drmaa2 implementation started with

singularty —debug exec —pid shub://GodloveD/lolcow /bin/sleep 600

RunJob() finally creates the process and returns immediately .

    job, err := js.RunJob(jt)
    if err != nil {
        panic(err)
    }

The job ID can be used to perform action on the container. It can be suspended:

    err = job.Suspend()

and resumed

    err = job.Resume()

or killed

    err := job.Terminate()

The implementation sends the necessary signals to the process group.

You can wait until it is terminated (with a timeout or drmaa2interface.InfiniteTime).

    err = job.WaitTerminated(time.Second * 240)

Don’t forget to close the JobSession at the end and destroy it properly.

    
    js.Close()
    sm.DestroyJobSession("jobsession")

While that is easy it still requires lot’s of code when writing real applications. That’s what we have wfl for. wfl now also includes the Singularity support as it is based on the drmaa2 interfaces.

Since last Sunday wfl can make use of default JobTemplate’s which allows to set a container image and parameters like OutputPath and ErrorPath for each subsequent job run. This further reduces the LOC to be written by avoiding RunT().

For starting a Singularity container you need a workflow based on the new SingularityContext which accepts a DefaultTemplate.

    var template = drmaa2interface.JobTemplate{
        JobCategory: "shub://GodloveD/lolcow",
        OutputPath:  "/dev/stdout",
        ErrorPath:   "/dev/stderr",
    }

    flow := wfl.NewWorkflow(wfl.NewSingularityContextByCfg(wfl.SingularityConfig{DefaultTemplate: template}))

Now flow let’s allow you manage your Singularity containers. For running 100 containers executing the sleep binary in parallel and re-executing any failed containers up 3 times you can do like that.

    job := flow.Run("/bin/sleep", "15").
        Resubmit(99).
        Synchronize().
        RetryAnyFailed(3)

For more capabilities of wfl please check out also the other examples at https://github.com/dgruber/wfl.

Of course you can mix backends in your applications, i.e. you can manage OS processes, k8s jobs, Docker containers, and Singularity containers within the same application.

Any feedback welcome.

Creating Processes, Docker Containers, Kubernetes Jobs, and Cloud Foundry Tasks with wfl (2018-08-26)

There are many workload managers running in the data centers of the world. Some of them take care of running weather forecasts, end-of-day reports, reservoir simulations, or simulating functionalities before they are physically built-in in computer chips. Others are taking care of running business applications and services. Finally we have data-based workload types aggregating data and trying to discover new information through supervised and unsupervised learning methods.

All of them have a need to run jobs. Jobs are one of the old concepts in computer science which exists since the 50s of the last century. A job is a unit of work which needs to be executed. That execution needs to be controlled which is typically called job control. Examples of implementations of job control systems are command line shells (like bash, zsh, ...), but also job schedulers (like Grid Engine, SLURM, torque, pbs, LSF) and the new generation container orchestrators (like Kubernetes), or even application platforms (like Cloud Foundry).

All of the mentioned systems have different characteristics if, when, and how jobs are executed. Also how jobs are described is different from system to system. The most generic systems treat binaries or shell scripts as jobs. Container orchestrators executes applications stored in container images. Application platforms are building the executed containers in a transparent, completely automatic, easy to manage, and very secure way.

In large enterprises you will not find a single system handling all kind of workloads. That is unfortunate since it is really something which should be achieved. But the world is moving - new workload types arise, some of them are removed, and again others will stay. New workload types have different characteristics by nature. Around a decade ago I've seen the first appearance of map/reduce workloads with the advent of Hadoop. Data centers tried to figure out how to run this kind of workload side-by-side with their traditional batch jobs.

Later containers became incredibly popular and the need for managing them at scale was approached by a new class of workload managers called container orchestrators. Meanwhile also batch schedulers are enhanced to handle containers and map/reduce systems can do containers. Container orchestrators are now looking for more. They are moving up the stack were application platforms like Cloud Foundry have learned and provide functionalities since years.

But when looking at those systems you will also find lot's of similarities. As mentioned in the beginning they all have the need for running a job for different reasons. A job has a start and a defined end. It can fail to be accepted by the system, fail during execution, or succeed. Often they need more context, like environment variables to be set, job scheduling hints to be provided, or data to be staged.

Standards

If you are reading this blog you are most likely aware that there are standards already defined handling all these tasks. Even been much longer around than most of the new systems they are still applicable for new systems.

In some free time I adapted the DRMAA2 standard to Go and worked on an implementation for different systems. Currently it is working on processes, Docker, Kubernetes, and Cloud Foundry. The Grid Engine implementation is based on the C library, the others are pure Go.

When working on the implementation I wanted to write quickly test applications so I've build a much simpler abstraction around it, which I called wfl.

With wfl it makes even more fun to write applications which need to start processes, Docker containers, Kubernetes jobs, or Cloud Foundry tasks. You can even mix backend types in the same application or point to different Kubernetes clusters within one application. It is kept as simple as possible but still grants access to details of a DRMAA2 job when required.

There are a few examples in the wfl repository demonstrating its usage. A general introduction you find in the README on github.