Google Go and Univa Grid Engine: A Go DRMAA Language Binding Implementation - Part 1 (2012-10-07)

Google's Go looks for me is like the most interesting programming language published in recent years (of course there are others, like “Julia”, but they are more domain specific (technical computing)).  It is compiled, has automatic garbage collection, multiple return values, offers pointers, methods, interfaces, built-in maps, slices, range expressions, coroutines and channels, and its easy to use. There are a lot of articles which discusses all the features, so there is no need to go in further details. The amount of keywords is small (about 20), which makes it clearly arranged. But there is one little thing which I really miss: Method overloading. Having for each different parameter set a different method could IMHO lead to a method name zoo. There is a work-around to use the default interface{} (which is implemented by each Go type) which could be combined with an ellipsis (in Go: “…”) and type checking, but then the signature is hard to read. It would be easier if this would just be possible out-of-the box.

In order to gain more expertise in Go I developed a DRMAA (1.0) language binding in some free time. The current  release is not well tested and there are still some details which I want to re-work, so I would describe the version as a first “Proof of Concept”. Nevertheless you can download it and use it for free without any bigger restrictions. There are no guarantees and I don't take any responsibility for anything bad which could happen (like that an asteroid destroys your computer while using ;). But when you have any issues feel free to contact me (Daniel) at This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , so that I can repair the defect or we can at least discuss about the issue. I developed it using Univa Grid Engine 8.1. (feel free to download the 48-core limited free demo version), but it should work with any other Grid Engine fork which offers the C DRMAA 1.0 binding. Like the Java DRMAA or Python binding it uses internally the C library, hence in theory it should work also for any other job scheduler which offers a C DRMAA 1.0 binding. If you are trying that with success or without success, please let me know.

How to use the Go DRMAA binding


The Go DRMAA binding consists of the Go DRMAA library and a small piece of documentation. For a more detailed description of the functions please checkout the DRMAA spec or the Grid Engine DRMAA man pages. Internally the Go lib needs to access a C DRMAA 1.0  library, hence you must set the LD_LIBRARY_PATH to the right location. For Univa Grid Engine it is $SGE_ROOT/lib/lx-amd64. Additionally you need to source the GE settings.sh (or settings.csh) file before. Then your Go programs can be compiled and started.   Here is a small example of a command which submits a binary program with one argument to the job scheduler. I called it “dsub” for drmaa submission (similar POSIX standardized submission client name  “qsub”).

 

package main
 
import ("drmaa"
        "fmt"
        "os")
 
func main() {
   var session drmaa.Session
   args := os.Args[2:]
   
   if err := session.Init("session=dsubdrmaasession"); err == nil {
      jt, _ := session.AllocateJobTemplate()
      jt.SetRemoteCommand(os.Args[1])
      jt.SetArgs(args)
      id, _ := session.RunJob(&jt)
      fmt.Println(id)
      session.DeleteJobTemplate(&jt)
      session.Exit()
   } else {
      fmt.Println("Session init error: ", err)
   }
}
 

First you need to import the “drmaa” package then you have to create a DRMAA session. The DRMAA way to do it is to call “Init(“sessionname”)” where session name can be the name of a previously created session or an empty string (“”) which lets Grid Engine create a random session name. In order to simply that I also provide a “drmaa.MakeSession()” which returns a new initialized session back (which can be used with the “:=” operator). In order to submit a job you need to allocate a job template, which fields should be set accordingly. A job template can be get from the session method AllocateJobTemplate(). Internally there are some malloc calls hence you must call the session method DeleteJobTemplate() in order not to generate memory leaks. In order to run a job successfully it requires at least to provide a command / application name. This is done with the SetRemoteCommand() job template method. Arguments for the application are set by SetArgs() method, which requires a slice of strings as sing argument. A session should always be closed with an Exit() call.   Just two examples where method overloading would had been nice: Session.SetArgs([]string) requires to convert a single argument into a string slice. Hence an additional Session.SetArgs(string) could make life easier. Instead I implemented Session.SetArg(string). The other example is the Init(string) method: A Session.Init() could replace Session.Init(“”). Instead I'm using the MakeSession().
An example program that prints out the status of a job (which id is given as the first parameter) is shown below: dstat.go.

package main
 
import ("drmaa"
        "fmt"
        "os")
 
func main() {
   var session drmaa.Session
   args := os.Args
 
 
   if err := session.Init("session=dsubdrmaasession"); err != nil {
      fmt.Println(err)
      return
   }
 
  if len(args) <= 1 {
    fmt.Println("Usage: dstat jobid")
    return
  }
 
   ps, _ := session.JobPs(args[1])
   fmt.Println("Job is in state: ", ps)
 
   session.Exit()
}
 

...and the most important thing: You can download the Go DRMAA library here!

Don't forget to source the $SGE_ROOT/default/common/settings.sh file and set the LD_LIBRARY_PATH to $SGE_ROOT/lib/lx-amd64/ before you are starting programming in Go with the drmaa pacakge and Univa Grid Engine

 

The package documentation is here: Go DRMAA Language Binding Documentation