Fraunhofer Institut für Techno- und Wirtschaftsmathematik (ITWM) is a German research institute working to combine leading-edge science with solutions for industry problems. The Competence Center for High-performance Computing provides solutions for industry customers ranging from financial institutions to the oil and gas industry. ITWM commercial products include a high-performance parallel file system (FhGFS), specialized visual analytics tools for the oil and gas industry, and parallel programming tools such as the Global Address Space Programming Interface (GPI) to simplify distributed multi-core programming.
ITWM integrated Amazon Web Services (AWS) into their high-performance middleware PHASTGrid, an implementation of ITWM’s Service-oriented computing philosophy: Users access applications in the form of simple Web services integrated with user applications, and the middleware is responsible for distributing and parallelizing jobs and managing resources.PHASTGrid is implemented in C++ with Ruby bindings to access AWS. The first fully functional implementation of PHASTGrid on AWS took approximately one week—“surprisingly fast,” says Dipl.-Wirtsch. Ing. Mathias Dalheimer, who leads the distributed technologies group and handles all grid and cloud computing activities in his department.
The PHASTGrid infrastructure is designed around its service gateway, offering users access to all deployed applications. Users submit a job definition to the service gateway, which identifies the best resource pool to which to submit the job, depending upon both job type and the availability of resources.
Jobs are automatically executed, supervised, and—in the case of job failure—resubmitted (see Figure 1).
Figure 1. The PHASTGrid infrastructure
To maximize job throughput, the ITWM team adjusts applications, integrating them into the framework with an extension of the MapReduce algorithm that allows control over application execution within the system (see Figure 2).
Figure 2. Workflow for a financial Monte Carlo application
Each job is calculated in three steps:
PHASTGrid is deployed at numerous ITWM customer sites and manages several thousand cores in some installations. In addition, ITWM exports workloads to Amazon Elastic Compute Cloud (Amazon EC2), dynamically adjusting the size of the Amazon EC2 pool transparently. It is possible to run the entire PHASTGrid stack on Amazon EC2—in particular, the ITWM team currently offers services to the financial industry on Amazon EC2. “The job characteristics are a good match for the infrastructure,” says Dalheimer.
ITWM regularly evaluates new technologies for customers. AWS provides easy, low-overhead access to compute resources. “Since we can change all parts of the [Amazon] EC2 image, it was easy for us to integrate our own middleware. We also like the ability to select the type of instance,” says Dalheimer. “This way, we can choose the instance type that suits the application.”
ITWM’s main use for AWS is offloading certain intermediate computations when local resources are full. The ITWM team recently implemented an interface to AWS allowing a transparent shift in workloads from local resources to AWS, supporting the concept of different platforms within a single installation.
“Designing a distributed system is never an easy task. The simpler the better,” says Dalheimer with a smile. He suggests that engineers choose well-understood techniques and strive for simplicity; he calls AWS a good platform for implementing custom solutions. “The AWS interfaces are well defined and a pleasure to work with.”
Amazon Case Study: Fraunhofer ITWM – from Grid to Cloud
Date: 28-08-2013