Fraunhofer ITWM – from Grid to Cloud

Company Overview:

Fraunhofer Institut für Techno- und Wirtschaftsmathematik (ITWM) is a German research institute working to combine leading-edge science with solutions for industry problems. The Competence Center for High-performance Computing provides solutions for industry customers ranging from financial institutions to the oil and gas industry. ITWM commercial products include a high-performance parallel file system (FhGFS), specialized visual analytics tools for the oil and gas industry, and parallel programming tools such as the Global Address Space Programming Interface (GPI) to simplify distributed multi-core programming.

FRAUNHOFER-ITWM-Case-Study-Foetron Inc.ITWM integrated Amazon Web Services (AWS) into their high-performance middleware PHASTGrid, an implementation of ITWM’s Service-oriented computing philosophy: Users access applications in the form of simple Web services integrated with user applications, and the middleware is responsible for distributing and parallelizing jobs and managing resources.

PHASTGrid is implemented in C++ with Ruby bindings to access AWS. The first fully functional implementation of PHASTGrid on AWS took approximately one week—“surprisingly fast,” says Dipl.-Wirtsch. Ing. Mathias Dalheimer, who leads the distributed technologies group and handles all grid and cloud computing activities in his department.

The PHASTGrid infrastructure is designed around its service gateway, offering users access to all deployed applications. Users submit a job definition to the service gateway, which identifies the best resource pool to which to submit the job, depending upon both job type and the availability of resources.

Jobs are automatically executed, supervised, and—in the case of job failure—resubmitted (see Figure 1).

Fraunhofer ITWM infrastructure diagram

Figure 1. The PHASTGrid infrastructure

To maximize job throughput, the ITWM team adjusts applications, integrating them into the framework with an extension of the MapReduce algorithm that allows control over application execution within the system (see Figure 2).

Fraunhofer ITWM workflow diagram

Figure 2. Workflow for a financial Monte Carlo application

Each job is calculated in three steps:

  • The job is transformed into smaller tasks to be executed in parallel.
    • The implementation of this transform function is specific to the job and incorporates a performance model of the application. For financial Monte Carlo simulations, the ITWM team creates tasks based on the number of iterations necessary. In addition, snapshots for performing fault tolerance operations are created.
  • Tasks are submitted and executed in parallel.
  • Task results are collected and aggregated.

PHASTGrid is deployed at numerous ITWM customer sites and manages several thousand cores in some installations. In addition, ITWM exports workloads to Amazon Elastic Compute Cloud (Amazon EC2), dynamically adjusting the size of the Amazon EC2 pool transparently. It is possible to run the entire PHASTGrid stack on Amazon EC2—in particular, the ITWM team currently offers services to the financial industry on Amazon EC2. “The job characteristics are a good match for the infrastructure,” says Dalheimer.

ITWM regularly evaluates new technologies for customers. AWS provides easy, low-overhead access to compute resources. “Since we can change all parts of the [Amazon] EC2 image, it was easy for us to integrate our own middleware. We also like the ability to select the type of instance,” says Dalheimer. “This way, we can choose the instance type that suits the application.”

ITWM’s main use for AWS is offloading certain intermediate computations when local resources are full. The ITWM team recently implemented an interface to AWS allowing a transparent shift in workloads from local resources to AWS, supporting the concept of different platforms within a single installation.

“Designing a distributed system is never an easy task. The simpler the better,” says Dalheimer with a smile. He suggests that engineers choose well-understood techniques and strive for simplicity; he calls AWS a good platform for implementing custom solutions. “The AWS interfaces are well defined and a pleasure to work with.”


Amazon Case Study: Fraunhofer ITWM – from Grid to Cloud

Date: 28-08-2013

We look forward to working with you for Cloud, Security, and other IT product-related opportunities. You can connect with us at +91-8826294740