Middleware Shootout

In a nutshell, middleware is the glue between the Web and the outside world of applications, users and their data, and it's taking on an increasingly larger role for application developers. Making sure customers and business partners have access to the information they need has become more critical, and application integration is still one of the biggest hurdles for IT managers as they deploy their Internet strategies.

No doubt, serving as the glue is a big responsibility. Obviously, there are integration issues, and developing the connections between Web and database applications is a complex task that can take months of development work.

That's why choosing the right middleware platform is so important, and understanding how the applications work and are constructed is mandatory for resellers and integrators. In this product evaluation, KeyLabs, VARBusiness' testing partner, examines the scalability and performance optimization characteristics of three middleware application-development platforms: BEA WebLogic, IBM WebSphere and Microsoft .Net.

KeyLabs frequently provides proof-of-concept testing services for enterprise developers who are grappling with middleware implementations. The three most often-asked questions include:

• Based on the Web application solution I have chosen, how much hardware do I need?
• How well does this middleware environment scale?
• What kind of expertise do I need to deploy a middleware solution?
• While those questions appear simple on the surface, anyone capable of providing the answers is going to provide the same response,it all depends on your Web application code. Because these are environments that depend on the quality of developed solutions, and because they are at the mercy of the complexity of the application, providing an answer to those questions is no simple task. So, KeyLabs created a test based on very simple workloads. The goal is to provide baseline information that will be helpful in determining resource requirements.
• We are not attempting to compare products in this article. However, once an environment has been selected, this review does provide insight into what to expect from that environment. Obviously, results will vary based on the quality of your code and the complexity of your application.
• Here are our findings:
• Installation and Configuration
• To gain an appreciation of the complexities of each product, KeyLabs used an engineer with considerable network and coding experience, but who was not familiar with the product-specific implementations. This engineer was required to configure the hardware and network, install the operating systems, and install and configure the solution and populate the database. Then the evaluator coded the application to test for scalability.
• Platforms
• BEA WebLogic
  With thousands of implementations, WebLogic is a veteran of the Internet. It is possible to implement WebLogic in a number of different hardware/software environments, such as Solaris, Linux and Windows. The use of Java as the programming language is another nice touch in allowing for maximum flexibility and portability.
• WebLogic required 16 hours for installation, configuration and application development. Several third-party texts were available at a local bookstore, and a search on Amazon.com for "WebLogic" returned a total of 14 available texts. Some information was available via the Internet, and BEA provided good documentation.
• WebLogic is not a difficult environment to install, configure or work in. Still, the relative lack of third-party documentation and the specialized nature of the product warrants careful consideration by an in-house individual with specific WebLogic experience. If such a resource is not available, consider the use of an outside resource, either for implementation or for consultation.
• IBM WebSphere
  It took a little more than 26 hours to install and configure WebSphere. No third-party texts were available locally, although a search on Amazon.com
• yielded 24 results using "WebSphere" as the search criteria. General searches on the Internet provided very little usable information.
• WebSphere boasts a large installed base and has the resources of IBM behind it. It installs on a number of
• different platforms, including Solaris, AIX, Linux and Windows. The use of Java as the programming language adds to the flexibility and portability of the overall solution.
• The WebSphere installation, though not troublesome, did have some complexities, including required patches for even our basic applications to run successfully. While WebSphere is obviously a capable Web solution environment, the environment's complexities and lack of widespread third-party documentation make it all but necessary to have a WebSphere-savvy employee or consultant on the team.
• Microsoft .Net
  Microsoft .Net is a newcomer to the Web-solution arena. There are also mixed evaluations of its success to this point. That said, .Net has the full weight of Microsoft, and the availability of third-party documentation suggests that more than just those at Microsoft expect the platform to catch on. Currently, the .Net environment is available only on Windows, although it is possible for the environment to be ported to other platforms at a later date.
• The Microsoft .Net environment required 10 hours to complete the tasks required for installation and configuration. Nevertheless, the .Net installation and configuration was simple and straightforward. There were hundreds of titles on the .Net environment and languages at local bookstores, and a search on Amazon.com returned in excess of 1,100 texts on the subject.
• The online help was used almost exclusively, even though third-party texts were purchased. Because of the vast amount of third-party documentation, as well as information on the Internet and the adequate help system, it's
• likely that a talented and dedicated developer would be able to adapt to .Net. Still, if a complex application is the goal, the use of a resource with expertise developing Internet solutions is the safest bet.
• Resource Requirements and Scalability
• Two data points have been used to provide insight into the resource requirements for each of the platforms. First, given our methodology, we measured how many transactions per second are accomplished with the specified number of processors. Second, at each processor level we show how much memory is committed for utilization.
• Processor utilization: The first graph in each section shows the number of transactions per second accomplished by the specified environment at multiple processor levels. In all cases, the number of transactions completed was limited by processor saturation. The data contained in these graphs shows that all of the environments have a one-to-one scaling ratio as processors are added. This should provide some comfort that, given a properly coded application, all of these platforms are good choices for applications that need to scale with hardware.
• It is important to note that the original methodology was altered in the Microsoft test to accommodate an aberration in the way .Net handled mathematical calculations. Our original test methodology included a mix of transaction types, but given that .Net required so much more time on mathematical functions than the other platforms, the number of transactions were reduced so those delays wouldn't affect the overall test results.
• Memory utilization: The memory utilization graph in each section shows the amount of memory used at each processor level. It's interesting to note that as the number of transactions accomplished by a particular application increases, memory does not increase in equal proportions.
• These graphs indicate a baseline memory configuration. This data was collected every 20 seconds, then averaged for each processor count. With this in mind, KeyLabs recommends that a minimum of twice the memory shown as utilized be considered the baseline memory configuration.
• The memory utilization for BEA WebLogic actually decreased as the number of processors and the number of transactions increased. These results are counterintuitive; as such, they were scrutinized and validated.
• Which Middleware Is Right For You?
• Picking the right middleware platform is more of an art than a science. WebLogic offers the best memory usage, while .Net has the widest development community and support resources; WebSphere is somewhere in between the two. All three products performed adequately and scaled up as processors were added to the test servers. The ultimate decision comes down to what expertise is available in-house and what existing applications use.
• Methodology
• We evaluated each product in somewhat different environments. BEA WebLogic Workshop was implemented using Oracle9i on the back end. IBM WebSphere 4.0.1 was installed on Intel hardware using DB2 version 7.2 for a database solution. Microsoft .Net was implemented using Windows 2000 Advanced Server and the .Net Package using MS SQL2000 as a database solution.
• For the scalability test, we tried to eliminate (or at least decrease) as many variables as possible by concentrating on primitive tasks that are the foundation of most solutions. These tasks include mathematical calculations, string manipulation, maintaining user context and database access.
• A simple implementation of the building blocks mentioned above was created using ASP.Net and Java. Each of those tasks was assigned to a specific http request that required no parameters and returned a single-line text string for validation. Load was applied by calling each of those methods via an http request using WebLoad by RadView. While the systems were under the test, the time required for specific tasks was measured along with statistics gathered from the server.
• Two servers were used throughout the test. The middleware application server was a Compaq 760 using two, four and eight 900-MHz Intel Pentium III processors and 8 GB of RAM. The database server was a Dell PowerApp with two 1.2-GHz processors and 512 MB of RAM; each system used a single 10K RPM SCSI drive. Clients were generated using 110 physical computers, each running a single user. Client resource utilization remained less than 10 percent throughout the test. The test network consisted of Cisco Catalyst 2900 switches hard set to 100-MB full duplex, as was each PC. Cisco 6500 was used as the test network backbone.
• The middleware server was configured with two, four and eight processors, and a series of tests was executed at each processor level. Specific percentages were assigned to each of the tasks, remaining the same throughout all tests with all three products and at all processor levels. A single exception was made to this methodology in the mathematical function used to test .Net and will be addressed in the specific results.
• Doug Coombs is senior test engineer at KeyLabs (www.keylabs.com), a technology assurance solution provider located in Lindon, Utah.