This is the latest post in the series on deploying a Weblogic cluster in Amazon EC2. Previous posts have shown how to create and configure a weblogic cluster using either standard Amazon EC2 images or RightScale ServerTemplates and RightScripts.
In the first post in the series, I explained how to deploy a load-balanced two-node Weblogic cluster in Amazon EC2. If you want to deploy a cluster with more than two nodes in it, you need to introduce a proxy server to the mix. This will keep track of which Weblogic sessions are registered with which Weblogic nodes in the cluster (a maximum of two) and therefore will be capable of redirecting each request to a Weblogic node that has a session for the user.
Ideally, you want two proxy servers in case one goes down – we don’t want a single point of failure – and you want some kind of load balancer to direct requests to one of the two proxy server instances. In summary, you want an architecture such as that shown below, with an Amazon Elastic Load Balancer directing requests to one of two Apache instances acting as the proxy servers which, in turn, redirect requests to one of the nodes in the Weblogic cluster – I’ve shown 3 here but there could be many more nodes in the cluster.
Such an architecture is the ideal in terms of load balancing and high-availability and this post will explain how to achieve it in Amazon EC2.
Configuring the 3 Oracle Weblogic Cluster Nodes
Follow the steps outlined in the following sections of the first post in the series to create the Oracle cluster nodes. The only variation is that you should create 3 nodes instead of 2, naming them Server-0, Server-1 and Server-2:
- Launching the instances
- Creating the Weblogic domain
- Starting the Weblogic Administration Server and configuring the cluster
- Copying the managed server configuration
- Checking that the servers work OK
Make sure you configure the cluster to work with the Weblogic proxy. To do this, in the Weblogic Administration console, click on Clusters and then cluster-0. In the General > Configuration tab, expand the “advanced” option and tick the “WebLogic Plug-In Enabled” option.
At this stage, you should have 3 EC2 machines running which we will refer to as [machine-0] [machine-1] and [machine-2]. Each machine should be running a weblogic server and a weblogic admin server should also be running on machine-0.
Configuring the Apache Servers
We will use two instances of the Apache Web Server as proxy servers for the weblogic cluster members. In EC2, start two machines based on the Alestic Ubuntu Server 9.04 32-bit (ami-bf5eb9d6) image. We will refer to these machines as [machine-3] and [machine-4].
SSH into machine-3 and run the following commands to install Apache 2.2:
apt-get update apt-get install apache2
Download the Oracle Weblogic Apache plugin bundle to your local machine. To find it, go to this page and download the item marked “Apache Plug-ins zip”. Open the ZIP file and just extract the Apache 2.2 Linux module which you can find at /linux/i686/mod_wl_22.so. Upload this file from your local machine to machine-3 using SCP. How you do this between Windows and Linux users. Below is the Linux command, in which you will need to substitute [machine-3] for the public DNS address of machine-3:
scp mod_wl_22.so root@[machine-3]:/usr/lib/apache2/modules/mod_wl_22.so
Now that we have uploaded the Weblogic proxy plugin, we need to configure it. On machine-3, create a new file called /etc/apache2/mods-enabled/weblogic.load and add the following content:
LoadModule weblogic_module /usr/lib/apache2/modules/mod_wl_22.so
On machine-3, create a file called /etc/apache2/mods-enabled/weblogic.conf and add the following content, substituting [machine-*] with the relevant public DNS address:
<IfModule mod_weblogic.c> WebLogicCluster [machine-0]:7002,[machine-1]:7002,[machine-2]:7002 MatchExpression /clustered-webapp/* Debug ON DebugConfigInfo ON WLLogFile /tmp/weblogic.log </IfModule>
This is the core Apache configuration for the Weblogic proxy plugin. The first two lines are the most important – they tell the plugin which nodes are in the cluster and which URL patterns it should match against to proxy requests to the cluster. Note that I believe you don’t have to specify all the nodes in the cluster – the proxy should be capable of finding all nodes in the cluster by asking just one member – although here I’ve specified all 3.
Now reload Apache so it picks up the changes:
Go to http://[machine-3]/clustered-webapp and check that you get a screen saying something like “Hello World! This machine’s IP Address is: ip-10-250-10-63/10.250.10.63″. This shows that the Proxy running in the Apache server has connected to the cluster and is proxying requests to it. Given that we specified the “DebugConfigInfo” option in the configuration earlier, we can easily get debug information by accessing the following URL: http://[machine-3]/clustered-webapp/?__WebLogicBridgeConfig. The screen you should see is something like that shown below:
This screen is very useful for debugging problems with the proxy configuration but, as you can probably guess, it’s not a good idea to make this information publically available in a production system. The output above indicates that the proxy has correctly identified the cluster and the 3 servers available within it.
Now that one Apache server works ok, configure machine-4 in exactly the same way so that there are two apache servers (machine-3 and machine-4) which act as proxies to the 3 weblogic servers in the cluster.
Configuring the Elastic Load Balancer
To complete the organization proposed in the introduction, we have to introduce an Amazon Elastic Load Balancer which will balance requests to the two Apache web servers. If one goes down, the load balancer should be able to redirect all requests to the remaining server, signifying no drop in availability for end users.
If you have not already installed the Amazon load balancer command line tools, follow the steps below:
- Configure your machine to use the Amazon command line tools if you have not already done so
- Download and unzip the Amazon Elastic Load Balancing API Tools from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2536&categoryID=88
- set an environment variable $AWS_ELB_HOME to point to where you unzipped the tools to.
- add $AWS_ELB_HOME/bin to your path.
Run the following on your local machine to create the load balancer. My instances are in us-east-1b (you can find this out from the “zone” attribute when you click on a running instance in the AWS Management console). Note the DNS name that you are given when the command finishes – let’s call this [elb-dns]. This command configures the load balancer to listen on port 80 (standard HTTP port) and redirect all traffic to port 80 on each of the two Apache web servers which we will register with the load balancer.
local:-$ elb-create-lb Test --availability-zones us-east-1b --listener "protocol=http,lb-port=80,instance-port=80"
Register your instances with the Load Balancer. Use the AWS Management console to discover your instance IDs.
local:-$ elb-register-instances-with-lb Test --instances [instance-id-machine-3] [instance-id-machine-4]
Configure the health check. This specifies that the load balancer should access the URL /clustered-webapp/ on port 80 on each of the Apache Web Server instances registered with the load balancer every 5 seconds. It should wait up to 3 seconds for a response from each instance and if it gets 2 response failures, it won’t send any requests to that instance. When it gets 2 successful responses from an instance, it starts sending requests to it again.
local:-$ elb-configure-healthcheck Test --target "HTTP:80/clustered-webapp/" --interval 5 --timeout 3 -unhealthy-threshold 2 --healthy-threshold 2
Now, when you go to http://[elb-dns]/ you should get the same page as you got when you accessed each Apache web server separately. The IP address should change, depending on which Weblogic server the load balancer uses to serve the request.
Proving High availability
This deployment architecture should enable the system to cope with the failure of an Apache proxy server and at least one Weblogic server with no apparent consequences for the end user. Let’s prove it!
Go to the URL http://[elb-dns]/clustered-webapp/sessionCounter and you should see a HTTP page that says how many times you have accessed the page. Click refresh a few times so that the number is greater than 1 and keep this page open in your web browser.
On your local machine, run the elb-describe-instance-health command to show the health of the two instances that the load balancer is balancing requests against. The output should show both Apache servers “InService”:
$ elb-describe-instance-health Test INSTANCE-ID i-4320782a InService INSTANCE-ID i-21346c48 InService
Now, in the SSH console for machine-3, shut down Apache by executing the following:
Go back to the Session Counter web page in your browser and click refresh. The number of times you have accessed the page should not be reset to 1 – it should keep rising each time you click refresh. This proves that the failure of one Apache server does not affect the end user. To prove that the apache service is down, on your local machine, run the elb-describe-instance-health command again, which should give you results similar to these:
$ elb-describe-instance-health Test INSTANCE-ID i-4320782a OutOfService INSTANCE-ID i-21346c48 InService
Leave the Apache service shut down and terminate the Weblogic server running on machine-1 by pressing ctrl-c in the console window. Return to the Session Counter web page which is displayed in your browser. Click refresh a few times to prove that the session counter is maintained thereby proving that there is no effect on an end-user when a weblogic node fails.
How does all of this work?
There is more detailed information in the Weblogic documentation but basically when you connect to a Weblogic cluster for the first time, the node that serves your request creates a session ID cookie that contains the session ID and the IDs of two nodes in the cluster in the form [sesionID]![PrimaryNodeID]![SecondaryNodeID] e.g. 2VyWK6hdcJGjDHp3t11GKL8Pv2l6kK0V2p113Gc0Sp12Y2H0vcG8!1074248988!1767416532. The primary and secondary nodes will both contain a copy of your session.
When the Weblogic proxy running in Apache receives a request, it looks at the session ID cookie to work out which nodes in the cluster are the primary and secondary nodes that contain the user’s session. It will then try to redirect the request to the primary, only resorting to the secondary if the primary does not respond in a reasonable time. If both nodes fail at the same time – which should be an extremely rare occurrence – then the user’s session is lost. Weblogic always tries to ensure that each session is stored on two nodes in the cluser so if a node fails, Weblogic will move sessions that were stored on that machine to others.