Ben Wang ben.wang@jboss.com August, 2004

Nowadays http web clustering (or failover) typically employs either a hardware or software front-end load balancer to distribute the load (with capability of session affinity). And the session state replication is then delegated to the web container. Naturally, JBoss also has a session replication layer for Tomcat since release 3.x. In JBoss release 3.2.6, we have re-factored the Tomcat 5 http session replication framework using JBossCache (http://www.jboss.org/products/jbosscache) as the underlying in-memory replication store. JBossCache is an open-source replicated, transactional, and fine-grained cache system. Using JBossCache as the framework for session replication offers separation of replication layer concerns (e.g., replication mode, lock behavior, partitioning, persistence, and transaction) from the Tomcat session handling logics (e.g., sticky session, replication granularity, and trigger). The end result is a reduced development effort in Tomcat application layer and more robust behavior in the overall system.

This section describes briefly the high level design and the web application deployment process pertaining to the session replication.

The following is a class diagram focusing on the http session replication using JBossCache as the distributed store service under Tomcat5.

In addition to the JBossCacheService class that provides replication service, there are three main interfaces: org.jboss.web.tomcat.service.session.SnapshotManger. Interface that defines the API to check for necessary session replication. Note that it has a reference to JBossManager to perform session replication. Two concrete implementation classes are: org.jboss.web.tomcat.service.session.InstantSnapshotManager . Use of this snapshot manager will check for dirty session after every single http request (through org.jboss.web.tomcat.service.session.ClusteredSessionValve ). If a session is dirty, it will then be replicated. org.jboss.web.tomcat.service.session.IntervalSnapshotManager . Use of this snapshot manager will check for dirty session(s) only periodically (with a configurable timer interval). The dirty session(s) will be replicated when the timer kicks in. org.apache.catalina.Manager. This is a Tomcat catalina interface that handles the session management. The JBossCacheManager is the implemented class that also manages both in-memory session and the sessions located in the distributed store. The reason that we need to use the in-memory session instance is because of the web class loader can be different from the system one. As a result, we will need to keep a "serialized" version of the session data in the distributed store (done via org.jboss.invocation.MarshalledValue) so that the class loading scope will be correct. Otherwise, it will be simply too expensive to retrieve and de-serialize every time from the underneath store. This manager is also called periodically by catalina to process expired session(s) (or session timeout). When a session is timed-out, it will be deleted from the store. However, the operation is local only. That is, the delete operation will not replicate to other cluster nodes since the other nodes will have similar expiring process in place already. Therefore, the session will be expired simultaneously from the whole cluster view. org.apache.catalina.Session. This is a catalina interface that manages the session attributes and track the session life time itself. We have two different concrete implementation classes in this release: org.jboss.web.tomcat.service.session.SessionBasedClusteredSession. This corresponds to the setting in jboss-web.xml of replication-granulairty set to session. That is, whenever a session attribute is changed, it will replicate the whole session, regardless. org.jboss.web.tomcat.service.session.AttributeBasedClusteredSession. This corresponds to the setting in jboss-web.xml of replication-granulairty set to attribute. That is, whenever a session attribute is changed, it will replicate only the modified attribute. Note that this option which is finer grain in nature has the potential for session replication performance boost, especially when the session is carrying a large amount of data.

When a user deploys (or hot-deploys) a web application, the JBoss org.jboss.web.tomcat.service.TomcatDeployer will first determine if the web application is declared distributable. If it is, JBoss will instantiate JBossCacheManager, create SnapshotManager and then add a ClusteredSessionValve to the Tomcat interceptor chain. As a result, every single http request will be intercepted by this valve to process the necessary session replication (through SnapshotManager).

Here are the features provided with this release. For clarity, we will divide them into Tomcat and JBossCache level aspects, meaning the configuration will be done either from JBossCache or the TC5 layer.

In jbossweb-tomcat5.0/META-INF/jboss-service.xml , you can configure the followings: SnapshotMode instant Session replication is performed during each HTTP request. But whether it is instantaneously or not also depends on the replication mode (synchronous or asynchronous) in JBossCache setting. Interval Replicated during a specified time interval. There is an accompanying flag, SnapshotInterval, that indicates the snapshot interval in milliseconds. The IntervalSnapshotManager will perform session dirty check every SnapshotInterval. This manager keeps track of the dirty session(s) internally. UseJK A flag indicating whether to use the Apache mod_jk(2) for the front end software load balancing with sticky session combined with JvmRoute. If set to true, it will insert a JvmRouteFilter to intercept every request during deployment and replace the JvmRoute if it detects a failover in runtime. In addition, you will need to set the JvmRoute inside Tomcat, e.g., <Engine name="jboss.web" jmvRoute="Node1" defaultHost="localhost"> in jbossweb-tomcat4.0/server.xml. Note that the above configuration parameters are applied to per Tomcat instance. Change of the parameter will cause Tomcat to re-deploy. In each web application, you can also specify in jboss-web.xml to customize the clustering behavior. Specifically, there is an element called replication-config that contains two sub-elements so far: replication-trigger It determines what triggers a session replication (or when is a session is considered dirty). It has 4 options: ACCESS Indicating if accessing the session is considered dirty. If set, it is considered dirty with pure access and thus will cause replication. Note that a session is "accessed" during each http request regardless. It will update the access time stamp in the session instance as well. Since the time stamp may not be updated in other clustering nodes (because of no replication), this may cause session in other nodes to expire before the active node if http request does not retrieve or modify any session attributes. When this option is set, the session timestamps will be synchronized throughout the cluster nodes. Note that use of this option can have a significant performance impact so you will need to use with caution. SET_AND_GET Every session get and set attributes are considered dirty. This option also can have significant performance impact as well. SET_AND_NON_PRIMITIVE_GET Session get and non-primitive get are considered dirty. For example, the http session request may retrieve a non-primitive object instance from the attribute and then modify the instance. If we don't specify that non-primitive get is considered dirty, then the modification will not be replication properly. This is the default value. SET Only set attribute operation is considered dirty. If you are certain that only set attribute operation is necessary for replication, this option will be most optimized in term of performance. replication-granularity session Replication is per session instance. As long as it is considered modified when the snapshot manager is called, the whole session object will be serialized. attribute Replication is only for the dirty attributes plus some session data, like, lastAccessTime. For session that carries large amount of data, this option can increase replication performance. field Will need aop (aspect-oriented programming) and be available in 4.0.4. User can have control of even finer grain replication through its own object graph. For example, if one stores another map object into the session as an attribute, we can track the dirty field down to the value object inside the map.

The main configuration file is jboss-web-cluster-service.xml of which is basically a minimum version of JBossCache configuration xml file (more details later). CacheMode REPL_SYNC Replication is synchronous and call is blocking until replication succeeds or fails. That is, replication will be completed before the http request returns. REPLY_ASYNC Replication is asynchronous and call is non-blocking. Does not wait for results for the http request to return. When session state replication is allowed to have some latency, this is useful and will have a much better performance throughput. In the future release, we will also add the following features from JBossCache. (sub)-Partitioning. This feature will partition the cluster into different sub-partitions such that replication only happens in the sub-partition nodes. With this feature, the session replication cluster can then be easily scaled. Persistence. Persist the session data to the backed store. Currently, clustered session does not support any persistency. Eviction policy and cache data overflow memory size or time-based eviction policy with overflow to back end persistent store. This is useful for users who have long running sessions that needs to support passivation.

The following features have been deprecated in the new release: replication-type in jboss-web.xml. User should directly configure the replication type from jboss-web-cluster-service.xml with the element CacheMode. UseLocalCache in jbossweb-tomcat5.0/META-INF/jboss-service.xml. Local in-vm session is used by default now.

JBossCache can be configured through a xml configuration and deployed under JBoss3.2.6 as a MBean service. Since JBossCache is used not only for clustering but also for SSO (single sign on) replication as well, we have introduced a JBossCache clustering MBean service for everyone to share (such as SSO replication). The xml configuration file will be called jboss-web-cluster-service.xml. Once the file is deployed, a MBean called TomcatClusteringCache will be created. In the file, there are JBossCache configuration parameters but customized for the JBossWeb clustering. Here is a snippet of the xml: <server> <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TomcatClusteringCache"> <depends>jboss:service=Naming</depends> <depends>jboss:service=TransactionManager</depends> <!-- Configure the TransactionManager. Default should be JBossTransactionManager. --> <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute> <!-- Valid modes are LOCAL REPL_ASYNC REPL_SYNC --> <attribute name="CacheMode">REPL_ASYNC</attribute> <!-- Name of cluster. Needs to be the same for all clusters, in order to find each other --> <attribute name="ClusterName">TOMCAT-Cluster</attribute> <!-- JGroups protocol stack properties. Can also be a URL, e.g. file:/home/bela/default.xml <attribute name="ClusterProperties"></attribute> --> <attribute name="ClusterConfig"> <config> <UDP mcast_addr="228.1.2.5" mcast_port="45566" ip_ttl="64" ip_mcast="true" mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000" loopback="false" /> ... more config params here ... </config> </attribute> </mbean> </server>

All http session clustering test cases are located under JBoss-3.2/testsuite/src/main/org/jboss/test/cluster/apache-tomcat directory.

In this section, we describe how to setup a sample web application for http session replication. User should refer to other documentation on how to setup a front end load balancer (e.g., Apache mod_jk(2)). To run a web application session replication under the new JBossWeb clustering, you will need to set the application to be distributable as usual. For example, in web.xml (under the web app WEB-INF directory), <web-app> <distributable/> ... </web-app> Next, you can create an optional jboss-web.xml (also under WEB-INF directory) that has the following options: <jboss-web> <replication-config> <replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger> <replication-granularity>SESSION</replication-granularity> </replication-config> </jboss-web> In addition, you will need to start jboss from all configuration directory. Under the new all configuration, it includes the JBossCache library and also the jboss-web-cluster-service.xmlfile to deploy the cache as a specific JBossWeb clustering MBean service. Finally, before you start JBoss, you can edit jboss-service.xml under jbossweb-tomcat5.0/META-INF directory for the following: <jboss-service> <attribute name="SnapshotMode">instant</attribute> <attribute name="UseJK">false</attribute> </jboss-service> You are now ready to use http session replication!