Managing threads and connections

This section discusses the use of multiple threads in client programs and object implementations, and will help you understand the VisiBroker thread and connection model.

Using threads

A thread, or a single sequential flow of control within a process, is also called a lightweight process that reduces overhead by sharing fundamental parts with other threads. Threads are lightweight so that there can be many of them present within a process.

Using multiple threads provides concurrency within an application and improves performance. Applications can be structured efficiently with threads servicing several independent computations simultaneously. For example, a database system may have many user interactions in progress while at the same time performing several file and network operations.

Although it is possible to write the software as one thread of control moving asynchronously from request to request, the code may be simplified by writing each request as a separate sequence, and letting the underlying system handle the synchronous interleaving of the different operations.

Multiple threads are useful when:

•	There are groups of lengthy operations that do not necessarily depend on other processing (like painting a window, printing a document, responding to a mouse-click, calculating a spreadsheet column, signal handling).

•	There will be few locks on data (the amount of shared data is identifiable and small).

•	The task can be broken into various responsibilities. For example, one thread can handle the signals and another thread can handle the user interface.

Thread and connection management occurs within the scope of an entity known as a server engine. Several default server engines are created automatically by VisiBroker, which include thread pool engines for IIOP, for LIOP, and so forth. Additional server engines can be used and created in a VisiBroker server by applications. See the example in <install_dir>/examples/vbe/poa/server_engine_policy/Server.C

Server engines are created, configured, and used independently. The creation and configuration of one server engine does not affect other server engines in the same server. Usually, each server engine has one transport end point, called the listen point/socket.

The relationship between server engines and POAs is many-to-many. Each server engine can be used by multiple POAs, and each POA may also use multiple server engines.

Server engines can consist of multiple Server Connection Managers (SCMs). An SCM is composed of managers, listeners, and dispatchers. The properties of managers, listeners and dispatchers can be configured to determine how the SCM functions. These properties are discussed in “Setting connection management properties”.

Listener thread, dispatcher thread, and worker threads

Each server engine has a listener and a dispatcher thread. The listener thread is responsible for:

•	Accepting new connections. Therefore, it listens on the listen end-point.

•	Monitoring readability on idle GIOP connections.

•	Updating the monitoring list.

•	Idle connection garbage collection based on property settings.

The dispatcher determines which threads to send requests.

Each server engine uses a certain number of worker threads to receive and process requests. Different requests may handled by different worker threads. For a given request, the request reading, processing (include server side interceptor intercepting), and replying are all handled by the same thread. The number of worker threads used by a server engine depends on:

•	The thread model.

•	The number of concurrent requests or connections.

•	The property settings.

Thread policies

The two major thread models supported by VisiBroker are the thread pool (also known as thread-per-request, or TPool) and thread-per-session (also known as thread-per-connection, or TSession). Single-thread and main-thread models are not discussed in this document. The thread pool and thread-per-session models differ in these fundamental ways:

•	Situation in which they are created

•	How simultaneous requests from the same client are handled

•	When and how threads are released

The default thread policy is the thread pool. For information about setting thread-per-session or changing properties for the thread pool model, see “Setting dispatch policies and properties”.

Thread pool policy

When your server uses the thread pool policy, it defines the maximum number of threads that can be allocated to handle client requests. A worker thread is assigned for each client request, but only for the duration of that particular request. When a request is completed, the worker thread that was assigned to that request is placed into a pool of available threads so that it may be reassigned to process future requests from any of the clients.

Using this model, threads are allocated based on the amount of request traffic to the server object. This means that a highly active client that makes many requests to the server at the same time will be serviced by multiple threads, ensuring that the requests are quickly executed, while less active clients can share a single thread, and still have their requests immediately serviced. Additionally, the overhead associated with the creation and destruction of worker threads is reduced, because threads are reused rather than destroyed, and can be assigned to multiple new connections.

VisiBroker conserves system resources by dynamically allocating the number of threads in the thread pool based on the number of concurrent client requests by default. If the client becomes very active, new threads are allocated to meet its needs. If threads remain inactive, VisiBroker releases them, only keeping enough threads to meet current client demand. This enables the optimal number of threads to be active in the server at all times.

The size of the thread pool grows based upon server activity and is fully configurable, either before or during execution, to meet the needs of specific distributed systems. With the thread pool model, you can configure the following:

•	Maximum and minimum number of threads

•	Maximum idle time

Each time a client request is received, an attempt is made to assign a thread from the thread pool to process the request. If this is the first client request and the pool is empty, a thread will be created. Likewise, if all threads are busy, a new thread will be created to service the request.

A server can define a maximum number of threads that can be allocated to handle client requests. If there are no threads available in the pool and the maximum number of threads have already been created, the request will block until a thread currently in use has been released back into the pool.

Thread pool is the default thread policy. You do not have to set up anything to define this environment. If you want to set properties for the thread pool, see “Setting dispatch policies and properties”.

Figure 9

Pool of threads is available

The figure above shows the object implementation using the thread pool policy. As the name implies, there is an available pool of worker threads in this policy.

Figure 10

Client application #1 sends a request

In the above figure, Client application #1 establishes a connection to the Object Implementation and a thread is created to handle requests. In the thread pool, there is one connection per client and one thread per connection. When a request comes in, a worker thread receives the request; that worker thread is no longer in the pool.

A worker thread is removed from the thread pool and is always listening for requests. When a request comes in, that worker thread reads in the request and dispatches the request to the appropriate object implementation. Prior to dispatching the request, the worker thread wakes up one other worker thread which then listens for the next request.

Figure 11

Client application #2 sends a request

As the above figure shows, when Client application #2 establishes its own connection and sends a request, a second worker thread is created. Worker thread #3 is now listening for incoming requests.

Figure 12

Client application #1 sends a second request

The above figure shows that when a second request comes in from Client application #1, it uses worker thread #4. Worker thread #5 is spawned to listen for new requests. If more requests came in from Client application #1, more threads would be assigned to handle them, each spawned after the listening thread receives a request. As worker threads complete their tasks, they are returned to the pool and become available to handle requests from any client.

Thread-per-session policy

With the thread-per-session (TSession) policy, threading is driven by connections between the client and server processes. When your server selects the thread-per-session policy, a new thread is allocated each time a new client connects to a server. A thread is assigned to handle all the requests received from a particular client. Because of this, thread-per-session is also referred to as thread-per-connection. When the client disconnects from the server, the thread is destroyed. You may limit the maximum number of threads that can be allocated for client connections by setting the vbroker.se.iiop_ts.scm.iiop_ts.manager.connectionMax property.

Figure 13

Object implementation using the thread-per-session policy

The above figure shows the use of the thread-per-session policy. The Client application #1 establishes a connection with the object implementation. A separate connection exists between Client application #2 and the object implementation. When a request comes in to the object implementation from Client application #1, a worker thread handles the request. When a request from Client application #2 comes in, a different worker thread is assigned to handle this request.

Figure 14

Second request comes in from the same client

In the above figure, a second request has come in to the object implementation from Client application #1. The same thread that handles request 1 will handle request 2. The thread blocks request 2 until it completes request 1. (With thread-per-session, requests from the same Client are not handled in parallel.) When request 1 has completed, the thread can handle request 2 from Client application #1. Multiple requests may come in from Client application #1. They are handled in the order that they come in; no additional threads are assigned to Client application #1.

Connection management

Overall, VisiBroker's connection management minimizes the number of client connections to the server. In other words there is only one connection per server process which is shared. All requests from a single client application are multiplexed over the same connection, even if they originate from different threads. Additionally, released client connections are recycled for subsequent reconnects to the same server, eliminating the need for clients to incur the overhead of new connections to the server.

In the following scenario, a client application is bound to two objects in the server process. Each bind() shares a common connection to the server process, even though the bind() is for a different object in the server process.

Figure 15

Binding to two objects in the same server process

The following figure shows the connections for a client using multiple threads that has several threads bound to an object on the server.

Figure 16

Binding to an object in a server process

As the above figure shows, all invocations from all threads are serviced by the same connection. For that scenario, the most efficient multi threading model to use is the thread pool model. If the thread-per-session model is used in this scenario, only one thread on the server will be allocated to service all requests from all threads in the client application, which could easily result in poor performance.

The maximum number of connections to a server, or from a client, can be configured. Inactive connections will be recycled when the maximum is reached, ensuring resource conservation.

ServerEngines

Thread and connection management on the server side is performed by ServerEngines, which can consist of one or more Server Connection Managers (SCMs). An SCM is a collection of properties of the manager, listener, and dispatcher.

Defining a ServerEngine consists of specifying a set of properties in a properties file. For example, if on UNIX the property file called myprops.properties is in home directory, the command line is:

prompt> vbj -DORBpropStorage=~/myprops.properties myServer

ServerEngine properties

The set of Server Connection Managers associated with a ServerEngine is defined by the property:

vbroker.se.<srvr_eng_name>.scms=<srvr_connection_mngr_name1>,<srvr_connection_mngr_name2>

The name specified in the above property as the <srvr_eng_name> is the name of the ServerEngine. The SCMs listed here will be the list of SCMs for the associated server engine. SCMs cannot be shared between ServerEngines. However, ServerEngines can be shared by multiple POAs.

The other properties are

vbroker.se.<se>.host

The host property is the IP address for the server engine to listen for messages.

vbroker.se.<se>.proxyHost

The proxyHost property specifies the proxy IP address to send to the client in the case where the server does not want to publish its real hostname.

Setting dispatch policies and properties

Each POA in a multi-threaded object server can choose between two dispatch models: thread-per-session or thread pool. You choose a dispatch policy by setting the dispatcher.type property of the ServerEngine.

vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=
ThreadPool
vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=
ThreadSession

For more information about these properties see “Using POAs” and the online VisiBroker for C++ API Reference.

Thread pool dispatch policy

ThreadPool (thread pooling) is the default dispatch policy when you create a POA without specifying the ServerEnginePolicy.

For ThreadPool, you can set the following properties:

•	vbroker.se.default.dispatcher.tp.threadMax

This property sets a TPool server engine's maximum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property

vbroker.se.default.dispatcher.tp.threadMax=32

or

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=32

sets the initial maximum worker thread limitation to 32 for the default TPool server engine. The default value of this property is unlimited (0). If there are no threads available in the pool and the maximum number of threads have already been created, the request is blocked until a thread currently in use has been released back into the pool.

•	vbroker.se.default.dispatcher.tp.threadMin

This property sets a TPool server engine's minimum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property

vbroker.se.default.dispatcher.tp.threadMin=8

or

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=8

sets the initial worker thread minimum number to 8 for the default TPool server engine. The default value of this property is 0 (no worker threads).

•	vbroker.se.default.dispatcher.tp.threadMaxIdle

This property sets a TPool server engine's idle thread check interval. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property

vbroker.se.default.dispatcher.tp.threadMaxIdle=120

or

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=120

sets the initial idle worker thread check interval to 120 seconds for the default TPool server engine. The default value of this property is 300 seconds. With this setting, the server engine will check the idle state of each worker thread every 120 seconds. If a worker thread has been idle across two consecutive checks, it will be recycled (terminated) at the second check. Therefore, the actual idle thread garbage collection time is between 120 to 240 seconds under the above setting, instead of exactly 120 seconds.

•	vbroker.se.default.dispatcher.tp.coolingTime

The ThreadPool dispatcher allows a “cooling time” to be set. A thread is said to be “hot” when the GIOP connection being served is potentially readable, either upon creation of the connection or upon the arrival of a request. After the cooling time (in seconds), the thread can be returned to the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the startup property

vbroker.se.default.dispatcher.tp.coolingTime=6

or

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=6

sets the initial cooling time to 6 seconds for the default engine (or the IIOP TPool server engine).

The default value of this property is 3 seconds. The maximum value is 10 seconds.

Note

The vbroker.se.default.xxx.tp.xxx property is recommended when vbroker.se.default=iiop_tp. When using with ThreadSession, it is recommended that you use the vbroker.se.iiop_ts.scm.iiop_ts.xxx property.

Thread-per-session dispatch policy

When using the ThreadSession as the dispatcher type, you must set the se.default property to iiop_ts.

vbroker.se.default=iiop_ts

Note

In thread-per-session, there are no threadMin, threadMax, threadMaxIdle, and coolingTime dispatcher properties. Only the Connection and Manager properties are valid properties for ThreadSession.

Coding considerations

All code within a server that implements the VisiBroker ORB object must be thread-safe. You must take special care when accessing a system-wide resource within an object implementation. For example, many database access methods are not thread-safe. Before your object implementation attempts to access such a resource, it must first lock access to the resource using a synchronized block.

If serialized access to an object is required, you need to create the POA on which this object is activated with the SINGLE_THREAD_MODEL value for the ThreadPolicy.

Setting connection management properties

The following properties are used to configure connection management. Properties whose names start with vbroker.se are server-side properties. The client side properties have their names starting with vbroker.ce.

Note

The command line options for VisiBroker 3.x backward-compatibility are less obvious in terms of whether they are client-side or server-side. However, the connection and thread management options that start with the -ORB prefix set the client-side options whereas the options with the -OA prefix are used for the server-side options. There are no common properties which are used for both client-side and server-side thread and connection management.

The distinction between client and server vanishes if callback or bidirectional GIOP is used.

•	vbroker.se.default.socket.manager.connectionMax

This property sets the maximum allowable client connections to a server engine. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property

-Dvbroker.se.default.socket.manager.connectionMax=128

or

-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax=128

sets the initial maximum connection limitation on this server engine to 128. The default value of this property is unlimited (0 [zero]). When the server engine reaches this limitation, before accepting a new client connection, the server engine needs to reuse an idle connection. This is called connection swapping. When a new connection arrives at the server, it will try to detach the oldest unused connection. If all the connections are busy, the new connection will be dropped. The client may retry again until some timeout expires.

•	vbroker.se.default.socket.manager.connectionMaxIdle

This property sets the maximum length of time an idle connection will remain open on a server engine. The property can be set statically on server startup or dynamically reconfigured using property API. For instance, the start up property

-Dvbroker.se.default.socket.manager.connectionMaxIdle=300

or

-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=300

sets the initial idle connection maximum lifetime to 300 seconds. The default value of this property is 0 (unlimited). When a client connection has been idle longer than this value, it becomes a candidate for garbage collection.

•	vbroker.ce.iiop.ccm.connectionMax

Specifies the maximum number of the total connections within a client. This is equal to active connections plus the ones that are cached. The default value of zero means that the client does not try to close any of the old active or cached connections. If a new client connection will result in exceeding the limit set by this property, the VisiBroker for C++ will try to release one of the cached connections. If there are no cached connections, it will try to close the oldest idle connection. If both of them fail, the CORBA::NO_RESOURCE exception will result.

Valid values for applicable properties

The following properties have a fixed set or range of valid values:

•	vbroker.ce.iiop.ccm.type=Pool

Currently, Pool is the only supported type.

In the following properties, xxx is the server engine name and yyy is the server connection manager name:

•	vbroker.se.xxx.scm.yyy.manager.type=Socket

Other possible values are Local for LIOP and BIDIR for bidir (bidirectional) SCMs.

•	vbroker.se.xxx.scm.yyy.listener.type=IIOP

You can also use the value LIOP for local IPC and SSL for security

.

•	vbroker.se.xxx.scm.yyy.disptacher.type=ThreadPool

The other possible values are ThreadSession and MainThread.

•	vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime

The default value is 3, and the maximum value is 10, so a value greater than 10 will be clamped to 10.

Effects of property changes

The effect of a change in a property value depends on the actions associated with the properties. Most of the actions are directly or indirectly related to the utilization of system resources. The availability and restrictions of the system resources to the CORBA application vary depending on the system and the nature of the application.

For instance, increasing the garbage collector timer may increase the system activities, as the garbage collector will run more frequently. On the other hand, increasing its value means the idle threads will remain in system unclaimed for longer periods of time.

Dynamically alterable properties

The following properties can be changed dynamically and the effect will be immediate unless stated otherwise:

vbroker.ce.iiop.ccm.connectionCacheMax=5
vbroker.ce.iiop.ccm.connectionMax=0
vbroker.ce.iiop.ccm.connectionMaxIdle=360
vbroker.ce.iiop.connection.rcvBufSize=0
vbroker.ce.iiop.connection.sendBufSize=0
vbroker.ce.iiop.connection.tcpNoDelay=false
vbroker.ce.iiop.connection.socketLinger=0
vbroker.ce.iiop.connection.keepAlive=true
vbroker.ce.liop.ccm.connectionMax=0
vbroker.ce.liop.ccm.connectionMaxIdle=360
vbroker.ce.liop.connection.rcvBufSize=0
vbroker.ce.liop.connection.sendBufSize=0
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax=0
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=0
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=0
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=100

The new dispatcher threadMax properties will be reflected after the next garbage collector run.

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=300
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=3
vbroker.se.iiop_tp.scm.iiop_tp.manager.garbageCollectTimer=30
vbroker.se.liop_tp.scm.liop_tp.listener.userConstrained=false

Determining whether property value changes take effect

For this purpose, the server manager needs to be enabled, using the property vbroker.orb.enableServerManager=true, and the properties can be obtained through the server manager query either through the Console or through a command-line utility.

Impact of changing property values

It is very difficult to determine the impact of changing the value of a property to something other than the default. For thread and connection limits, the available system resources vary depending on the machine configuration and the number of other processes running. The setting of properties allows performance tuning for a given system.

Garbage collection

A dispatcher's thread pool in VisiBroker has an idle timeout vbroker.se.xxx.scm.xxx.dispatcher.threadMaxIdle. The default value is 300 seconds, and after the idle timeout expires the dispatcher will remove any idle worker threads in the thread pool.

A Server Connection Manager (SCM) has its own garbage collection timeout vbroker.se.xxx.scm.xxx.manager.garbageCollectTimer. The default value is 30 seconds, and after the timeout expires any idle connections are garbage collected.

Since the SCM only garbage collects idle connections, the property vbroker.se.xxx.scm.xxx.manager.connectionMaxIdle needs to be set greater than 0 (zero) in order for connections to go to an idle state. The default value is 0 (zero), which means that a connection is never considered idle and nothing is collected, even if the SCM's garbage collection timeout expires.

The dispatcher and the SCM perform garbage collection independently and there is no garbage collection performed by the ORB itself. Hence given the values below.

vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=5
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=5
vbroker.se.iiop_tp.scm.iiop_tp.manager.garbageCollectTimer=10

When the thread pool worker thread, T1, has been idle for 5 seconds it is immediately removed from the dispatcher's thread pool. The connection, C1, which has been idle for 5 seconds is only garbage collected by the SCM after 10 seconds.

Figure 17

Collection of resources by the SCM GC

On the Client side the Client Connection Manager's (CCM) cached connections can be given an idle timeout by setting the property vbroker.ce.xxx.ccm.connectionMaxIdle. The default value is 0 (zero), meaning that the cached connections do not have an idle timeout. Given an idle timeout, the idle cached connections in the connection pool/cache are marked eligible for garbage collection. Unlike the SCM, the CCM has no garbage collection timer, however whenever any connection is being cached it will attempt to garbage collect any cached connections that are marked eligible for collection.