VisiTelcoLog Guide : Understanding the Log interface

Understanding the Log interface
Log characteristics are the same for both event-based log objects and basic log objects. These characteristics are captured in the DsLogAdmin::Log interface. All log objects inherit from this interface and therefore have common characteristics.
In this chapter, the following topics will be explained:
Log and Typed Log records
When an event aware or event unaware application uses the VisiTelcoLog Service to write records to logs using push, pull, or write_record operations, for each received event or each CORBA Any in the Any sequence a LogRecord is created. Similarly, TypedLogRecord is the log record created for each typed event received.
The LogRecord and TypedLogRecord structures are described in the following IDL snippet.
struct LogRecord
{
RecordId id;
TimeT time;
NVList attr_list;
any info;
};

struct TypedLogRecord
{
RecordId id;
TimeT time;
NVList attr_list;
RepositoryId interface_id;
Identifier operation_name;
ArgumentList arg_list;
};
For more detailed structure definitions, please see the OMG Telecom Log Service Specification.
In the structures given in the IDL snippet above, RecordId id is a unique number assigned to the record by the log and is unique in the log only.
TimeT time is the time stamp for the record, when the record was written to the underlying back end.
NVList attr_list can store a list of user-defined attributes for each log record. The attributes are not attached to the log records at the time of writing, but using separate set_attribute() API. See “Setting log attributes” for further information on setting attributes.
The log data itself is stored in the CORBA Any. For typed events, the log data is encapsulated in the argument list for the typed event operation.
RepositoryId interface_id and Identifier operation_name are the repository ID of the interface and the operation name of the operation that emitted the typed event.
Log Quality of Service
In compliance with OMG Telecom Log Service Specification, VisiTelcoLog Service provides a lightweight Quality of Service framework with set_log_qos() and get_log_qos() APIs. This is in addition to the extensive quality of service framework of the Notification Service specification.
VisiTelcoLog Service supports the following Quality of Service properties:
When this is specified no Quality of Service is promised. Calling flush() operation will not flush log records.
When this is specified, calling flush() will flush/commit all the log records to the back end.
VisiTelcoLog Service takes only the highest value of the Quality of Service specified in the set_log_qos() operation. For example, if all the three Quality of Service properties are specified, then only QoSReliability is taken. This is reflected in get_log_qos() operation. The following code snippet illustrates this point.
C++
DsLogAdmin::QosList qos;
qos.length(3);
qos[0] = DsLogAdmin::QoSNone;
qos[1] = DsLogAdmin::QoSFlush;
qos[2] = DsLogAdmin::QoSReliability;

// set all the three QoS
basic_log->set_log_qos(qos);

// Only QoSReliability
qos = basic_log->get_log_qos();
Log size and manipulation
This section explains how to control the log size, determine the log full action, and control log record life.
Controlling the log size
The maximum size (in bytes) of the log can be specified at log creation time. All the log factory log creation operations take a log size parameter (see the code snippets in “Using log factories” for examples). Log size is the maximum size the log can grow to. Size of 0 (zero) means that there is no predefined limit, and the log can grow indefinitely. Once the size has been set it can be altered again by using the set_max_size() and get_max_size() operations. The maximum size of the log is different from current size. Current size is the number of bytes taken up by the log records.
Calling the set_max_size() with a new value less than the current size of the log throws InvalidParam user exception. Calling set_max_size() with any value less than 1 MB will also throw InvalidParam. A minimum of 1 MB is required for the maximum size value. This is an implementation limit. Attempting to create a log with initial maximum size less than 1 MB will automatically set the maximum size to 1 MB.
Log full action
If the current size of the log reaches the maximum size, then the log is said to be in a log full condition. Under such a log full condition, VisiTelcoLog Service specifies the log full action that needs to be taken. The default log full action of any log is specified when the log is created.
By calling set_log_full_action(), the action to be taken in a log full condition can be specified to wrap or halt the log. When the log full action is wrap, the oldest log records are deleted until there is enough space that the new log record can be written.
When the log full action is halt, and if the maximum record life for the log is specified, then all the log records that have expired are deleted from the log. Once the expired records are deleted the write operation attempt is repeated. If the write fails again appropriate exceptions are thrown. See “Logging for event aware applications” and “Logging for event unaware applications” for the exceptions thrown and detail on write operations.
Log record life
Log record life can be specified by the set_max_record_life() API, with units in seconds. Specifying a value of 0 (zero) for maximum record life creates a condition where no log records ever expire.
If the log record life is specified, a garbage collector thread will attempt to delete all expired log records periodically. By default the garbage collector thread starts every 60 minutes. The time interval for this thread can be configured using the property vbroker.dslog.backend.garbageCollectorInterval.
Setting log attributes
In compliance with OMG Telecom Log Service Specification, VisiTelcoLog Service allows client applications to define an attribute list of name-value pairs that are meaningful to the application for log records. These log record attributes (as shown in the log record structure) are readable and writable.
Using the log record ID or grammar and constraint, attributes can be set or retrieved for log records. Using the set_record_attribute() API, attributes can be set on log records based on log record ID. Similarly, using the set_records_attribute() API, attributes can be set on multiple log records which meet the constraint expression specified in the grammar and constraint parameters.
Please note that VisiTelcoLog Service is optimized for log writing. For this reason these operations are comparatively expensive. While setting attributes, the entire log is copied and then replaced.
Copying logs
In compliance with OMG Telecom Log Service Specification, VisiTelcoLog Service provides two copy operations to make a copy of an existing log object. The copy() operation creates an empty log with similar characteristics as the original log. The log ID of the new log object copy is returned in the out parameter.
The copy_with_id() operation takes a log ID and creates an empty log with the input log ID with characteristics similar to the original log. If a log with the input log ID already exists, the LogIdAlreadyExists user exception is thrown. Both of the operations throw NO_RESOURCES system exception if the log factory cannot create a new log because of resource constraints.
Log record query, retrieval and iterators
In compliance with OMG Telecom Log Service Specification, VisiTelcoLog Service provides two methods to query for log records:
The retrieve method retrieves records based on time.
The query method retrieves records based on constraint.
For typed log records the corresponding methods are:
The typed_retrieve method retrieves records based on time.
The typed_query method retrieves records based on constraint.
The retrieve and query methods return an iterator as an out parameter to handle large record retrievals. Please note that the query and retrieve operations are sequential in nature, and they may be time consuming if the number of log records is very large.
Retrieving records based on time
The Log interface provides the retrieve() and typed_retrieve() methods to perform queries based on time. You can also specify how many records in sequence forwards or backwards to retrieve from the specified time. An iterator may be provided to handle large record retrievals. The following code snippet is an example of how to retrieve records based on time.
C++
DsLogAdmin::TimeT from_time;
DsLogAdmin::RecordList_var time_recs;
DsLogAdmin::Iterator_var time_itr;
...
// Starting from 'from_time' retrieve 10 records backwards (i.e -10).
// Store any remaining records in an Iterator 'time_itr'
// if the number of records to retrieve is greater than 1000
time_recs =
log->retrieve( from_time, -10, time_itr.out() );
...
Java
org.omg.DsLogAdmin.TimeT from_time;
org.omg.DsLogAdmin.RecordList time_recs = null;
org.omg.DsLogAdmin.Iterator time_itr = null;
...
// Starting from 'from_time' retrieve 10 records backwards (i.e -10).
// Store any remaining records in an Iterator 'time_itr'
// if the number of records to retrieve is greater than 1000
time_recs =
log.retrieve( from_time, -10, time_itr );
...
Querying for records based on constraint
The Log interface provides the query() and typed_query() methods to perform queries based on a given constraint. The constraint is based on the VisiBroker VisiNotify Filter Constraint. See Writing Filter Constraint Expressions in the VisiBroker VisiNotify Guide for information about writing constraints using the Extended Trader Constraint Language (Extended TCL). A query call takes in a grammar to use and the constraint expression, and an iterator may be provided to deal with a large number of records.
When you write constraints to query LogRecord or TypedLogRecord structures see “Log and Typed Log records” for their definition.
The following example illustrates how to query using constraints. Note that VisiTelcoLog Service only recognizes the default EXTENDED_TCL as the grammar for constraints.
C++
DsLogAdmin::RecordList_var recs_found;
DsLogAdmin::Iterator_var itr;
...
// Query using the "EXTENDED_TCL" grammar and
// search for log records with an id below 100 "$.id
Java
omg.org.DsLogAdmin.RecordList recs_found = null;
omg.org.DsLogAdmin.Iterator itr = null;
...
// Query using the "EXTENDED_TCL" grammar and
// search for log records with an id below 100 "$.id
Iterators
Iterators are returned by a retrieve() or query() method when a large number of log records is returned. The number of records that a retrieve() or query() method should return before using an iterator is controlled by the vbroker.dslog.getRecMaxList property. If the number of records matched from a query() or a retrieve() operation is greater than the value specified by vbroker.dslog.getRecMaxList the excess matched log records will be added to an iterator. Note that when typed_retrieve() or typed_query() is called a TypedRecordIterator is returned.
A log iterator provides two methods: get() and destroy(). The get() method allows the caller to retrieve the records stored by the iterator. When you call the get() method you need to indicate the position and how many records to obtain from the specified position. Note that the position in the iterator moves forward only, therefore you cannot request values before the position of the last request. Requesting for invalid values will throw an InvalidParam exception.
The following code snippet is an example of how to use an iterator's get() method.
C++
DsLogAdmin::RecordList_var recs_found;
DsLogAdmin::Iterator_var itr;
...
// Query using the "EXTENDED_TCL" grammar and
// search for log records with an id below 100 "$.id
Java
omg.org.DsLogAdmin.RecordList recs_found = null;
omg.org.DsLogAdmin.Iterator itr = null;
...
// Query using the "EXTENDED_TCL" grammar and
// search for log records with an id below 100 "$.id
When an iterator has been exhausted, and we call get() and use the position of the last record in the iterator, the get() method will return an empty log record list to the caller. This indicates that the iterator has been exhausted. The application must ensure that the destroy() method is called in order to destroy the object from the VisiTelcoLog Service.
Deleting log records
The Log interface allows deletion of log records and typed log records using either grammar and constraint expression or by ID. Two APIs, delete_records() and delete_records_by_id(), are provided for this purpose and are described in the following table.
VisiTelcoLog Service optimizes event log records and typed event log record deletion by not deleting them immediately, but marking them as deleted. Over time, the log can become fragmented because of this optimization. Therefore, when the fragmentation exceeds a defined limit, the delete operation automatically triggers defragmentation. The defragmentation limit defaults to 75 percent and can be configured using the vbroker.dslog.backend.file.fragmentationLimit property. Defragmentation logic is essentially a copying operation, where all the log records are re-flowed. Please note that the defragmentation operation is expensive.
The following code snippet illustrates deleting a log record of ID 200 using grammar and constraint expressions. The same thing can also be achieved using delete_records_by_id().
C++
// constraint for log rec with id = 200
const char* grammar = "EXTENDED_TCL";
const char* constraint = "$.id == 200";

// delete the log record matching the constraint
basic_log->delete_records(grammar, constraint);
Java
// delete the log record where the log record id = 200
basic_log.delete_records("EXTENDED_TCL", "$.id == 200");