Stopping the Intelligence Event Consumers

Applies only when you have Intelligence in your environment

You will need to stop the Intelligence consumers after they have read past the offsets that the database has read. Then you must remove older events so the database does not receive duplicates after the upgrade. Although this procedure prevents the database from receiving duplicate events, Intelligence might receive duplicate events but then it would automatically de-duplicate them.

Complete the following steps to prevent the loss of events during the upgrade. This procedure needs the db_scheduler_offset.csv file that you created in Step 6.

1. Log in to a Transformation Hub node. Ensure that you have copied the db_scheduler_offset.csv script to this node.
2. Create an executable shell script file, named compare-current-offsets.sh, with the following content:

   #!/bin/bash
   dof="$1"
   if [[ -z $dof || ! -f "$dof" ]]; then
        echo "Database offsets not found: $vof"
        exit 1
   fi
   NS=$(kubectl get ns | awk '/^arcsight-installer/{print $1}')
   t="mf-event-avro-enriched"
   ig="interset-logstash-es"
   ic=Intelligence
   tmp=$(mktemp -u --tmpdir offsets.XXXXX)
   printf "\nGetting current %s offsets for Intelligence\n\n" $t
   kubectl exec -it -n "$NS" th-kafka-0 -- bash -c "kafka-consumer-groups --bootstrap-server localhost:9092 --offsets --describe --all-groups" | grep "$t\s" | sed -re "s/^${ig}/$ic/" | awk '{c=$1; p=$3; o=$4; printf( "%s,%s,%s\n", p, o, c );}' | sort -n > "$tmp"
   partns=$( cut -d',' -f1 "$tmp" | sort -u)
   for p in $partns ; do for c in $ic ; do
        vo=$( awk -v FS="," "/^$p,.*,DB/"'{print $2}' $dof)
        if [[ -n $vo ]]; then
            desc="no $c offset data"
            co=$(awk -v FS="," "/^$p,.*,${c}$/"'{print $2}' "$tmp")
   	  if [[ -n $co && $co =~ ^[0-9]+$ ]]; then
                  lead=$(( co - vo ))
                  desc="$c is $(( - lead )) messages behind the db"
   		if [[ $lead -ge 0 ]]; then
   			desc="$c is caught up to the db and is $lead messages ahead"
   		fi
   	fi
   	printf "partition %d: %s\n" "$p" "$desc"
        fi
   done ; done
   rm -f $tmp
   			

3. Using the CSV file and script that you created previously, run the following command:

   ./compare-current-offsets.sh db_scheduler_offset.csv

   This command generates output that shows the offsets between the database and the data consumers for all partitions:

   Getting current offsets for Intelligence

   partition 0: Intelligence is caught up to the db and is 100231 messages ahead

   partition 1: Intelligence is caught up to the db and is 100123 messages ahead

   partition 2: Intelligence is 1002 messages behind the db

   partition 3: Intelligence is caught up to the db and is 100141 messages ahead

   partition 4: Intelligence is caught up to the db and is 100343 messages ahead

   partition 5: Intelligence is caught up to the db and is 100999 messages ahead

4. Monitor the output from the command, which can indicate the following scenarios:

   • Whether a consumer has caught up to the database offset. For example, Intelligence is reading the mf-event-avro-enriched topic events ahead of the database:

     partition 0: Intelligence is caught up to the db and is 9995 messages ahead

   • Whether the database offset is ahead of the consumers reading the mf-event-avro-enriched topic. For example, Intelligence has not caught up to the database:

     partition 0: Intelligence is 1002 messages behind the db

   • Whether a consumer is not reading the mf-event-avro-enriched topic. For example, when this occurs, the output indicates no offset data:

     partition 0: no Intelligence offset data

   If a capability is not reading messages from mf-event-avro-enriched, you do not need to compare the offsets.

5. When Intelligence has caught up to the database, complete the following steps to stop the Intelligence consumers:
   1. Log in the master node of the cluster.
   2. Run the following command:

      NS=$(kubectl get namespaces | awk '/arcsight-installer/{print $1}')
      kubectl scale statefulset interset-logstash -n $NS  --replicas=0

   3. Run the following command periodically until you see a response indicating that none of the instances are ready:

      kubectl -n $NS get sts interset-logstash

      For example, you would want to see a response that indicates 0 out of 3 instances are ready:

      
   4. In the READY column, note the second number, which represents the number of instances. For example, 3. You will need this number for the "Restarting the Event Consumers" procedure.