Message CASPA2104E

If the SEP that processes the initial request detects that one of the regions in the PAC has not been communicating with the PSOR, it will not notify that region of the request (this is to prevent blocking the SEP indefinitely). If this happens, then the SEP will write a CASPA2104E message to the console and continue. The message includes the name of the target region, the type of operation, and the resource on which the operation was to be applied.

This means that the region that was not notified will not process the operation. When this message occurs, an administrator might need to take specific action depending on the results.

If the region is no longer active then the PAC will continue operating. If the region is active but not responding to any requests then it could be holding locks that will affect other regions in the PAC. In this case, you should first collect diagnostic information before performing a forced shutdown of the region to remove any remaining locks.

If the operation is an FCT close, you might receive a message like the following:

CASPA2104E FCT close error for VSO23RW: ITR not sent to RAS2(ras2machine) because it has not pinged the PAC SOR for 0000000482 seconds. 10:55:52

In this case, the FCT VSO23RW will be marked as closed in the PAC and all processes in all regions that successfully processed the close request will have physically closed the file if they had it open and removed any related locks.

The region referenced in the message, in this case RAS2, might not have reconnected to the PAC's SOR. This can be checked using the caspac utility, for example:

caspac -aStatus=<PacName> -sredis,<IP of PSOR>:<Port of PSOR> -n<PAC SOR Name|Sentinel Master Name>

This will output content similar to the following:

Connected to Redis server at 127.0.0.1 port 06741
Region Heartbeat status for endpoint: 127.0.0.1:06741
RAS1(ras1machine)@****::****:****:****:****: 04/03/2020@UTC:11:55:41.088
RAS2(ras2machine)@****::****:****:****:****: 04/03/2020@UTC:10:50:33.044

From this message you can see that RAS2 is no longer sending a heartbeat message to the PAC SOR. If the FCT references a catalogued file and any processes in RAS2 have opened the file, then there will be a shared lock left in the system (where the resource name is the data set name and the queue name is SYSZDSN). This would stop any batch jobs that need to access the same file from running. RAS2 might also interpret that CICS still has the FCT open.

If the region in question is still able to process requests, then check the FCT status in that region:

If it displays an FCT status of open and the other regions in the PAC see it as closed, then send a request to close the FCT directly to that region rather than via the load balancer (where the request might get routed to a different region). This will ensure that the region removes any SYSZDSN locks that it has in place (related to the FCT), and that its local copy of the FCT is marked as closed.
If it displays an FCT status of closed, then the region must have subsequently refreshed its status which means it has reconnected to the PSOR. This could mean that SYSZDSN locks are still held by that region. To resolve this, send a request to the region to open the FCT and then another request to close it.

If the operation is a CICS NEWCOPY or PHASEIN command, then the new copy of the program will not be deployed to the region referenced in the message. If that region regains communication with the PAC, then the command can be reissued directly to that region.