text: Cisco Remote Expert Manager 1.9.6 Troubleshooting and Serviceability Guide - Serviceability Administration [Cisco Remote Expert Manager] - Cisco
  Cisco Remote Expert Manager 1.9.6 Troubleshooting and Serviceability Guide
  REM Troubleshooting
  PDF - Complete Book (1.2 MB) PDF - This Chapter (707.0 KB)
  Chapter: Serviceability Administration
  Errors and Alerts from REAC
  Errors and Alerts from READ
  Errors and Alerts from REIC
  REM Server (RESC) Errors
  Events Tab in the IEM
  Call Related Errors
  Agent Registration When Done From REAC Shows a Red Status
  REIC Displays the Error Message "System error, please try after some time"
  Customer Pod Displays "Expert not Available" Message
  Customer Pod Displays "Kiosk phone is out of service" Message
  Expert Registration Issue
  Rsync Command Failure
  REM Node Failure
  Call Hold and Release from Hold
  Dual Content Issue
  Failure of RESC Web (API) Services
  Verifying VM Settings
  Effect on Database
  TUI Troubleshooting Tools
  System, RE, and Application Server Logs
  Troubleshooting Tools in REAC
  Single and Dual Node Backups
  Dual Node Backup
  Different Scenarios in Backup and Restore Processes
  Backup Sequence Diagram
  Location of Backup and Restore Tools
  Dual Node Setup
  Enabling or Disabling Backup and Restore Tool
  Restoring the REM Server
  Backup Archive Rotation
  This chapter is intended for service engineers and server administrators who are accountable for keeping the REM up and running to its complete usability and performance. The scope of this chapter is restricted to topics that assist administrators with maintaining the REM application. This chapter also explains how to backup and restore the REM.
  "Serviceability Overview"
  – "Errors and Alerts from REAC"
  – "Errors and Alerts from READ"
  – "Errors and Alerts from REIC"
  – "REM Server (RESC) Errors"
  – "Log Monitoring"
  – "Events Tab in the IEM"
  – "Call Related Errors"
  – "JTAPI Issues"
  – "Web Service"
  – "Dual Content Issue"
  – "Failure of RESC Web (API) Services"
  "Verifying VM Settings"
  "TUI Troubleshooting Tools"
  "Troubleshooting Tools in REAC"
  Serviceability administration provides descriptions and procedures for the following functionality:
  Supports daily management of REM applications
  Schedules monitoring and maintenance of the each component
  Captures events for troubleshooting and provides alert/error message definitions
  Saves trace information to various log files for troubleshooting
  Monitors the disk usage of the log partition on a server
  Monitoring involves keeping track of various errors and warning being thrown by various components of the REM. It involves checking the state or health of all REM components for availability and appropriate functioning of all solution features.
  There are various errors and alerts that are thrown by REM which needs to be resolved to keep the REM up and running.
  The tables below document the errors and alerts that could be thrown by the Remote Expert Admin Console (REAC) application and shown on the user interface (UI).
  Table 2-1 Generic REAC Errors
  Unable to contact the web server. Please check web server status.
  Occurs when the REM server is down
  Bring the REM server up by using the TUI: Main Menu > Services Control > Application Server > b) Start Service
  Occurs during Kiosk restart if RESC has encountered an internal error
  Check the RESC log file for the specific error condition by using the TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) RESC Logs: resc.log
  Unable to connect to REM Server
  Occurs during Kiosk restart when REAC is unable to connect to RESC (if RESC is deployed on a separate server)
  1. Bring the REM server up by using the TUI: Main Menu > Services Control > Application Server > b) Start Service
  2. Check the RESC IP by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties > REM Core Properties
  Unable to restart kiosk
  Occurs during Kiosk restart if REM has failed to restart due to issues on the server
  1. Check the Kiosk settings > IEC Serial Number
  2. Perform the same operation again
  3. Restart IEC from IEM
  4. Restart IEC manually
  Illegal input provided
  Occurs during Kiosk restart if Kiosk has been deleted from REM
  Check the REAC log file for the specific error condition by using the TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > d) REAC Logs: reac.log
  Unable to connect to Kiosk
  Occurs during Kiosk restart when REM server is unable to make connection to the IEM
  Verify if the IEM IP address is correct in the REM Properties file by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  HTTP (500): The server reported an Error
  The server has encountered an internal error, which prevented it from fulfilling the request
  1. Check REAC log file for the specific error condition by using the TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > d) REAC Logs: reac.log
  2. Check RESC log file for the specific error condition by using the TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) RESC Logs: resc.log
  An error occurred during File Transfer
  1. Check RESC IP by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties > REM Core Properties
  2. Check the following parameters in REAC PROPERTIES by using the TUI: Main Menu > REM Server Administration > Edit REM Templates > a) REAC Properties:
  remote_server_ip=$*[RESC_IP]*
  remote_server_uname=ENC($*[RESC_SERVER_USER]*)
  remote_server_pwd=ENC($*[RESC_SERVER_PASSW]*)
  Unable to synchronize resources across cluster nodes
  Occurs during file transfer when handshaking between the two nodes fails
  1. Refer to the JTAPI Issues section of this chapter to fix the issue
  2. Upload the file from REAC again
  Table 2-2 Remote Service Call Errors
  Remote call to xxxx failed
  Occurs when REAC is unable to communicate to REM server
  2. Verify if RESC services are available by invoking "https://<REM_IP>:8443/resc/services/listServices"
  Remote call to xxxx returned an empty response
  Occurs when REM server sends invalid response
  Check RESC log file for the specific error condition by using the TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) RESC Logs: resc.log
  Table 2-3 Cluster Management Errors
  The requested operation cannot be executed due to incomplete configuration
  Occurs if the REM server details and DB details are missing in REM configuration file
  1. Verify the values of $*[NODE_IP_1]* and $*[DATABASE_1]* in the REM Properties file by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  2. Use the TUI: Main Menu > REM Server Administration > c) Run Configuration Tool
  The selected operation failed
  Occurs if there is an error while activating the DB
  The selected node ''{0}'' is not found
  1. Verify the values of $*[NODE_IP_1]* and $*[DATABASE_1]* in REM Properties location by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  The errors generated by the READ application can be grouped in two scenarios:
  1. Direct Connect (DC) errors in the READ application
  2. Errors or alerts from the READ application
  Table 2-4 Direct Connect Errors
  DC pops up in READ and after clicking on the Start sharing button but it remains in the Waiting state
  The 'cv_service_url' in DirectConnect.exe.config is incorrect. File direction C:\CSI\DirectConnect\bin\DirectConnect is incorrect
  1) Fix 'cv_service_url' to point to the correct REM IP address
  2) Restart the DC Server from the desktop
  The Expert is not registered in REM
  Verify in the Expert tab in REAC that the expert is added
  DC license has expired. The error displayed in error.log available at C:\CSI\DirectConnect\Server\tomcat\logs\error.log is the following:
  2012-04-13 11:46:47,584 ERROR [/] Exception sending context initialized event to listener instance of class pureweb.servlet.ContextListener
  java.lang.RuntimeException: com.reprisesoftware.rlm.RlmException: License has expired (-3)
  at pureweb.servlet.ContextListener.contextInitialized(ContextListener.java:88) ~[ContextListener.class:na]
  Table 2-5 READ Errors
  READ not initialized
  Occurs when READ is unable to contact RESC
  Verify if services of RESC are available by invoking "https://<REM_IP>:8443/resc/services/listServices"
  Previous collaboration request is still in progress
  Occurs if trying to start a READ module when the current one is still in progress
  In READ, check job status if Document Camera, Scan, Signature Capture, or DirectConnect are running before trying to start another module
  Checking for an active session…
  Occurs if CAD Admin has incorrect values
  Refer to the CAD Administration Guide
  Communication Failure: Please contact Administrator
  Occurs if the REM server goes down during the session
  404 error in document sharing panel for local file upload
  Unable to fetch the url of the uploaded file
  Check if the 'docstore.url' or 'docstore.path' parameter is configured properly in READ properties by using the TUI: Main Menu > REM Server Administration > Edit REM Templates > f) READ properties
  Occurs when printing a local file
  Check if the 'webservice.url' parameter is missing in READ properties by using the TUI: Main Menu > REM Server Administration > Edit REM Templates > f) READ properties
  There are different types of error pages used in REIC. All the error codes with detailed description are tabulated below. A message specific to particular error code is displayed on the REIC screen. If there is no message associated with the error code, REIC displays the error code itself on the touch screen collaboration panel.
  Table 2-6 REIC Errors
  System is not available, Please try after some time
  Occurs due to the following reasons:
  1. If TP details (TP Directory Number or TP Type) is incorrect
  2. TP MAC address is not added in CUCM> Application user
  3. IVR Phone Number is incorrect
  1. Verify the TP details in REAC > Kiosk
  2. Add the TP MAC address in CUCM >Application user
  3. Check IVR Phone Number in REAC > Expert Type
  Kiosk phone is out of service
  Occurs if connection within the Kiosk phone is not available
  1. Verify the values of CUCM in REM Properties by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  2. Kiosk phone is not reachable. Please check the connection status.
  Sorry, there is already an active call to an Expert. Please try again after the current call ends
  Occurs when an expert is engaged in another call
  Make sure that the current call ends on the TP and try again after some time
  Locale File does not exist
  Occurs if the file is missing in the system
  Make sure Message Bundle is added in REAC > Locale
  Kiosk serial number is not configured, cannot start a call
  Occurs when Kiosk serial number is not configured properly
  Verify the IEC Serial Number in REAC > Kiosk
  Selected Locale is not associated to any expert type. Please contact admin
  Occurs if the locale associated to the kiosk is not associated with any expert type
  Associate the Expert Type with locale in REAC > Expert Type
  Agents are not available currently. Please try after sometime
  Occurs when all agents are busy
  Try again after a period of time
  1. TP is down
  2. TP IP address in REAC > Kiosk is incorrect
  1. Power on the TP and restart IEC from the IEM
  2. Edit TP details (TP IP address in REAC > Kiosk)
  REM server is down
  Kiosk is not registered
  Occurs if the particular device number is not added in REAC
  Make sure this kiosk is added in REAC > Kiosk
  Please contact admin. Probable reasons for this error:
  RESC is not running
  Occurs during the loading of Kiosk home page
  1. Verify if services of RESC are available by invoking "https://<REM_IP>:8443/resc/services/listServices"
  2. Verify the values of CUCM in REM Properties by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  3. Restart the REM server by using the TUI: Main Menu > Services Control > Application Server > c) Stop Service > b) Start Service
  Improper loading of REIC
  1. Occurs if the value of $*[RESC_IP]* in rem.properties is incorrect
  2. Expert type is not configured properly in REAC
  3. If Startup URL in IEM is incorrect
  2. Check Expert type in REAC > Expert Type
  3. Check Startup URL in the IEM
  REIC hangs
  Occurs if there are un-terminated sessions
  Terminate sessions in REAC > Clean Call Cache
  REIC is showing Cobra browser page with message "Startup URL is not configured."
  Occurs if the policy is missing in IEM
  Check if the IEM has correct policy assigned to this Kiosk
  REIC is showing "Service Temporarily Unavailable"
  Occurs if the Startup URL in IEM is incorrect or REM server is unable to make connection to the IEM
  1. Verify if the IEM IP address is correct in REM Properties by using the TUI: Main Menu > REM Server Administration > b) Edit REM Properties
  2. Verify correct Startup URL in the IEM policy
  In order to check if RESC is running properly, run this URL in the browser: https://<REM_IP>:8443/resc/services/AdminService/getVersionInfo
  If RESC is deployed or running properly, then you will see a result similar to the following:
  <ns:getVersionInfoResponse>
  <ns:return>REM-1.9.6</ns:return>
  </ns:getVersionInfoResponse>
  If RESC is not deployed or running properly, then you will see a different result such as:
  The service cannot be found for the endpoint reference (EPR) /resc/services/AdminService/getVersionInfo
  Refer to the "Call Related Errors" section of this chapter to troubleshoot these issues.
  The log files for all the REM components are tabulated below.
  Table 2-7 Log Files
  TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) RESC Logs: resc.log
  TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) REAC Logs: reac.log
  Refer to the "Events Tab in the IEM" for REIC log location
  TUI: Main Menu > Troubleshooting > Logs > Remote Expert Logs > a) READ Logs: read.log
  Look into the respective log files mentioned above if there are any errors in any of the REM components.
  The IEM allows you to do the following:
  View all the IECs registered with the IEM
  Check the REIC logs
  To perform the tasks above, follow these steps:
  Step 1 Enter http://<IEM_IP> in a browser.
  Step 2 Enter the credentials and click Enter.
  Step 3 Click Devices on the left navigation pane.
  All the IECs registered with the IEM in your account are visible in both the left and center panes.
  Step 4 Click the Show screenshots button on the top right corner of the display view to view the screenshot of each of the IECs registered with the IEM.
  The screenshots are pulled from the last time that the IECs were polled by the IEM.
  The REIC displays the welcome screen and the dual content TelePresence screen.
  Step 5 Default polling time for each IEC image is 10 minutes. The most recent image for the specific IEC can be obtained by clicking the Refresh button beneath the IEC's screenshot.
  Step 6 In order to enlarge the screenshot of the IEC, click the Zoom button beneath the IEC's screenshot.
  Now you will check the logs in the IEM.
  Step 7 In the Devices menu (left pane), choose the IEC by double-clicking on the icon for that particular IEC.
  Step 8 Click the Events tab.
  Step 9 You can filter the logs related to REIC by checking the browser check box in the Facilities list. Click Apply.
  When call is initiated from REIC, all call flow related information is generated in the database.
  The different flows of a call both successful and failed are:
  The duration of each event can also be seen in the queries.
  All the call related errors for each failed event are listed in the resc.log under the $REM_HOME/resc/logs directory.
  The following are examples of how system administrators can search for the errors using Unix commands:
  1. The error for a particular event can be extracted from the resc.log located under the $REM_HOME/resc/logs directory. Go to the logs folder within the $REM_HOME directory by executing the command: cd $REM_HOME/resc/logs/
  2. Use the grep command to find all Call Connect related errors.
  3. To search for errors at a specified time, use the command vi <log_name>|grep <time>. For example, cat resc.log|grep 00:53:18
  4. To search for errors that occurred on a particular device, use the command vi <log_name>|grep <device> and enter the IEC or TP serial number. For example, cat resc.log|grep 656015030030
  5. To search for call related errors that occurred, use the command vi <log_name>|grep <string>. to fetch all the lines containing the string "call". For example, cat resc.log|grep -i call
  1. Check if the correct CUCM IP and credentials were provided when the REM was setup.
  2. For a call connection to work properly, the JTAPI adapter should be configured correctly.
  a. The IP address and TP details should be configured properly to initiate the call.
  b. To receive the call, the agent DN must be configured in REAC. The registration status of the agent DN is available in the Expert tab of REAC. Refer to the Cisco Remote Expert Manager Administration Guide for instructions on how to configure the agent DN.
  If the parameters above are not configured properly then log messages are logged in the resc.log file under $REM_HOME/resc/logs.
  To check if incorrect CUCM IP or credentials have been setup, check for the error in the resc.log. Check the IP address of CUCM in the rem.properties using the TUI. Go to Main Menu > REM Server Administration > Edit REM Properties.
  If the REIC displays the error message "System error, please try after some time", the TP details (TP DN or TP type) in REAC are incorrect. To fix this issue, refer to the Cisco Remote Expert Manager Administration Guide to configure SSH in REAC.
  The customer pod displays the "Expert not Available" error message when the video endpoint IP Address is incorrect. As a result, the Virtual Agent Services failed to function properly. To fix this issue, refer to the Cisco Remote Expert Manager Administration Guide to configure SSH host name in REAC.
  If the customer pod displays the "Kiosk phone is out of service", the video endpoint IP phone terminals are not registered in CUCM.
  The expert registration on one of the nodes or both is shown as red. This occurs when the REM database is not in sync.
  To correct this issue, perform database synchronization and re-start Postgres on the node where the expert registration has failed by following these steps:
  Step 1 SSH into REM via the TUI.
  Step 2 In the Main Menu, select c) Services Control.
  Step 3 In the Services Control menu, select c) Database Server.
  Step 4 In the Database Server menu, select c) Stop Service and follow the on-screen instructions.
  Step 5 In the Database Server menu, select a) Start Service and follow the on-screen instructions.
  Step 6 Go to REAC and click the DB Cluster tab.
  Step 7 Choose the DB node and click the Activate button.
  An rsync command failure occurs when a RSA fingerprinting issue happens (e.g. it goes out of sync). This occurs during a file transfer when handshaking between the two nodes fail.
  30:2b:2c:e4:34:7d:3f:88:05:50:09:5b:5e:13:1f:fc.
  RSA host key for 10.76.8.192 has changed and you have requested strict checking.
  To fix this solution, remove the known host keys from all the other nodes in the REM HA cluster and set up the rsync utility again by following these steps:
  Step 1 SSH into REM via the TAC account.
  sudo cp /root/.ssh/known_hosts /root/.ssh/known_hosts.bak
  sudo vi /root/.ssh/known_hosts
  Step 3 In the VI editor, delete all entries.
  Step 4 Save the file and exit the VI editor.
  REM node failure occurs when the sequence of REM node restoration is incorrect:
  Incorrect approach of shutting down the nodes: If Node 1 (Database 1 has outdated data), which is the first node to go down, is started first then followed by the startup of Node 2, the online recovery will sync the data from Database 1 (outdated) to Database 2 (updated). With this sequence, you will lose the updated data which was available in Database 2 though both nodes will have synched up data. No operational issue will be experienced but any change to Node 2 when Node 1 was down is lost.
  Correct approach of shutting down the nodes: The correct approach of shutting down the nodes to ensure no data loss is to first start the database that is the last to go down (Database 2 in this case) since it will have extra data. Once Node 2 is up, then start Node 1.
  After both nodes are running, online-recovery (i.e. click the Activate button for the nodes in the DB Cluster tab of REAC) will sync (copy) data from Node 2: Database 2 to Node1: Database 1. This will ensure that both the nodes have updated data and no data is lost.
  Communication failure occurs when the communication between two nodes has failed due to a network issue. In the following text, you can observe the following in the haz.log.
  2013-02-07 09:38:58,646 INFO [hz._hzInstance_1_dev.ServiceThread] com.hazelcast.cluster.ClusterManager - [192.168.2.4]:5701 [dev]
  Member [192.168.2.4]:5701 this
  2013-02-07 09:38:58,650 DEBUG [hz._hzInstance_1_dev.ServiceThread] com.hazelcast.impl.ListenerManager - [192.168.2.4]:5701 [dev] AddListenerOperation from Address[192.168.2.4]:5701, local=true key:null op:REMOVE_LISTENER
  2013-02-07 09:38:59,637 DEBUG [hz._hzInstance_1_dev.cached.thread-5] com.hazelcast.impl.SplitBrainHandler - [192.168.2.4]:5701 [dev] Searching for other clusters.
  2013-02-07 09:38:59,637 DEBUG [hz._hzInstance_1_dev.cached.thread-5] com.hazelcast.impl.TcpIpJoiner - [192.168.2.4]:5701 [dev] Address[192.168.2.3]:5701 is local? False
  2013-02-07 09:38:59,637 DEBUG [hz._hzInstance_1_dev.cached.thread-5] com.hazelcast.impl.TcpIpJoiner - [192.168.2.4]:5701 [dev] Address[192.168.2.4]:5701 is local? true
  2013-02-07 09:38:59,637 DEBUG [hz._hzInstance_1_dev.cached.thread-5] com.hazelcast.impl.TcpIpJoiner - [192.168.2.4]:5701 [dev] Address[192.168.2.4]:5701 is connecting to Address[192.168.2.3]:5701
  2013-02-07 09:38:59,638 DEBUG [hz._hzInstance_1_dev.cached.thread-4] com.hazelcast.nio.SocketConnector - [192.168.2.4]:5701 [dev] Starting to connect to Address[192.168.2.3]:5701
  2013-02-07 09:38:59,638 DEBUG [hz._hzInstance_1_dev.cached.thread-4] com.hazelcast.nio.SocketConnector - [192.168.2.4]:5701 [dev] Connecting to /192.168.2.3:5701, timeout: 0, bind-any: true
  2013-02-07 09:38:59,638 DEBUG [hz._hzInstance_1_dev.cached.thread-4] com.hazelcast.nio.SocketConnector - [192.168.2.4]:5701 [dev] Could not connect to: /192.168.2.3:5701. Reason: ConnectException[Connection refused]
  2013-02-07 09:38:59,638 DEBUG [hz._hzInstance_1_dev.cached.thread-4] com.hazelcast.nio.SocketConnector - [192.168.2.4]:5701 [dev] Connection refused
  at com.hazelcast.nio.SocketConnector.tryToConnect(SocketConnector.java:106)
  at com.hazelcast.nio.SocketConnector.run(SocketConnector.java:55)
  The following log shows that the call from the customer is connected successfully to the agent.
  2013-02-07 09:36:31,153 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3cdf672a)] com.cisco.big.sm.impl.SessionManager - Session Activated :21594114
  2013-02-07 09:36:31,153 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3cdf672a)] com.cisco.big.call.cm.observer.AgentObserver - 1004Adding GCID :2600
  2013-02-07 09:36:31,158 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3cdf672a)] com.cisco.big.call.cm.observer.BaseObserver - Lock Acquired
  2013-02-07 09:36:31,158 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3cdf672a)] com.cisco.big.call.cm.observer.AgentObserver - 1004Session added to Agent and Lock tried
  The following log shows that the call is disconnected successfully.
  2013-02-07 09:11:05,996 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@7ab2f62)] com.cisco.big.call.cm.observer.AgentObserver - 1005 Event : CallObservationEndedEv
  2013-02-07 09:11:05,996 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@7ab2f62)] com.cisco.big.call.cm.observer.AgentObserver - 1005 Call:1036 --> 1005 Call Id :2591
  2013-02-07 09:11:05,996 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@7ab2f62)] com.cisco.big.call.cm.observer.AgentObserver - 1005 Call Key:1036
  The following log shows that the call is placed on hold and is released from hold successfully.
  2013-02-07 09:13:12,037 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3612afd8)] com.cisco.big.sm.impl.SessionManager - Session onHoldSession :21594113
  2013-02-07 09:13:12,037 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3612afd8)] com.cisco.big.call.cm.observer.AgentObserver - Call Put on Hold
  2013-02-07 09:14:11,849 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3612afd8)] com.cisco.big.sm.impl.SessionManager - Session resumeSession :21594113
  2013-02-07 09:14:11,849 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@3612afd8)] com.cisco.big.call.cm.observer.AgentObserver - 1005Call Put on Talking
  The following log shows that the call is transferred successfully.
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallTransferEventHandler - 1004Transfer Sequence Handler Event : CallObservationEndedEv
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallTransferEventHandler – 1004
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - --------------- Agent Observer - DN: 1004 ---------------------
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Event : CiscoTransferEndEv
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Call:1036 --> 1005 Call Id :2593
  2013-02-07 09:33:41,836 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004Connections :1036 1005
  2013-02-07 09:33:41,851 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Call Key:1036
  2013-02-07 09:33:41,851 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004Sequence Handler exists
  2013-02-07 09:33:41,851 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallTransferEventHandler - 1004---------------- Transfer Sequence Destruction Handler ----------------
  2013-02-07 09:33:41,851 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallTransferEventHandler - 1004Detected Parts: Transfer Dn :1005 Kiosk dn :null
  2013-02-07 09:33:41,863 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallTransferEventHandler - Kiosk Dn Assigned
  The following log shows that the call is successfully added to a conference call.
  2013-02-07 09:31:01,637 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - --------------- Agent Observer - DN: 1004
  2013-02-07 09:31:01,637 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Event : CiscoConferenceEndEv
  2013-02-07 09:31:01,637 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Call:1005 --> Unknown Call Id :2593
  2013-02-07 09:31:01,637 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004Connections :1036 1004 1005
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004 Call Key:null
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.AgentObserver - 1004Sequence Handler exists
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallConferenceEventHandler - Received handleDestructionEvent
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallConferenceEventHandler - Destroying this handler
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallConferenceEventHandler - 1004---------------- Conference Sequence Destruction Handler
  2013-02-07 09:31:01,642 INFO [ObserverThread(com.cisco.big.call.cm.observer.AgentObserver@75775dde)] com.cisco.big.call.cm.observer.sequencehandler.CallConferenceEventHandler - 1004 Detected Parts: Target DN :1005 Kiosk DN :1036
  The web service correctly loads if the configurations for web service in rem.properties are correct. The contents of rem.properties are the following:
  $*[REM_VIRTUAL_IP]*= Virtual IP denotes the Cisco ACE Load Balancer Virtual IP. In case of Single Node provide the same IP as that of $*[RESC_IP]*
  $*[RESC_IP]*= Provide the IP address of REM
  # For a Call Flow via CCX set the below property to true
  $*[IS_CCX]*=True, if the call flow is via UCCX. False, if the call flow is via UCCE
  # For a Call Flow via CVP set the below property to true
  $*[IS_CVP]*=False, if the call flow is via UCCX. True, if the call flow is via UCCE
  #CUCM Credentials
  $*[CUCM_HOST]*= Provide CUCM Host name or IP address.It can have multiple addresses separated by comma, if you have CUCM HA setup
  # =================================- HA Properties -==================================
  $*[Total_Nodes_In_Cluster]*=1 if you have single node; 2 if you have REM HA
  # REM Server IP & database details
  $*[NODE_IP_1]*=same as RESC_IP if it is for a single node; private IP of node 1 if it is for a HA setup
  $*[NODE_IP_2]*= ignore this one if it is for a single node; private IP of node 2 if it is for a HA setup
  # RSYNC true in clustered setup, false in single node setup
  $*[RSYNC_ENABLED]*=false if it is for a single node; true if it is for a REM HA
  # REIC Customer logo and background image can be configured using below properties file
  $*[REIC_BIG_CUSTOMER_LOGO]*=This is required to customize the kiosk screen. Refer REIC UI configuration
  $*[REIC_SMALL_CUSTOMER_LOGO]*= This is required to customize the kiosk screen. Refer REIC UI configuration
  $*[REIC_BACKGROUND_IMAGE]*= This is required to customize the kiosk screen. Refer REIC UI configuration
  $*[EXPERT_TYPE_WIDTH]*= This is required to customize the kiosk screen. Refer REIC UI configuration
  $*[REIC_SCREEN_MODULE]*=null is for regular RE; keyboard is for keypad support; magstripe is for card reader
  $*[EXTENSION_MOBILITY]*=false is for regular RE; true is for Connected Justice
  Warning Providing incorrect REAC credentials will result in a file transfer error.
  Warning Providing incorrect CUCM credentials will result in a call connect error due to a JTAPI issue.
  Refer to the "Events Tab in the IEM" section to monitor whether dual content is displaying properly in the secondary screen attached to the IEC. If dual content is not loaded, follow the steps below to troubleshoot the issue.
  Step 1 Check if the dual screen image and time are configured properly in the REAC. (Refer to the Cisco Remote Expert Manager Administration Guide.)
  Step 2 Enter https://<REM_IP>:8443/resc/services/AdminService/getContentByTimeInterval?time=< dd-MM-yyyy HH:mm:ss> in a browser (e.g. https://10.76.8.171:8443/resc/services/AdminService/getContentByTimeInterval?time=22-06-2012 09:52:00).
  Step 3 Verify that the status code indicates "SUCCESS".
  Step 4 Enter https://<REM_IP>:8443/reic/dualcontent.html in the Firefox browser.
  Step 5 Check whether the image loads.
  Note You can use the Firebug tool in Firefox to track the flow in loading the dual content URL.
  Step 6 If the image loaded in the browser, go to the IEM and choose the IEC in question.
  Step 7 Look at the screenshot of the IEC. The REIC application screen and the dual screen image should be displayed in the IEC screenshot pane.
  Sometimes after the reboot of the REM VM, services provided by the REM Server fail to start due to delay in setting up of the $REM_HOME environment variable path.
  In order to circumvent the problem above, perform the following steps after the REM VM server is rebooted:
  Step 1 Run the main.sh file from the bash shell using the following command: #.$REM_HOME/tools/ias/scripts/main.sh
  Step 2 Check whether all the services are up and running by accessing the list of services provided by the RESC server by navigating to the following URL: https://<REM_IP>:8443/resc/services/AdminService/getVersionInfo
  Step 3 If web services are running properly, then you will see a result similar to:
  Installation of a virtual machine is done using the vShpere client. The REM VM setup requires some of the settings to be configured and maintained, such as time zone setup, master configuration setup, NTP and SMTP server configuration, etc. For installation of the VM, refer to the Cisco Remote Expert Manager Installation Guide.
  To troubleshoot time-related errors (e.g. dual content is not working or file timestamp mismatch), REM components configuration-related errors, connectivity errors, or mail-related errors, review the following topics.
  The master configuration file contains all the required settings that are required to get the system up and running. At the time of installation of the REM ISO, some basic inputs are required to be configured before the environment could be prepared for the successful execution of the various REM applications (i.e. REIC, RESC, READ, and REAC) and third-party services like Postgres and Tomcat.
  All the variables or inputs required are enlisted in the Master template which needs to be populated beforehand or by the administrator at the time of installation. The Master template is located at $REM_HOME/tools/ias/templates/rem.properties. Refer to the Cisco Remote Expert Manager Installation Guide.
  NTP setup is part of the REM ISO installation. NTP synchronizes the clock of the local server with the NTP server. This is required for synchronization of files with timestamps. All NTP-related settings can be found in /etc/ntp.conf.
  If you plan to use internal NTP servers, follow the steps below. Otherwise, you may skip this step.
  Step 1 Using the TUI, choose a) System Settings in the Main Menu.
  Step 2 In the System Settings menu, choose b) Date and Time Settings.
  Step 3 In the Date and Time Settings, choose a) Setup NTP Source.
  Step 4 A VI window will open. Replace the IPs or hostname of your NTP servers in the following entries. If you only use one or two NTP servers, comment out the unused entries by adding # at the beginning of lines.
  Step 5 Save and exit the VI.
  Step 1 To set up the network, SSH into REM via TUI.
  Step 2 In the Main Menu, type a to choose the System Settings menu.
  Step 3 In the System Settings menu, type a to choose Network Settings.
  Step 4 In the Network Settings menu, type a to choose Setup Network Information.
  Step 5 In the Setup Network Information screen, press any key to continue.
  Step 6 Configure for either Single Node or REM HA setup.
  a. Single Node:
  1. Within the Select Action screen, choose Edit Devices.
  2. Select eth0 to configure. Press the Enter key.
  3. Uncheck the Use DHCP check box.
  4. Enter the desired IP addresses for the Static IP address, Netmask IP address, and Default gateway IP address fields. Select Ok.
  5. Choose Save and press the Enter key.
  6. Choose Edit DNS Configuration.
  7. Enter the Hostname, Primary DNS, Secondary DNS, Tertiary DNS, and Search fields. Select Ok.
  8. Select Save and Quit.
  10. To restart network service, select c) Services Control in the Main Menu. Then choose a) Networking in the Services Control menu. Finally, choose a) Restart networking in the Networking menu.
  If network is set up properly, you should see a message similar to "Updating REM DB with IP Address: 172.20.20.20".
  b. REM HA (Dual Node):
  1. Choose Edit Devices.
  4. Enter a public IP address in the Static IP address, Netmask IP address, and Default gateway IP address fields. Select Ok.
  5. Select eth1 to configure. Press the Enter key.
  6. Uncheck the Use DHCP check box.
  7. Enter a private IP address (e.g. 192.168.10.100) in the Static IP address field. In the Netmask IP address field, enter 255.255.255.0. Leave the Default gateway IP address field empty. Select Ok.
  8. Choose Save and press the Enter key.
  9. Choose Edit DNS Configuration and enter the Hostname. Leave the Primary DNS, Secondary DNS, Tertiary DNS, and Search fields empty. Select Ok.
  Note Each node should be given a unique hostname.
  10. Select Save and Quit.
  11. Go back to the Main Menu.
  12. To restart network service, select c) Services Control in the Main Menu. Then choose a) Networking in the Services Control menu. Finally, choose a) Restart networking in the Networking menu.
  13. If network is set up properly, you should see a message similar to "Updating REM DB with IP Address: 172.20.20.20".
  14. Repeat substeps 1-13 for the second node.
  Step 7 To verify if the network is configured properly:
  a. In the Main Menu, type a to choose the System Settings menu.
  b. In the System Settings menu, type d to choose System Information.
  Note If the IP address needs to be changed later, you may need to follow this section to reconfigure your network.
  First, make sure that you SSH to REM via the TAC account. Execute the commands below on the Shell prompt of REM server to check the REM DB Details:
  Step 1 On the Shell prompt of the REM server, execute /etc/init.d/postgresql-<version> status to check Postgres status.
  Step 2 Log into the node where REM DB is running.
  Step 3 At the DB prompt of the REM server, execute the following:
  [REM1-9]$ sudo su -l postgres
  -bash-3.2$ psql postgres
  Step 4 Check the list of tables available in REM_DB.
  Step 5 At the DB prompt, execute \d to list the REM database tables.
  Step 6 To take a database schema backup, enter pg_dump -Upostgres -hlocalhost --schema-only -f <location/of/backup.sql> REM_DB.
  Step 7 To take a database data backup, enter pg_dump -Upostgres -hlocalhost --data-only -f <location/of/backup.sql> REM_DB.
  Step 8 Restore the schema from the sql file by entering the psql -Upostgres -hlocalhost -d REM_DB -f <location/of/dumpfile.sql> command.
  Step 9 Log into the database.
  Step 10 Execute the \i <location/of/backup.sql> command to restore the data from the sql dump file.
  Step 11 Drop the database using the DB dropdb -Upostgres -hlocalhost REM_DB command.
  Step 12 Create the database using the createdb -Upostgres -hlocalhost 'REM_DB' command.
  Step 13 Check the list of data available in the table.
  Step 14 At the DB prompt, execute Select * from <table_name>.
  Step 15 To logout from REM database, enter \q at the REM_DB prompt.
  After adding the document from REAC, the document will be added in the database with creation date (in the doc_table it is called 'added_date'). If time zone is not configured properly, it is very difficult to maintain the database.
  When customer touches an expert type button on the REIC, a call is initiated. This call will come to the remote expert desk. After answering the call, a session has been established. This call session information can be found in the session table of the database.
  Note The session information contains the time when the session was generated. This information is generated from RESC. If the time zone is not configured properly, then this call start time will not be correct compared to the local time where the server is located. If that is the case, it is very difficult to inspect the time when the session was generated.
  All the videos added by REAC are added into the REM database. The video is added to the database with the time when it was added to the REAC. For maintaining the database time zone, the VM should be configured properly.
  When a call is initiated from the REIC, all the call related information such as call connect, call on hold, or call disconnect are visible in the database.
  The start time for each call status is also added to the database. An incorrectly configured time zone affects this time.
  Tomcat Catilana log
  REAC log
  REM Backup log
  REM Restore log
  Note Time zone affects the generation of log files. Logs are generated with current time and date in the log files.
  This section explains how to use the Troubleshooting tools in the TUI that is accessed by a SSH client.
  Use the Ping utility to verify that the REM servers are up. This is particularly useful for HA where there are four IP addresses that should be verified.
  Step 1 In the Main menu, type e to access the Troubleshooting menu.
  Step 2 In the Troubleshooting menu, type a to ping a host.
  Step 3 Enter the IP address of the REM server.
  The host will be pinged and the results are displayed.
  Step 4 When finished, press any key to return to the previous menu.
  There are a number of logs that are generated that can help you with troubleshooting issues.
  System logs: These are basically hardware-related logs including system console messages, authentication and authorization logs, and driver messages.
  Remote Expert logs: The majority of logs that you will use are located in this menu which includes logs for all the REM components (REAC, READ, RESC), IAS tool, and Report Generation tool as well as logs for Backup and Restore.
  Application Server logs: These are the Tomcat and Postgres logs.
  Step 1 In the Troubleshooting menu, type b to access logs.
  Step 2 Choose a log menu: System, Remote Expert, or Application Server.
  Note For security reasons, the TUI will disconnect after three minutes of no activity on the part of the administrator. If the administrator wants to view log activity for more than three minutes, the administrator needs to interact with the TUI in some way such as press the Enter key in order to keep a connection.
  Step 3 Press ESC :q! to stop watching a log.
  Step 4 When finished, press the R, <, or, key to return to the previous menu.
  The Download Logs tab in REAC contains lists of logs that can be used for troubleshooting issues:
  1. Admin Console: Contains the reac.log from this REAC
  2. Agent Desktop: Contains the read.log from READ
  3. Session Manager: Contains the resc.log from the RESC
  4. RE Events: Capture RE events during a specified time
  5. Tomcat: Contains the catalina log for the Tomcat service
  Refer to the "Downloading Logs" section of the Cisco Remote Expert Manager Administration Guide for instructions on how to download those logs.
  The backup tool automatically takes a backup of various REM assets such as IAS templates (for all REM component configurations), videos, images, documents and the database from the primary REM server at a scheduled time and stores the backup archive in a secondary REM server or remote FTP/SFTP location.
  The restore tool is used to restore the REM server to the last working state using the backup archive in case the REM server goes down.
  Any Linux-based system can be used for the backup and restore server.
  Note The SSH port (22) must be open.
  In this section are high level block diagrams for the backups of single node and dual node REM servers.
  The figure below shows the steps for a single node backup. It assumes that the administrator has logged into REM and accessed the IAS tool and Tomcat, PostgreSQL and SSH are running on the REM server. The backup is taken from the REM node and stored in the remote FTP/SFTP/SSH server.
  1. The IAS tool triggers schedule script located in the Backup utility.
  2. The schedule script adds job scheduling parameters related to the backup and starts the Cron process.
  3. The Cron process executes the backup script at the configured interval.
  4. The backup script backs up the data and sends the backup archive to a remote FTP/SFTP server or a remote SSH server (Linux machine).
  Figure 2-1 Single Node Backup
  The figure below shows the steps for a dual node backup. It assumes that the administrator has logged into REM and accessed the IAS tool and Tomcat, PostgreSQL and SSH are running on both REM servers. The backup files are created from both REM nodes and stored in the remote FTP/SFTP/SSH server.
  1. The IAS tool triggers the schedule script located in the Backup utility in the primary REM server.
  2. The schedule script adds job scheduling parameters related to the backup and starts the Cron process in the primary server.
  3. The IAS tool triggers the schedule script located in the Backup utility in the secondary server.
  4. The schedule script adds job scheduling parameters related to the backup and starts the Cron process in the secondary server.
  5. The Cron process executes the backup script at the configured interval in the primary server.
  6. The Cron process executes the backup script at the configured interval in the secondary server.
  7. The backup script backs up the data from the primary server and sends the backup archive to a remote FTP/SFTP server or a remote SSH server (Linux machine).
  8. The backup script backs up the data from the secondary server and sends the backup archive to a remote FTP/SFTP server or a remote SSH server (Linux machine).
  Figure 2-2 Dual Node Backup
  Following are the different scenarios incorporated in the backup process:
  1. PostgreSQL service status: The backup process checks whether the PostgreSQL service is running. If it is not running then the PostgreSQL service is started automatically. This check is mandatory to create the PostgreSQL database backup.
  2. Log file existence: Log files are used to track the backup process. When executing the backup process, it checks if there are log files in the log directory. If there are no log files, then it is assumed that the backup process is executing for the first time and it creates the log file in the specified directory.
  3. Artifacts existence: Artifacts are the different REM components such as template files for configurations, images, videos, etc. If any of the artifacts to be backed up are missing in the primary REM server, then the backup process logs error messages in the log file and continues.
  4. Remote copying: If there is an error in storing backup archive remotely, then the backup process logs errors with appropriate error messages in the log file.
  5. Backup archive rotation: The backup archive rotation is done in the remote storage server to keep the most recent backup archives. In case an error occurs in backup file rotation, (for details refer to the "Log File Rotation" section of this chapter) the backup process logs errors in the backup log file.
  If email notification is enabled for backup and restore, an email notification is sent to the system administrator regarding the last backup status. In case of an error in the last backup, an email is sent with appropriate error messages so that system administrator can debug the issue to find the root cause.
  For details on different error messages used in the tool refer to the "Error Messages" section of this chapter.
  Note If there is a problem in the database backup, the backup process terminates. After creation of the entire backup, the backup process creates an archive with the.tgz extension. If there is a failure in creating a backup archive, the backup process is terminated instantly. Restoration activity cannot be performed in this case.
  Following are the different scenarios incorporated in the restore process:
  1. PostgreSQL service status: The restore process checks whether the PostgreSQL service is running in the REM server. If it is not running, then the PostgreSQL service is started automatically. This check is mandatory to restore the database.
  2. Tomcat service status: During the restoration of artifacts, the restore process checks whether the Tomcat service is running. If it is running, then it stops the service. This check is required because the Tomcat service may use some configuration files of REM components at the same time that the restore process is overwriting them.
  3. Directory structure: If there is something wrong in the directory structure of the artifacts path during restoration of artifacts, then the restore process logs errors in a log file located in the log directory.
  4. Database restoration: If there is an error in the database restoration, then the restore process logs that error in the log file.
  Note The restore process continues to restore the REM server irrespective of failure of one of the artifacts restoration. Restoration terminates if there is failure in restoring the database since the REM does not work with an invalid database. After restoration, the Tomcat service is started automatically by the restore script.
  In this section are sequence diagrams for backup and restore processes.
  Figure 2-3 Backup Sequence Diagram
  The sequence diagram includes the update of the properties file by the REM configurator and automatic backup script execution. Before doing the backup, all the properties files used by the backup script must be updated by the IAS (Installation and Automation Script) tool. Backup scheduling is done using Crontab, an utility to schedule jobs in UNIX-like operating systems. In the backup process, the backup archive is generated in the primary REM server and the archive is sent to the secondary REM server or a FTP/SFTP server (storage server). After storing the backup archive, another script is invoked in the secondary server by the backup script for backup archive rotation. This is required to keep the last modified backup archives in the storage server.
  After each backup process, an email notification is sent to the system administrator containing information about the last backup. If the last backup is successful, a success message is sent to the system administrator. If the last backup is not successful, appropriate error messages are sent to the system administrator.
  The flows in the sequence diagram are:
  1. The IAS tool reads rem.properties, templates and environment (env) variables and updates the configuration files for the backup process by executing 'main.sh'.
  2. The script for scheduling backup (discussed in the "Scheduling" section of this chapter) is invoked in the primary REM server by the system administrator. It adds the job parameters for automatic backup in Crontab if scheduling is enabled by the REM configurator.
  3. Executing the backup script at the configured time interval using Crontab if the Crond process, a daemon process for scheduling used by Crontab, is running in the REM server.
  4. Before backing up the data in the primary REM server, the backup script checks and validates the pre-conditions. The backup starts if the backup feature is enabled by the REM configurator and the PostgreSQL service is running.
  5. After creating the backup archive, the script sends the backup archive to a REM server or a remote FTP/SFTP server (storage server).
  6. The remote server sends an acknowledgement to the primary REM server that the archive was sent successfully.
  7. A request for file rotation is sent to the storage server.
  8. Backup archives files are rotated in the storage server.
  9. The storage server sends an acknowledgement to the primary REM server that file rotation was successful.
  10. Log files in the primary server are rotated after each backup process.
  11. Backup script sends an email notification after each backup process with appropriate messages to the system administrator.
  Figure 2-4 Restore Sequence Diagram
  If the REM server goes down, the restoration process uses the backup archive located in the storage server to restore the REM server.
  In the sequence diagram, the different flows are:
  1. The IAS tool reads rem.properties, templates and environment (env) variable and updates the configurations file for the restore process.
  2. The restore script takes the backup archive name with its full path (e.g., $REM_HOME/tools/backup/data/rem_Aug-29-2012_02-13-32_10.76.8.178.tgz) as an argument.
  3. The restore script checks the pre-conditions. The restoration starts if the restore feature is enabled by the REM configurator, PostgreSQL is running, and the Tomcat service has stopped.
  4. All the REM artifacts are restored to the proper location in the REM server.
  5. REM server acknowledges the script that artifacts restoration is done. If an error in artifacts restoration occurs, appropriate error messages are displayed on the console.
  6. The script restores the PostgreSQL database in the REM node.
  7. The REM node acknowledges that the database was restored successfully.
  8. The rem.propeties (master template) is updated.
  9. The command main.sh is executed in the IAS tool is executed.
  10. The IAS tool starts Tomcat in the REM node.
  11. Tomcat service is started in the REM server.
  The following is terminology used for artifacts:
  REM_HOME = Base installation folder path containing necessary configuration setting for REM application (e.g., REM_HOME = /var/rem)
  tomcat_home = Absolute path of apache tomcat (e.g., tomcat_home = /opt/apache-tomcat-6.0.29)
  The backup archive contains the following list of artifacts:
  1. REM database
  2. REIC assets that contain Wait videos for REIC located under <tomcat_home>/webapps/reic/assets
  3. Locale properties for REIC located under <tomcat_home>/webapps/reic/resources
  4. Images for the REIC application located under <tomcat_home>/webapps/images
  5. Properties related to Connected Justice (CJ) located under <REM_HOME>/cj/conf/segments
  6. Template configuration files for REIC located under <REM_HOME>/tools/ias/templates/reic
  7. Template configuration files for RESC located under <REM_HOME>/tools/ias/templates/resc
  8. Template configuration files for REAC located under <REM_HOME>/tools/ias/templates/reac
  9. Template configuration files for READ located under <REM_HOME>/tools/ias/templates/read
  10. Template configuration files for CJ located under <REM_HOME>/tools/ias/templates/cj
  11. Master template for REM (<REM_HOME>/tools/ias/templates/rem.properties)
  12. All the documents which are shared with customers located under <tomcat_home>/webapps/docstore
  The following scripts for the backup process are located under $REM_HOME/tools/backup/scripts/:
  backup (main script for backup)
  schedule (for scheduling backup)
  logrotation (for log file rotation)
  The script for the restore process is located under $REM_HOME/tools/restore/scripts/:
  restore (main script for restore)
  The backup process uses the following properties file located under $REM_HOME/tools/backup/conf/:
  schedule.properties
  logrotate.properties
  The restore process uses the following property file located under $REM_HOME/tools/restore/conf/:
  System administrators can use the log files to track previous processes - both backup or restore.
  The backup.log file is generated under $REM_HOME/tools/backup/logs/ during the backup process. The restore.log file is generated under $REM_HOME/tools/restore/logs/ during the restore process.
  All the third party libraries for the backup process are located under $REM_HOME/tools/backup/lib. There are no third-party libraries used by the restore process.
  The binaries (rem-backup.jar) for the backup process are located under $REM_HOME/tools/backup/dist. There are no binaries for the restore process.
  Restoration can be done in either a single node or a dual node setup.
  The following are the minimum set of requirements for restoration to be done successfully for a single node setup:
  The node should have all the REM components (REAC, READ, RESC and REIC) and IEP Licensing service deployed
  The following are the minimum set of requirements for restoration to be done successfully for a dual node setup:
  Primary and secondary REM servers should be synchronized by time zone
  Both REM servers should have the same REM components (REAC, READ, RESC and REIC) and IEP Licensing service deployed
  Both servers should have the same database schema
  All the artifacts path should be same in both machines
  The backup and restore tool can be enabled or disabled by the REM configurator (IAS tool) during installation of the REM ISO. The flag which is used for enabling or disabling is located under the following properties and updated by the IAS tool during REM ISO installation:
  In the rem.properties file, set the flag to True to enable the feature or False to disable the feature. The flag is the FEATURE_ENABLE property:
  #To enable backup and restore for this release make it 'True' and make it 'False' to disable this feature
  $*[FEATURE_ENABLE]*=
  This section explains the procedure to back up the various artifacts (database, template configuration files, assets and images) from the REM servers and then restore the REM servers.
  All properties related to the backup process must be configured in the rem.properties file during REM ISO installation. For details on how to update rem.properties and create properties file for backup refer to IAS documentation.
  In the steps below, you will update the properties used for backup. These must be updated before executing the IAS tool.
  Step 1 Configure the rem.properties file. All properties related to backup process are typically configured in the rem.properties file during REM ISO installation but they can be modified later.
  a. Open the rem.properties file.
  b. To enable backup and restore, change the FEATURE_ENABLE property to true.
  c. In the ARCHIVE_IDENTIFIER property, enter the IP address or hostname of the server from where the file will be backed up. This value will be appended to the backup file name so its origins can be easily identified by the administrator.
  Note Do not modify the BACKUPDIR property. It is used for Cisco internal testing purposes only.
  d. SSH is the only mode currently supported so enter ssh for the MODE property.
  e. In the SERVER_ADDRESS property, enter the IP address of the remote archiving server where the backup file will be sent.
  f. For the SERVER_BACKUP_PATH property, enter the path as to where the backup files should be stored on the remote archiving server.
  g. Configure the frequency of backups by entering the time and day that the backup should occur. This will create a Cron job inside the system.
  You can modify the backup frequency by changing the values of the property.
  The default is everyday at 6:30 p.m. which is configured as follows:
  – MIN=30
  – HOUR=18
  – DAY=*
  – MONTH=*
  – WEEKDAY=*
  Note For job scheduling, an asterisk (*) can be used to indicate that every instance (i.e. every hour, every weekday, etc.) of the particular time period will be used.
  If you want to backup on the 15th of every month instead, for example, enter 15 for the DAY property. If you want to backup on Fridays only, enter 5 for the WEEKDAY property.
  h. If you want e-mail alerts for status of backups, change the MAIL_ENABLED property to true and then populate the SMTP server name, sender's mail ID, and recipients mail ID.
  i. After you make all the changes to the rem.properties file, you need to save it by pressing SHIFT : and then entering wq.
  j. Follow the instructions in the "Execute IAS" section of the Cisco Remote Expert Manager Installation Guide to run the configuration tool and encrypt the passwords.
  Step 2 Configure the backup.properties file:
  a. Navigate to the backup.properties file by using the following commands:
  #cd $REM_HOME/tools/ias/templates/backup
  #vi backup.properties.template
  b. Configure the following properties:
  Note Replace the brackets with the actual values. For example, replace <path to the temp location where backup folder was created> with /tmp/rem/backup.
  tomcat_home= <path to the installation of Tomcat server>
  workspace= <path to the temp location where backup folder was created>
  backupdir= <backup archive location in storage server>
  succsub=Backup was successful!
  failsub= Backup has failed!
  #Absolute path to the installation of Tomcat server
  tomcat_home=
  # Absolute path to the temp location where backup folder are created
  workspace=
  #(E.g., /tmp/rem/backup)
  #Backup archive location in storage server
  #(E.g., /var/rem/tools/backup/data)
  #Configure the subject of the mail message in case backup is successful (must be quoted if it contains spaces)
  succsub=
  #Configure the subject of the mail message in case backup is unsuccessful (must be quoted if it contains spaces)
  failsub=
  Step 3 Update the logrotate.properties file for file rotation:
  a. Open the logrotate.properties file located under $REM_HOME/tools/backup/conf/.
  b. Configure the properties in the file with the following values:
  feature_enable=True
  d. Execute the logrotation script located under $REM_HOME/tools/backup/scripts/ for adding parameters to /etc/logrotate.conf (in UNIX) for log file rotation by entering the command:./logrotation
  Note This step is necessary because the backup script uses log file rotation after backing up all the REM components each time. More details on log file rotation can be found in the "Log File Rotation" section of this chapter.
  Now follow the steps below to create the backup.
  Step 4 Log into the Primary REM Server through SSH.
  Windows users: Download putty from http://www.putty.org/.
  a. Execute the following command to login to the REM server: #ssh <user>@<host> (e.g. ssh root@172.21.52.176)
  b. Enter the password to log in.
  Step 5 Execute the script to create the backup.
  #cd $REM_HOME/tools/backup/scripts/
  #./backup
  Note The script can be executed from any directory by providing the full path (e.g. $REM_HOME/tools/backup/scripts/backup). On executing the backup script, the data and logs directories are created under $REM_HOME/tools/backup. The backup process creates an archive with the.tgz extension and the name appended with date, time, and REM server information from where the backup is created. It is stored in a REM server under the $REM_HOME/tools/backup/data directory (e.g. $REM_HOME/tools/backup/data/rem_Aug-29-2012_02-13-32_10.76.8.178.tgz) or a remote FTP/SFTP server. A log file for backup activity is generated under the $REM_HOME/tools/backup/logs/ directory after each backup process. Log file rotation is performed in the same directory. For details on log file rotation please refer to the "Log File Rotation" section of this chapter. For details on scheduling the backup and automation of backup refer to the "Scheduling" section of this chapter.
  This is the process of restoring the REM server in a single node or a dual node setup using the backup archive.
  Pre-conditions: The required environment (e.g., OS, PostgreSQL and Tomcat) and application binaries are installed on the secondary server. To perform the above, please refer to the Cisco Remote Expert Manager Installation Guide.
  First, you will need to update the rem.properties file during REM ISO installation and before executing the IAS tool.
  Follow these steps to restore the REM server:
  Step 1 Configure the FEATURE_ENABLE property in the rem.properties file to enable restoration:
  #To enable backup and restore for this release make it 'True' and make it 'False' to disable this feature
  Step 2 Configure the restore.properties file:
  a. Navigate to the restore.properties file by using the following commands:
  #cd $REM_HOME/tools/ias/templates/restore
  #vi restore.properties.template
  workspace= <path to the temp location where backup folder is extracted from archive>
  #Absolute path to the temp location where backup folder are extracted from archive
  Step 3 Log into the REM server where restoration has to be done using SSH.
  Step 4 Execute the script to restore the backup archive.
  Pass the argument to run the restore script backup archive location with the filename by entering the following commands:
  #cd /var/rem/tools/restore/scripts/
  #./restore </path/to/backup_archive_name>
  Note Replace </path/to/backup_archive_name> with the actual path. For example, #./restore $REM_HOME/tools/backup/data/rem_Aug-29-2012_02-13-32_10.76.8.178.tgz. The script can be executed from any directory by providing the full path (e.g., $REM_HOME/tools/restore/scripts/restore </path/to/backup_archive_name>).
  Restore is required on demand if the REM server goes down due to system failure.
  In dual node setup, restoration must be done in both REM nodes to prevent data inconsistency.
  Both REM nodes must be in the same time zone.
  After restoration, file integrity should be checked in both REM nodes using the md5 checksum. If the checksum is mismatched, then restoration has to be executed again.
  The following table contains all the error messages generated in backup and restore processes. These messages are logged in backup and restore log files respectively during the processes. If error messages are logged during a backup process, the messages are sent to the system administrator's email account. For restoration errors, all the error messages appear in the console.
  Table 2-8 Backup Error Messages
  Condition when the alert will be triggered
  Alert type (Success/ Warning/ Error)
  Configuration needs to be checked to resolve the issue
  Artifacts are not backed up successfully!
  Cannot copy: No such file or directory
  Check artifacts location in primary server
  Database backup is failed!
  Check DB properties in backup.properties
  Sending backup archive to ssh server<server_address> is failed!
  Authentication to <server address> is failed!
  Check server_username and server_password in backup.properties
  Connection to <server address> is failed!
  Check server_address in backup.properties
  Login credentials are incorrect for FTP server!
  Check server properties in backup.properties
  File upload is failed to < server_address >!
  FTP server refused connection
  Backup directory creation is failed in FTP server!
  Check backup directory property in backup.properties file
  File rotation is failed!
  Check server related properties and no_backup_files in backup.properties file
  Number of backup files cannot be negative
  Check no_backup_files in backup.properties
  Number of backup files cannot be zero
  Number of backup files should be a number
  Configured number of backup files is not correct
  Backup archive creation is failed!
  Backup is done with errors
  Check log file to find the root cause
  Table 2-9 Restore Process Error Messages
  Alert type (Success/
  Artifacts are not restored properly!
  Check artifacts path in REM server
  Database restoration is failed!
  Check DB in restore.properties
  Backups can be scheduled so that they are automatically performed at set intervals. The Crontab utility is used to schedule backups.
  During REM ISO installation, the IAS tool triggers schedule script located under $REM_HOME/tools/backup/scripts/ in REM node for single node as well as dual node setups. If backup is enabled, it adds the job to Crontab and starts the Cron job in the REM servers automatically.
  Step 1 Update the following parameters in rem.properties located under $REM_HOME/tools/ias/templates/ in the IAS tool. (For details refer to IAS documentation.)
  Note The following properties need to be updated in both REM servers if there is a dual node setup.
  #Configuring job scheduling parameters
  #To configure the schedule user has to configure the following parameters. User can use * as a value of any of these parameters.
  #An asterisk (*) is used to indicate that every instance (i.e. every hour, every weekday, etc.) of the particular time period will be used.
  #The values can not be blank. User can enter multiple values separated by commas.
  #E.g, 30 18 * * * (each day at 6:30 PM)
  #* * * * * (every min)
  #0 0 1,5,10 * * (midnight on 1st,5th & 10th of month)
  #*/2 * * * * (Every 2 min)
  #Configure min (between 0-59)
  $*[MIN]*=
  #Configure hour (between 0-23, 0=midnight)
  $*[HOUR]*=
  #Configure day (between 1-31)
  $*[DAY]*=
  #Configure month (between 1-12)
  $*[MONTH]*=
  #Configure day of week (0-6, 0=Sunday)
  $*[WEEKDAY]*=
  Step 2 Execute main.sh located under $REM_HOME/tools/ias/scripts.
  Note The log of the scheduling activity can be found in $REM_HOME/tools/backup/logs/schedule.log. It is recommended that backup scheduling should be done when the network load is low. It is not suggested to schedule backup during working hours.
  Backup archives are rotated after each backup in the backup storage server (SSH/FTP/SFTP server) to keep the most recently created backup archives. If the last backup archive is corrupted, then the system administrator has the option to choose another archive. File names include dates so that the administrator can choose a backup archive that was created on a particular day to restore the REM server to that date.
  The NO_BACKUP_FILES property must be updated in the rem.properties file during REM ISO installation to indicate the number of backup files that should be stored in the backup server.
  #No of backup files to be stored in backup server
  $*[NO_BACKUP_FILES]*=
  File rotation is done whenever a new backup archive is stored under $REM_HOME/tools/backup/data/ in case of the REM server (SSH) or FTP server.
  All the logs related to the backup archive rotation can be found in $REM_HOME /tools/backup/logs/backup.log.
  Logrotate is the default application used to rotate all application or process log files, which is provided by the UNIX operating system. For backup, it is backup.log file. Logrotate allows automatic truncation, rotation, compression, removal, and mailing of log files.
  If a log file grows too large, it must be truncated and rotated. All this is done more efficiently by the logrotate application.
  In the backup and restore process, the logrotate application is used to rotate the backup.log if it exceeds the user-specified size.
  You must configure the rotate and size properties in the logrotate.properties file located under $REM_HOME/tools/backup/conf/:
  #Number of log files to be rotated
  #Truncate the log file if it has size greater than the size specified here
  #If size is followed by M, the size is assumed to be in megabytes or if k is used, the size is in kilobytes
  Execute the logrotation script located under $REM_HOME/tools/backup/scripts using following command to configure logrotate.conf using properties above for log file rotation:
  #cd '$REM_HOME/tools/backup/scripts'
  #./logrotation
  The logrotate application is automatically executed by the backup script after creating the backup archive.
  For example, consider the case where rotate=2 and size=1M. After rotating and truncating the backup.log file, the $REM_HOME/tools/backup/logs folder may contain the following files:
  backup.log.1
  backup.log.2, etc.
  The backup.log file contains the most recent back up logs. The backup.log.1 file contains the logs which are older than the backup.log file. The backup.log.2 file contains the logs which are older than those in backup.log.1.