Have You Met Your Objectives?
Evaluation is the most important aspect of an exercise. It is the process that determines if capabilities are attained and objectives are met. Effective evaluation begins with the exercise design process and includes four key components:
- Planning for exercise evaluation;
- Observing the exercise and collecting exercise data during exercise conduct;
- Analyzing the data collected to identify strengths and areas for improvement; and
- Reporting exercise outcomes in an after action report (AAR).
Evaluators use exercise evaluation guides (EEGs) to observe and collect data related to the exercise objectives and the corresponding core capabilities. E
xercise designers and evaluators customize these forms in order to focus on the specific critical tasks that must be displayed by participants to confirm that they have achieved the capabilities targeted in the exercise.
EEGs include four elements to facilitate observation and evaluation: the objective; the related overall core capability; the organization’s targeted capabilities and the critical tasks.
The objective is the specific outcome the organization wants to achieve during the exercise. It is usually based upon one or more of the 31 core capabilities listed in the National Preparedness Goal. Achieving these capabilities is based upon the performance of associated critical tasks. When linked to operating conditions and performance standards, critical tasks provide the primary source of learning objectives for training and exercises and provide input to planning and performance evaluation.
Critical tasks allow evaluators to assess the level of participant capabilities in two of the primary learning domains: cognitive and psychomotor. Participants demonstrate what they know (cognitive) by what they can do (psychomotor). For operations-based exercises, designers must ensure that conditions in which critical tasks are performed are as realistic as possible.
Targeted capabilities and critical tasks may be drawn from a number of sources, including the National Preparedness Goal and the five National Planning Frameworks, or from a threat and hazard identification and risk assessment product (THIRA). The best source, however, may be from an organization’s own plans and evaluations from previous exercises.
Fix Problems Revealed in After Action Review
At the conclusion of the exercise the evaluations are analyzed. This is accomplished through an after action review (AAR) process, the main focus of which is the analysis of core capabilities. The analysis should consider the following questions:
- Were the capability targets met? If not, what factors contributed to this result?
- Did participant discussion or activities suggest the critical tasks met capability targets? If not, what was the impact or consequences?
- Were participants familiar with current plans, policies, and procedures, and did these documents support critical tasks and capability targets?
The analysis should produce corrective actions to be implemented by the participating organization in order to enhance its overall capabilities. These corrective actions can assist an organization with making changes to plans, procedures, organizational structures, management processes, equipment, resources and the training related to these items in order to improve performance. It can also help to identify lessons learned that can be applied to similar situations in the future.
Unfortunately this process is not always implemented. Even after Katrina, investigators found that “Too often, after action reports for exercises and real-world incidents highlight the same problems that do not get fixed.”
Paper Plans Can Only Go So Far
Until a comprehensive exercise program validates its assumptions and related concept of operations, even the most well written emergency operations plan remains only a theory. To ensure an adequate return on its investment in planning, training and its personnel, no organization should leave exercises out of its preparedness cycle.
Gene Komondor is a FEMA-certified master exercise practitioner with more than 20 years experience in healthcare and emergency services. For more information, visit www.OneStarTraining.com.
