December 21, 1999
TO: All Holders of Operations Manual
SUBJECT: Transmittal Memorandum for Operations Manual
Issuance 99-07
1. Material Transmitted:
WSOM Chapter A-63, Service Evaluation.
2. Summary:
The chapter has been completely rewritten to reflect changes resulting from the modernization and associated restructuring. References to Weather Service Evaluations Officers, evaluations at flight service stations, and other positions/functions that no longer exist have been removed. The new chapter emphasizes interaction with partners and customers and suggests strategies for operating a fully integrated evaluation program. It also establishes the requirement for a service evaluation plan. This plan is one requirement for declaring an office a Weather Forecast Office.
3. Effect on Other Instructions:
Supersedes WSOM Chapter A-63, Program Evaluation, Transmittal Memorandum 90-1, dated February 23, 1990.
John J. Kelly, Jr.
WSOM-A-63-99-0
| Issue Date | Org. Code | NATIONAL WEATHER SERVICE
Operations Manual |
Part | Chap |
| 12-21-99 | W/OM11 | A | 63 |
SERVICE EVALUATION
Table of Contents Page
1. Purpose
2. Background
3. Definitions
4. Organizational Responsibility
4.1 Weather Service Headquarters (WSH)
4.2 Regional Headquarters (RH)
4.3 Weather Forecast Offices (WFO)
4.4 National Centers for Environmental Prediction (NCEP)
6.1 Evaluation at Center Weather Service Units
(CWSUs)
6.2 Networking
7.1 NCEP Evaluation for Internal (NWS) Partners
7.2 NCEP Evaluation for External Partners and
Customers
8.1 Report Content
Appendices:
A Sample Process for Obtaining Field Office Input
into a National Requirement for Build Change,
Policy Development, Etc. A-1
B Teams B-1
C Recommended and Preferred Practices C-1
WSOM Issuance
1. Purpose. This chapter establishes policy for managing the National Weather Service (NWS) service evaluation program. The goal of the program is to measure service quality and, with the NWS' partners and customers, identify areas to improve the effectiveness of products and services.
2. Background. The success of the NWS mission is related to the quality and timeliness of products and services provided by each office. A public that more than ever demands quality customer service requires the NWS to be responsive to the needs of its partners and customers in those areas where the NWS has authority. The various warning programs and the product and service suite geared to the public and non-specific customer groups are included. An ongoing evaluation program is necessary to maintain a high level of service.
3. Definitions. For the purposes of this chapter, the following definitions apply.
Service Evaluation: the process of determining the worth of NWS products and services. The determination is made by qualitative and quantitative feedback from partners and customers.
• Qualitative feedback is the value of products and services to partners and customers and takes the form of subjective data (comments, compliments, complaints, etc.).
• Quantitative feedback is the utility of products and services to partners and customers and takes the form of objective data (timeliness, clarity, ease of use, etc.).
Partner: any individual or group (e.g., Federal Aviation Administration (FAA), media, emergency managers, commercial weather industry, etc.), inside or outside the NWS, that provides product(s) and/or service(s) to the public.
Customer: an individual member of the public or a group composed of, or representing, private citizens. This includes the private sector that uses weather information.
4. Organizational Responsibility. Management of the evaluation program shall be in accordance with this chapter with the following exceptions.
a. Policy for managing the Pilot Weather Briefing Program, and other aviation functions formerly assigned to the Weather Service Evaluations Officer, are in Weather Service Operations Manual (WSOM) Chapters D-26 and D-82.
b. Policy for managing evaluation activities in River Forecast Centers (RFCs), Regional Hydrologic Services Divisions, and the Office of Hydrology (OH) are in WSOM Chapters E-04, E-05, E-06, E-11, and E-21.
4.1 Weather Service Headquarters (WSH). WSH establishes policy for service evaluation. WSH shall manage a program that results in continuous improvement of products and services. This occurs through solicitation of opinions, and feedback for requirements, from partners and customers.
As part of this process, WSH shall develop an organized method of soliciting field input that ensures recommendations for changes and improvements are given serious consideration in the areas of:
• requirements for new technology,
• evaluation of the utility of new systems,
• prioritization of tasks for software modification,
• evaluation of draft new or revised policy governing the operations of the NWS.
An example of a process for obtaining field input is shown in Appendix A.
WSH shall support innovative efforts by the regions to enhance service to partners and customers and, where applicable, shall encourage use of teams in the evaluation effort (see Appendix B).
4.2 Regional Headquarters (RH). Regions are responsible for implementing evaluation policy and support innovative efforts by local offices to satisfy customer needs. This includes modifying policy in coordination with WSH through Regional Operations Manual Letters or other directives, suggesting alternate solutions, and facilitating dialogue between offices and partners/customers to reach acceptable resolutions. However, modifications to standard operating procedures shall be governed by instructions contained in WSOM Chapters A-40, Service Change Process, and A-06, Policy and Guidelines Governing NWS and Private Sector Roles. Examples of local changes that cannot be made except through procedures detailed in the above-mentioned chapters include product coding and format, and changes to products that would impact national partners.
4.3 Weather Forecast Offices (WFO). Meteorologists in Charge (MICs) are responsible for implementing and maintaining a service evaluation program at their offices. This program will assist in identifying partner and customer needs and will provide guidelines to satisfy these needs.
MICs shall coordinate with their National Weather Service Employees Organization steward following guidelines in the Quality Through Partnership, and any other applicable agreements.
4.4 National Centers for Environmental Prediction (NCEP). NCEP is responsible for running an evaluation program that ensures responsiveness to its partners, both inside and outside the NWS, and its customers.
5. Evaluation at WSH/RH. WSH and RH shall hold periodic workshops and/or use other methods to interact and cooperate with NWS partners. Feedback from partners shall contribute to policy changes and resulting service improvements. Where applicable and beneficial, WSH and RH should hold joint activities when working with WFOs, NCEP, partners and customers.
6. Evaluation at WFOs. Due to the number of partners WFOs support, teams are encouraged to obtain feedback. Each office shall document evaluation outreach activities and summarize changes made to products and services resulting from partner/
customer feedback. Based on outreach activities and interaction with customers, each office shall be able to summarize the overall level of customer satisfaction in the various program areas including: (1) major areas of customer concerns, and
(2) programs/efforts that have been particularly well received. Recommended and preferred practices can be found in Appendix C.
6.1 Evaluation at Center Weather Service Units (CWSUs). CWSU evaluations shall focus on internal activities and service to the FAA and aviation community. MICs of the CWSU and its associated WFO shall collaborate on CWSU evaluation and determine:
• frequency and scope of evaluation activities;
• reporting requirements; and
• the level of support (e.g., staff, training, coordination, etc.) provided by the WFO and RH.
6.2 Networking. WFOs are encouraged to share evaluation successes and failures with other offices. Regions should lead in the development and maintenance of this information exchange, monitor information flow, and publicize the most effective strategies for running a successful program. WFOs with national center-type responsibilities in aviation, marine, hurricane, and other programs should encourage the exchange of information with NCEP.
7. Evaluation at NCEP. The breadth and scope of NCEP's evaluation program, as well as the schedule of evaluation activities, is left to the discretion of NCEP management. Evaluation activities should be ongoing so partner/customer feedback is continuous and there is prompt identification and resolution of problems. Where applicable and beneficial, NCEP should hold joint evaluation activities with WSH, RH, and/or WFOs that have national center-type responsibilities.
7.1 NCEP Evaluation for Internal (NWS) Partners. Teams are recommended to obtain feedback from the various constituencies in the NWS. NCEP shall determine how these constituencies should be grouped, for example:
• meteorology, hydrology, computer science, etc.;
• aviation, marine, long-range prediction, etc.;
• WFOs, RFCs, CWSUs, etc.
Activities to be evaluated shall include but not be limited to:
• quality and useability of NCEP services, including models, model data, and other products;
• ease of access to NCEP employees;
• collaborative research efforts; and
• internal processes leading to more efficient operations.
NCEP shall also determine the methods whereby evaluations shall be accomplished (workshops, visiting scientist programs, etc.).
7.2 NCEP Evaluation for External Partners and Customers. Workshops or other methods of obtaining feedback should be employed on a periodic basis to determine:
• the satisfaction level of external partners and customers with NCEP products and services;
• effectiveness of NCEP-partner relationship in serving customers;
• ease of access to NCEP employees;
• quality and utility of NCEP services, including models, model data, and other products;
• improvements in products and services;
• partner/customer impact on future models, products and services; and
• other information as determined by NCEP and/or its partners and customers.
NCEP shall also determine the methods whereby evaluations shall be accomplished.
8. Reporting Requirements. NCEP and RH shall provide a concise written report to the Office of Meteorology (OM), Customer Service Core, at the same time reports are due to WSH in support of the NWS Annual Operating Plan.
Report format shall follow the outline in section 8.1. When the Regions determine that WSH action is required to resolve problems not addressed through other channels, they shall be identified in the report. OM shall assign these actions to appropriate NWS elements and shall track problems to resolution. Periodic reports on the status of unresolved issues shall be provided to WSH offices, RH, and NCEP.
Following submission of service evaluation reports, OM shall prepare a summary of practices reported by the regions and NCEP for distribution.
8.1 Report Content. Annual reports shall be concise and summarize accomplishments and areas of concern during the past year. Reports shall include regional/NCEP trends and issues occurring consistently across large parts, or all, of the region or NCEP. Problems uncovered by the evaluation process at individual offices or Centers shall be addressed by the respective region or NCEP unless the problem is national in scope or otherwise requires WSH collaboration.
The report shall contain:
a. Innovative evaluation processes/feedback methods used (other than normal interaction with customers) and the benefits derived from such activities;
b. Summary of the effectiveness of products, services, programs and initiatives. Include a description of any product and/or service which is particularly well received by partner/
customers;
c. Major concerns or problem areas associated with products and services;
d. Trends, as appropriate; and
e. Success stories.
APPENDIX A
SAMPLE PROCESS FOR OBTAINING FIELD OFFICE
INPUT INTO A NATIONAL REQUIREMENT FOR BUILD CHANGE,
POLICY DEVELOPMENT, ETC.
Step 1 - Team formed at WFO.
Step 2 - WFO team solicits input from employees.
Step 3 - WFO team oversees prioritization of input by staff.
Step 4 - RH form regional team(s) (representatives from WFOs)
and facilitates development of regional priority
list(s) from individual field sites.
Step 5 - WSH and RH form national team (representatives from
regions) and facilitate development of national
priority list(s) from regions.
APPENDIX B
TEAMS
While this chapter does not mandate the use of teams, the guidelines contained herein are written with the philosophy that for the NWS to accomplish its mission, team operations in the area of service evaluation are, with few exceptions, indispensable. When used in proper situations, teams:
a. empower employees, raise morale through staff ownership of the team process, and result in employee buy-in;
b. enable thorough analysis of problems and provide innovative solutions supported by staff;
c. create an atmosphere of constant improvement that allows an office to function with greater efficiency.
Managers are responsible for forming teams and should provide a charter for each team. The charter includes:
a. a vision (the ideal state for the future);
b. a mission (clear statement of the issue the team should address and goals the team will accomplish);
c. team membership, including the leader;
d. scope of authority (decision-making capacity, budget, other resources, limitations or constraints);
e. termination date (project completion);
f. success criteria (how the team will know it has accomplished its mission).
A sample team charter is shown below. Managers should charter teams for a specific time period. Before the termination date, the necessity for the team continuing should be determined and, if necessary, a new charter issued. In normal circumstances, the need for rechartering is for teams addressing ongoing issues on a periodic basis, such as customer feedback for a specific program.
Sample Team Charter
Vision. WFO XYZ and the media in the CWA will have a close working relationship that provides superior service to the public.
Mission. WFO XYZ Outreach Team will develop a process to provide prompt:
• updates to the media on changes to WFO products services and procedures, and,
• response to media questions and requests.
Team Membership.
Bob Black
Barbara Brown - Team Leader
George Gray
Gordon Green
Wendy White
Scope of Authority. The team can require all office staff except electronic technicians to perform research for the project, not to exceed 4 hours per person.
• The plan must be implemented with no additional office staff.
• The plan must be implemented with no changes in regional/national policy.
• The team may spend up to $1,000 to plan and implement; the cost of operation cannot exceed $1,500 per year.
Termination Date. Implementation by September 1, 1999. Adjust by October 1, 1999.
Success Criteria. The WFO XYZ Outreach Team will have been successful when 70 percent of county warning area media outlets state their relationship with the WFO has improved.
APPENDIX C
RECOMMENDED AND PREFERRED PRACTICES
1. WFO Evaluation (See Section 6).
a. The Warning Coordination Meteorologist should manage and coordinate all evaluation activities, make recommendations to the MIC on enhancing the office's evaluation program, and summarize recommended changes to products and services for office management.
b. Obtain evaluation data through feedback from partners and customers to determine the level of satisfaction with the office's products and services.
c. When weighing adherence to NWS policy against modifying products and services to conform to custom, practice, and need of partners and customers in their respective county warning areas, WFOs should seek the help of their respective RH. Examples of possible changes to products and services include using characteristic terminology or wording common in the local area, providing additional avenues of personal contact, utilizing non-operational dissemination/communications systems (e.g., Internet), etc. Any changes shall be coordinated with the RH and shall adhere to directives in WSOM Chapter A-40 (section 4.2) and the public/private sector roles detailed in WSOM Chapter A-06.
d. Teams should be used (Appendix B) to develop an ongoing relationship with partners and customers so that evaluation feedback is continuous and there is prompt identification and resolution of deficiencies. Methods of feedback can be face-to-face meetings, workshops, seminars, telephone/conference calls, announcements over the National Oceanic and Atmospheric Administration (NOAA) Weather Radio, mailings, etc.
2. WFO Internal Evaluation. Internal evaluation covers those areas of WFO operations "invisible" to NWS partners and customers and includes all activities that support or lead to provision of the office's products and services. Normally, internal changes should be driven by partner/customer requests or requirements. Changes that result from management directive, union negotiation, or employee/team suggestions, and have a major impact on products or services, should be discussed with partners and customers to ensure no decrease in satisfaction. Internal activities and processes chosen for evaluation should be those most likely to result in improvements that are readily apparent to partners/customers.
3. WFO External Evaluation. External evaluation covers those areas of WFO operations that are "visible" to partners and customers; in most cases, this means the products and services the office supplies. The following program areas, if applicable, shall be evaluated:
• public weather,
• aviation,
• marine,
• hydrology,
• fire weather.
Other activities for evaluation include but are not limited to:
• severe weather products and services;
• office interaction and partnership with the media, emergency managers, other government agencies;
• outreach activities (visiting schools, giving talks, participating at boat shows, training of HAM radio operator networks, etc.);
• NOAA Weather Radio, NOAA Weather Wire Service, Emergency Managers Weather Information Network, and other NWS dissemination systems; and
• public access (ease of use in accessing through digital telephone answering systems, visits, etc.).