Date of Issue:			April 13, 2001		Effective Date:		April 13, 2001
In Reply Refer To:				W/OPS12	File With:	A-13
Subject:	Reporting Systems, Equipment, and Communication Outages
References:		J-02 Backup Operations

1. Purpose. The purpose of this policy is to ensure senior level managers are made aware of system, equipment, and communication outages that threaten or could threaten public safety or are otherwise specified in Section 3 of this policy.

2. Responsibilities.

2.1 Weather Forecast Offices (WFO), River Forecast Centers (RFC), Data Collection Offices, Center Weather Service Units, West Coast/Alaska Tsunami Warning Center, Pacific Tsunami Warning Center, National Centers for Environmental Prediction (NCEP) Central Operations, Hydrometeorological Prediction Center (HPC), Aviation Weather Center (AWC), Storm Prediction Center (SPC), Marine Prediction Center (MPC), Tropical Prediction Center (TPC), National Weather Service Telecommunication Gateway (NWSTG), and National Data Buoy Center (NDBC). If public safety is or could be affected by system, equipment, or communication failure, the senior individual on duty at the site will report immediately by telephone (voice contact) or pager to a designated point of contact. Points of contact will be specified by the regional director, Director of NCEP, and Director of the Office of Operational Systems (OPS). These reports are referred to as "Incident Reports" and are described in Section 4.1. Sites and Centers will follow up incident reports with an email or other written documentation covering items listed in Appendix A. Incident reports will be documented and tracked in the daily report described in Section 4.2. For outages specified in Section 3 but not requiring incident reports, the senior individual on duty at the site will contact designated regional, NCEP, or OPS officials by e-mail or telephone(voice or answering machine). These outages also will be recorded and tracked in the daily report.

2.2 Regional Directors, Director of the National Centers for Environmental Prediction, and Director of the Office of Operational Systems. Each director will establish written procedures specifying points of contact for outages requiring immediate reporting. When a point of contact is notified of a system outage that threatens or could threaten public safety, he or she will notify the regional director, Director of NCEP, or Director of OPS. During normal business hours (Eastern time), the director will notify the Assistant Administrator for Weather Services and the Deputy Assistant Administrator for Weather Services of outages if mission impact, public visibility, or political sensitivity warrant such notification. Otherwise such notification to the Assistant Administrator and Deputy Assistant Administrator will take place at the beginning of the next business day by voice contact or the highest priority level email. Directors will provide daily reports to the Chief of the Maintenance, Logistics, and Acquisition Division, Office of Operational Systems by 11:30 AM, each business day. (Note: Alaska and Pacific Regions will provide reports as of their COB the previous business day.)

2.3 Regional Systems Operations Division Chiefs, Directors of NCEP Central Operations, HPC, AWC, SPC, TPC, NDBC, and Chief of the Telecommunication Operations Center. Each business day, the regional systems operations division chiefs will provide a report to the regional director on all outages specified in Section 3. Each business day, the directors of NCEP Central Operations, HPC, AWC, SPC, MPC, TPC, and the Chief of the Telecommunication Operations Center will provide a report to their director on all computer systems and communications outages specified in Section 3. The Director of NDBC will provide a report to the Director of OPS only when the status of an existing outage changes or a new outage occurs.

2.4 Chief of the Maintenance, Logistics, and Acquisition Division, Office of Operational Systems. The Chief of the Acquisition, Maintenance, and Logistics Division will prepare a consolidated daily report and submit it to the Assistant Administrator for Weather Services and the Deputy Assistant Administrator for Weather Services each business day.

3. Reporting Requirements for Systems, Equipment, and Communications Outages.

Failure of an Automated Weather Interactive Processing System (AWIPS) that requires implementation of service backup, as described in Weather Service Operations Manual Chapter J-02, during weather or hydrologic conditions that threaten or could threaten public safety will be reported immediately. In less critical circumstances, report all outages requiring implementation of service backup.

Weather Service Radar-88 Doppler (WSR-88D), NOAA Weather Radio, WFO/RFC voice communications, Frame Relay Circuit, or associated equipment failure during weather or hydrologic conditions that threaten or could threaten public safety will be reported immediately. In less critical circumstances, report all failures expected to last more than 12 hours.

All failures of upper air equipment expected to last more than 24 hours.

Failure of an Automated Surface Observing System (ASOS) that is not expected to be restored within established restoration times.

Total failure of Data Buoys and Coastal Marine Automated Network (C-MAN) stations will be reported immediately upon confirmation of the failure.

NCEP Central Operations will report immediately outages and missing individual model runs if an outage is projected to last longer than one forecast cycle.

Failure of mission-critical computer systems and communication capabilities at HPC, AWC, SPC, MPC, TPC, or the NWSTG (including the AWIPS Satellite Broadcast Network) for which on-site backup cannot be invoked and standard operating procedures fail to restore service. Report immediately when backup or restoration steps fail. Report all other outages if failure is expected to last more than 12 hours.

• Reports.
  
  

4.1 Incident Reports. Incident Reports will include the date and time the outage began, the projected restoration date and time, actions being taken to restore the outage, an assessment of the effect of the outage on services, and severe weather conditions. Incident reports will be followed up by an email or other written documentation. This documentation will address the items listed in Appendix A. When outages are restored, the time of restoration will be reported. All times given in reports will be UTC.

4.2 Daily Reports. The daily report will consist of two sections: current outages and outages closed since the last report. Within each part, the report will be organized by system (e.g., WSR-88D, ASOS, mainframe computer, FTP server). Within each system category, sites will be listed from longest to shortest outage. For each outage, the hours of outage to date and projected date and time of restoration will be separately listed along with the cause of the outage. For current outages, the projected total outage hours will be used in listing the sites from the longest to shortest outage. The cause of each outage, the actions being taken to restore the outage, the effect on services, and any severe weather that took place during the outage will be listed. The second part of the report, outages closed since the last report, will list the total hours of the outage and the date and time the outage was closed. In both parts of the report, outages that required incident reports will be distinguished by appearing in bold print. A report format for the daily report is included as Appendix B. All times given in reports will be UTC.

John J. Kelly, Jr.

APPENDIX A POLICY ON REPORTING SYSTEMS, EQUIPMENT, AND COMMUNICATION OUTAGES

Checklist for Email Follow-ups to Incident Reports

All incident reports follow-up emails should cover the following:

1. System, equipment, or communication capability that is out.

2. Site and responsible WFO/RFC.

3. Date/time outage began.

4. Projected restoration date and time. 5. Actions being taken to restore system, equipment, or communications capability.

6. Effect on services.

7. Severe weather conditions occurring during outage.

(Note: all times should be given in UTC.)

APPENDIX B POLICY ON REPORTING SYSTEMS, EQUIPMENT, AND COMMUNICATIONS OUTAGES

Organization:_________________________ Daily Outage Report

Date:_______ Time:________

I. Current Outages

COMMS OR SYSTEM	SITE (write out)	WFO/ RFC	DATE/ TIME OUTAGE BEGAN	OUTAGE HOURS TO DATE	PROJECTED DATE & TIME OF RESTORE	PROJECTED TOTAL OUTAGE HOURS	CAUSE	ACTIONS BEING TAKEN TO RESTORE	EFFECT ON SERVICES	SEVERE WX CONDITIONS
	-	-	-	-	-	-	-	-	-	-
	-	-	-	-	-	-	-	-	-	-
	-	-	-	--	-	-	-	-	-	-
	-	-	-	-	-	-	-	-	-	-
	-	-	-	-	-	-	-	-	-	-

APPENDIX B POLICY ON REPORTING SYSTEMS, EQUIPMENT, AND COMMUNICATION OUTAGES

II. Outages Closed Since Last Report

COMMS OR SYSTEM	SITE (write out)	WFO/ RFC	DATE/ TIME OUTAGE BEGAN	TOTAL HOURS OF OUTAGE	DATE & TIME RESTORED	CAUSE	ACTIONS TAKEN TO RESTORE	EFFECT ON SERVICES	SEVERE WX CONDITIONS
-	-	-	-	-	-	-	-	-	-
-	-	-	-	-	-	-	-	-	-
-	-	-	-	-	-	-	-	-	-
-	-	-	-	-	-	-	-	-	-
-	-	-	-	-	-	-	-	-	-

Notes: 1. Outages are to be grouped by system.

2. Within each system category, sites are listed from longest to shortest outage. For current outages "Projected Total Outage

Hours" is used for this purpose.

3. Outages that caused an Incident Report are listed bold print.

4. If the outage had no effect on services, enter "none" in that column.

5. If there were no severe weather conditions during the outage, enter "none" in that column.

6. NWSTG and NCEP Central Operations do not have to fill in the severe weather conditions column.

7. Times should be given in UTC.