Order JO 7210.633 Air Traffic Organization Quality Assurance Program
This order explains the responsibilities of ATO Safety and other offices within the ATO for conducting risk analysis, identifying system trends, and conducting assessments.
This document is cancelled. Cancelled by JO 7210.633A. Check the official source before relying on it.
FAA Order JO 7210.633
U.S. DEPARTMENT OF TRANSPORTATION
FEDERAL AVIATION ADMINISTRATION
Air Traffic Organization Policy
ORDER JO 7210.633
Effective Date: January 30, 2012
SUBJ: Air Traffic Organization Quality Assurance Program (QAP)
The Air Traffic Organization (ATO) has moved to a more systemic view of safety within the National Airspace System (NAS). This view places more value on discovering why adverse safety occurrences happen, and in identifying risks, rather than determining who was at fault. Historically, the ATO has not distinguished between quality assurance (QA) and quality control (QC) duties and responsibilities. A systemic approach requires clear understanding of the different roles under QA and QC. QA is responsible for identifying possible safety-related trends in the system rather than addressing single occurrences. QA is also responsible for ensuring all policies and procedures are being followed correctly and when not, whether mitigations, plans/efforts put in place are effective. QC is responsible for ensuring the quality of air traffic services provided at the service delivery point.
Elizabeth L. Ray Vice President, Mission Support Services Air Traffic Organization Date Signed
Distribution: ZAT-721; ZAT-464 Initiated By: AJS-0
ATO Safety
Table of Contents
Chapter 1. General … 1
1-1. Purpose of This Order … 1 1-2. Audience … 1 1-3. Where Can I Find This Order? … 1 1-4. Cancellation … 1 1-5. Distribution … 1 1-6. Definitions … 1
Table of Contents
Chapter 2. Safety Data … 3
2-1. Introduction … 3 2-2. Release of Safety Data … 3 2-3. Data Usage … 3 2-4. Collection Methods … 3 2-5. Information Requests … 5 2-6. Documentation Retention … 5
Chapter 3. Analysis … 7
3-1. Office of Safety Responsibilities … 7 3-2. Identification and Calculation … 7 3-3. Quality Assurance Validations in Support of Risk Analysis … 8
Chapter 4. Assessment … 9
4-1. Introduction … 9 4-2. Quality Assurance Assessment Responsibilities … 9 4-3. Assessment Responsibilities … 9 4-4. Non-EOR Radar ATC Facilities and Airport Traffic Control Towers (ATCT) Internal Review … 9
Chapter 5. Communications … 11
5-1. Recurring Reports … 11 5-2. Assessment Reports … 11 5-3. Significant RAE Occurrence Reports … 11
Appendices:
Appendix A. Separation Conformance (SC) … 12 Appendix B. Risk Analysis Event (RAE) Factors … 13 Appendix C. Commonly-Asked RAE Follow-Up Questions … 14
ii
Chapter 1. General › 1-6. Definitions
Chapter 1. General
1-1. Purpose of This Order. This order explains the responsibilities of ATO Safety and other offices within the ATO for conducting risk analysis, identifying system trends, and conducting assessments.
1-2. Audience. This order applies to all ATO personnel, FAA contract tower employees, and anyone using ATO directives.
1-3. Where Can I Find This Order? This order is available on the MyFAA Employees Web site at https://employees.faa.gov/tools_resources/orders_notices/ and the FAA Web site at http.//www.faa.gov/regulations_policies/orders_notices/.
1-4. Cancellation. This order cancels FAA Order JO 7210.56, Air Traffic Quality Assurance, and FAA Order JO 7010.1, Air Traffic Organization Safety Evaluations and Audits.
1-5. Distribution. This order is distributed to the following ATO service units: Terminal, En Route and Oceanic, Technical Operations, Mission Support, and System Operations; ATO Safety; the Air Traffic Safety Oversight Service (AOV); the William J. Hughes Technical Center; the Mike Monroney Aeronautical Center; National Air Traffic Controllers Association (NATCA); Professional Airway Systems Specialists; National Association of Government Employees; and to interested aviation public.
1-6. Definitions.
a. Closest Proximity – The smallest distance (point-to-point straight line in space) between two aircraft during an airborne loss of standard separation occurrence, regardless of geometry or percent of separation remaining.
b. Electronic Occurrence Report (EOR) – An alert identified by an automated system such as Traffic Analysis and Review Program (TARP) or Operational Error Detection Program (OEDP) that automatically uploads to the Comprehensive Electronic Data Analysis and Reporting (CEDAR) tool.
d. Mandatory Occurrence Report (MOR) – An occurrence involving air traffic services for which the collection of associated safety-related data and conditions is mandatory. See FAA Order JO 7210.632, Air Traffic Organization Occurrence Reporting, Appendix A, for a full listing of MORs.
e. Measure of Compliance (MOC) – During a loss of standard separation occurrence involving radar separation minima for which recorded radar data is available, the greatest percentage of remaining separation (vertical or lateral) at the point of lowest separation conformance, as calculated by ATO Safety.
f. Risk Analysis Event (RAE) – RAEs are loss of standard separation occurrences that have a MOC of less than 66%.
g. Risk Analysis Process (RAP) – The process used to identify key elements in RAEs and to give each RAE a value. ATO Safety will use occurrence data for input into a risk analysis tool that will then provide a risk assessment value associated with an individual occurrence. The process is conducted by a dedicated group of persons with air traffic control (ATC) experience and air transport-rated pilots with commercial experience.
h. Safety Risk Event Rate (SRER) – The rate of the most serious loss of standard separation for every thousand instances of loss of standard separation within the NAS. The SRER metric will first be calculated using only aircraft-to-aircraft loss of standard radar separation. As automated loss of standard
separation detection capabilities are improved, additional types of loss will be added to the calculation (for example, surface, oceanic, terrain/obstructions).
i. Separation Conformance – A value between 0 and 1.41 (1.41 = 100% separation on both axes) representing the total composite vertical and/or lateral separation maintained between each pair of recorded radar returns during a loss of standard separation occurrence. For example, an occurrence in which 3 miles lateral or 1000 feet vertical separation was required, at a point where two aircraft were separated by 2.3 miles and 800 feet, the separation conformance value would be 1.10. See Appendix A, Separation Conformance (SC), for detailed explanation of separation conformance value calculation.
j. Service Delivery Point (SDP) – An air traffic control facility or staffed technical operations facility.
Chapter 2. Safety Data › 2-4. Collection Methods
Chapter 2. Safety Data
2-1. Introduction. A successful safety system is highly dependent on accurate and timely data collection. ATO Safety is responsible for collecting all safety-related data within the ATO. ATO Safety will use collected data to properly identify and categorize suspected occurrences. Data will be analyzed for system trends and potential risks and will not normally be used as the basis for individual performance management.
2-2. Release of Safety Data. ATO Safety is the sole ATO office responsible for coordinating the release of safety data to organizations outside the ATO. Other organizations must not release safety data outside the ATO without prior coordination with ATO Safety.
2-3. Data Usage. ATO Safety will collect safety data to:
a. Accurately identify, categorize, and reconcile reported occurrences.
b. Accurately analyze RAEs.
c. Identify suspected risk trends.
d. Assess the effectiveness of risk mitigation actions.
e. Analyze NAS services.
f. Analyze policies and procedures for compliance and effectiveness.
2-4. Collection Methods. ATO Safety will collect safety data via the following methods:
a. CEDAR. ATO Safety will analyze data provided by ATC facilities and technical operations districts through the EOR and MOR submission processes within the CEDAR application.
b. Remote Collection. To minimize impact on ATC facilities’ and technical operations districts’ operations, ATO Safety will conduct remote data collection to the greatest extent practical using automation such as the National Offload Program (NOP), Falcon, Performance Data Analysis and Reporting System (PDARS), Digital Audio Legal Recorder (DALR), Remote Monitoring and Logging System (RMLS), etc.
c. Additional Data Requests. ATO Safety may require additional data, such as voice and radar data or NAS services data. The purpose of such requests is to accurately analyze an occurrence and is not intended to initiate a review of the occurrence by the facility. ATO Safety will indicate the format in which the information should be prepared, the scope of the information requested, and the method or means by which the information will be delivered. All data requests must be fulfilled within 2 administrative days.
d. ATC Facility and Technical Operations District Support of Analysis Data.
(1) ATO Safety will review all MORs/EORs indicating less than 66% separation maintained within 1 administrative day of acceptance of the work assignment in CEDAR. If ATO Safety determines a RAE has occurred, ATO Safety will notify the ATC facility POC via electronic communications by the close of business on the administrative day that the determination was made.
(2) The facility must ensure that all pertinent data are compiled and forwarded to ATO Safety in support of EOR/MOR validation and RAE analysis. Upon notification by ATO Safety that an
occurrence is a RAE or that additional data is required to assess the EORs/MORs, the facility must ensure the following:
(a) If requested by ATO Safety, all pertinent voice files must be attached to their associated EORs/MORs in CEDAR within 2 administrative days. At a minimum, these voice files must include all communications with and about the aircraft involved in the reported occurrence from either initial contact with the operating position responsible for the aircraft at the time of the suspected loss of standard separation occurrence, or 5 minutes before the reported occurrence, whichever happens last. The voice files must continue until 5 minutes after the reported occurrence or until final contact with the operating position, whichever happens first. Also include any recorded telephone conversations with involved flight crews concerning the occurrence. Ensure that occurrence-related communications with flight crews are conducted over recorded lines wherever available. Every effort should be made to use data files that are .wma, .avi, or .mp3, due to file size and bandwidth restrictions.
NOTE- Once notified by ATO Safety that an occurrence has been classified as a pilot deviation or is supporting other litigation, facility management must ensure that the original voice recording or acceptable waveform audio file (.wav) is retained according to applicable orders (see Paragraph 2-6, Documentation Retention).
(b) If requested by ATO Safety, all pertinent radar replay files must be attached to their associated EORs/MORs in CEDAR, within 2 administrative days. At a minimum, these radar data files should include the same time period covered by voice recordings described above.
(c) Within 5 administrative days, provide responses to all additional ATO Safety questions concerning the occurrence. See Appendix C, Commonly-Asked RAE Follow-Up Questions.
NOTE-
Technical Operations uses databases other than CEDAR. Until CEDAR is fully deployed throughout the ATO, ATO Safety data requests will require that Technical Operations personnel pull data from existing databases.
e. RAE Interviews. ATO Safety will advise SDP management when it is determined that additional data will be sought from interviews with employees with knowledge of an RAE. ATO Safety will advise facility management of all employees to be interviewed. Employees must be advised that RAE categorization is simply part of the analysis process and is not an indication of responsibility or severity. The purpose of this notification is to allow personnel to collect their thoughts and observations of the occurrence in support of possible interviews at a later date. Upon notification by ATO Safety of a RAE, facility management must complete notifications as follows:
(1) SDP management must advise the facility representative (FACREP) of all bargaining unit employees (BUE) to be interviewed within 1 administrative day. The Union representative or his/her designee may be present if the employee so requests. Union participants on interview panels will be present for all interviews.
(2) SDP management must notify all employees who may have knowledge about the RAE, including personnel responsible for controller-in-charge/watch supervision duties in the involved operational area(s) and all personnel responsible for associated operational positions (for example, hand-off, tracker, coordinator, etc.).
(3) Employee notifications must be accomplished as soon as practical, but no later than the end of the employee’s next assigned shift.
Chapter 2. Safety Data › 2-6. Documentation Retention
(4) No written statement is required from this notification; however, employees are encouraged to provide any information they may believe will benefit the review and to make personal notes for their future reference. Appendix B, Risk Analysis Event (RAE) Factors, provides a list of factors that may be considered by ATO Safety to complete its analysis of a RAE, and appendix C provides a list of questions commonly asked during RAE panels. ATC personnel involved in a RAE are encouraged to familiarize themselves with these factors and compile their recollections and observations of those pertinent to the occurrence.
(a) The facility must provide all posted work schedules of requested employees within 2 administrative days. Preferable dates/times that employees may be made available to minimize the impact to the operations should also be provided.
(b) Interviews of BUEs are voluntary; the union representative or his/her designee may be present if the employee so requests. Interviews will be conducted in collaboration with ATO Safety, the FACREP, and local facility management with at least 3 administrative days notice of the interview schedule (considering personnel and available dates/times). Once the interview time and date has been established, any changes to an employee’s schedule must be coordinated with the FACREP.
(c) The employee must be permitted to review all relevant data available within the facility prior to being interviewed. Union representation of the employee, at the election of the employee, must be granted.
(5) Information provided by involved employees is used in RAE panel deliberations and the panel determines what data points are captured regarding each RAE.
2-5. Information Requests. Information requests provide data needed to make informed decisions. They are for gathering additional data and do not require corrective action. Recipients may choose to initiate corrective action as a result of receiving a request. Requests to facilities or lines of business may be either informal or formal, but formal processes are documented and tracked. Formal information requests may be generated in those situations where informal data gathering may be impractical, not expedient, or otherwise inappropriate.
a. ATO Safety initiates formal information requests. The request initiates a bottom-up information exchange directed at the level most closely related to the issue.
(1) Recipients must provide a response within the identified timeline.
(2) If recipients do not provide a timely response, the issue may be escalated to a corrective action request (CAR).
b. If the response confirms the identified safety issue, ATO Safety will work with the respondent for resolution. If further action is required, the issue may be elevated to a CAR.
2-6. Documentation Retention.
a. FAA Order 1350.15, Records Organization, Transfer, and Destruction Standards, provides general requirements for data and record retention. FAA Order JO 8020.16, Air Traffic Organization Aircraft Incident Notification, Investigation, and Reporting, provides retention requirements for aircraft accidents, aircraft incidents, litigation, and enforcement support.
b. The following are the retention requirements for occurrences covered by FAA Order JO 7210.632 and this directive:
(1) ATO Safety must retain all data collected through the MOR and EOR processes for 24 months from the date of the initial report.
(2) Air traffic control facilities must retain all data concerning an RAE for 24 months from the date of the occurrence in facility files as follows:
(a) Label (maximum size 3"x5") clearly marked "Risk Analysis Event."
(b) RAE number assigned by ATO Safety.
(c) Incident local date and time.
(d) Date to be destroyed.
Chapter 3. Analysis › 3-2. Identification and Calculation
Chapter 3. Analysis
3-1. Office of Safety Responsibilities.
a. Providing trend analysis, statistical data, recommendations, and other pertinent information to assist field facilities with their risk mitigation efforts. Analysis of policy and procedures as established will be conducted periodically for compliance and effectiveness.
b. Analyzing safety data from NAS Services performance data; for example, RMLS, National Airspace Performance Reporting System.
c. Examining and reconciling occurrence reports collected through the EOR/MOR and assessment processes to ensure:
(1) The quality of the data is of the highest standard (for example, there are no duplicate or conflicting reports).
(2) Accurate categorization of occurrences to accomplish agency metric requirements (for example, loss of standard separation occurrences, Category A and B operational errors, runway incursions).
(3) Identification of occurrences that must be reported to other organizations (for example, pilot deviations, foreign facility deviation, and hazardous air traffic reports).
d. Conducting system risk analysis of all RAEs in accordance with this order and supporting standard operating procedures and ensuring findings are made available to the ATO (for example, observed and identifies trends, recommended mitigations).
3-2. Identification and Calculation. ATO Safety will identify or calculate the following from reconciled data:
a. The associated MOC of all airborne loss of standard separation occurrences.
b. All RAEs.
c. Applicable ATO and agency safety metrics.
d. All runway incursions.
e. Pilot deviations and near-midair collision (NMAC) reports which ATO Safety will forward to the responsible Flight Standards office.
f. Vehicle and pedestrian deviations which ATO Safety will forward to the Airports Division and other affected organizations.
g. Any foreign facility deviations which ATO Safety will forward to the appropriate state or organization.
h. Spillouts, military deviations, etc. which ATO Safety will forward to the Department of Defense.
Chapter 3. Analysis › 3-3. Quality Assurance Validations in Support of Risk Analysis
3-3. Quality Assurance Validations in Support of Risk Analysis.
a. ATO Safety validates loss of separation with associated radar data and determine which losses qualify as risk analysis events (RAE). QA analysis of RAEs provides data that supports the System Risk Event Rate (SRER) metric.
b. Validation and Measure of Compliance (MOC) calculation of airborne aircraft to aircraft loss of separation with radar data are made using the following methods.
(1) QA uses radar and voice data to validate loss of separation EOR and MORs.
(2) SC is calculated for each pair of recorded radar returns.
(3) The pair of radar returns with the lowest SC value is used to determine the MOC.
(4) MOC is calculated by assessing the percentage of separation maintained in the lateral and vertical axis. The greatest percentage of separation maintained is the MOC for that loss of separation.
NOTE- Closest proximity calculation values are utilized in evaluating severity of RAEs. Closest proximity is not used in validations or MOC determinations.
Chapter 4. Assessment › 4-4.
Chapter 4. Assessment
4-1. Introduction. Assessment is the process ATO Safety uses to confirm suspected risk trends identified from analysis and to determine the effectiveness of mitigation efforts. ATO Safety must document standard operating procedures for conducting assessments.
4-2. Quality Assurance Assessment Responsibilities.
a. ATO Safety will conduct both on-site and remote independent assessments in order to evaluate:
(1) The effectiveness of Safety Management System (SMS) performance and operations in the service units.
(2) The effectiveness of controls used to mitigate hazards identified via the Safety Risk Management (SRM) process.
(3) The effectiveness of ATC facilities’ and technical operations districts’ internal QC efforts (for example, operational skills assessment, system service review, certification, periodic maintenance, data integrity, modifications, availability, etc.).
(4) The effectiveness of QC mitigation efforts in response to identified trends and risks.
(5) Suspected trends identified from analysis conducted by QA and QC.
(6) The effectiveness of safety-related policies and procedures.
b. On-site assessment notification:
(1) Scheduled assessments such as of SMS and QC: ATO Safety will provide appropriate management, unless otherwise coordinated, with at least 90 days notice.
(2) Ad-hoc assessments such as of mitigation and risk trend: ATO Safety will provide appropriate management, unless otherwise coordinated, with at least 30 days notice.
(3) FAA Order 1100.161, Air Traffic Safety Oversight, requires that the ATO conduct no-notice inspections. ATO Safety must ensure compliance with this directive and ensure advance notification of such assessments is not provided.
4-3. Assessment Responsibilities. Upon notification by ATO Safety of an assessment, affected managers must ensure all personnel and data requested in support of the process are made available.
4-4. Non-EOR Radar ATC Facilities and Airport Traffic Control Towers (ATCT) Internal Review. All radar control facilities that do not have functioning capability for continuous automated loss of standard separation detection systems (for example, TARP, OEDP, etc.) and all ATCTs must establish written procedures to review random samples of radar and voice data to assess their effectiveness in identifying and reporting loss of standard separation occurrences and MORs.
a. Procedures must ensure that reviews include:
(1) Periods of known high-risk factors (for example, peak traffic times, low instrument meteorological conditions, etc.)
(2) System alerts (for example, low altitude alert, conflict alert, Airport Surface Detection Equipment (ASDE)/Airport Movement Area Safety System (AMASS), etc.)
(3) Reports from flight crews concerning quality of services, including Traffic Collision and Avoidance System (TCAS) occurrences.
b. These procedures must ensure a minimum of 2 hours of radar, if feasible, and 2 hours of voice data are reviewed each month, as well as the process for ensuring that all aspects of an air traffic control facility’s operations are periodically reviewed (for example, midnight-shift operations, unusual configurations, etc.).
c. All occurrences identified in the course of these reviews must be reported following FAA Order JO 7210.632 and other applicable directives.
d. To support oversight efforts, all radar, voice, and other supporting data used to conduct these reviews must be retained for 12 months following the date of the review.
Chapter 5. Communications › 5-3. Significant RAE Occurrence Reports
Chapter 5. Communications
5-1. Recurring Reports. ATO Safety will provide recurring safety data and trend findings as required by the Chief Operating Officer and senior executives. Copies of these reports will also be provided to external organizations and labor unions, as appropriate.
5-2. Assessment Reports.
a. ATO Safety will provide preliminary results of assessments to the service unit Vice President(s) with a copy provided to the ATO Deputy Chief Operating Officer and any affected union at the national level within 30 days of completion.
b. Corrective Action Requests. CARs are formal requests initiating action to resolve an identified concern. The CAR process is initiated when it becomes unlikely that identified safety issues will or should be resolved informally. CARs begin a top-down process to inform ATO of reported safety issues.
(1) CARs typically identify systemic safety issues and rarely are based on a single data point.
(a) All available information must accompany the request.
(b) Recipients must provide a response within the identified timeline. Although some issues are very complex and require additional time to develop a comprehensive corrective action plan, a response indicating ATO intentions is still required within the identified timeline.
(c) If recipients do not provide a timely response, or if the recipient requests an extension, ATO Safety will notify the appropriate Vice President.
(2) If the corrective action plan is sufficient, the issue will be closed.
(3) If ATO Safety does not initially concur with the response, they will work with the respondents in order to achieve resolution.
(4) If ATO Safety determines that the final response does not appropriately address the issue, the Office of Safety may elevate to the ATO Deputy Chief Operating Officer to pursue appropriate action.
5-3. Significant RAE Occurrence Reports. ATO Safety will notify the ATO Deputy Chief Operating Officer of all RAEs that are evaluated as most serious for the purposes of the SRER metric.
Appendices › Appendix A. Separation Conformance (SC)
Appendix A
Appendix A. Separation Conformance (SC)
SC is based on the percentage of required separation that was maintained in both the vertical and horizontal dimensions.
SC is used to establish equivalent values of separation for situations in which there is distance between aircraft in both dimensions that are equivalent to the separation in a single dimension (separation in the other dimension being zero). An equivalent loss of separation is assumed to correspond to a “composite slant range” in which the percent of the vertical separation maintained is equivalent to the percent of the horizontal separation maintained.
Given these assumptions, the composite slant range is a function of both vertical and horizontal and is computed as the following:
“Composite slant range” =
Appendices › Appendix B. Risk Analysis Event (RAE) Factors
Appendix B
Appendix B. Risk Analysis Event (RAE) Factors
The review of RAEs must include an in-depth inquiry into all causal factors. In a comprehensive review, the following factors should be considered:
a. Facility procedures.
b. Facility training.
c. Facility supervision.
d. Equipment.
e. Control environment.
f. External factors.
g. Controller action vs. inaction.
h. Airspace configuration.
i. Traffic flow/volume/initiatives.
j. Pilot actions, including the consequence of any Traffic Collision and Avoidance System (TCAS) occurrence.
k. Route of flight.
l. Weather.
m. Position configuration.
n. Coordination procedures.
o. Human factors.
p. System times of all pertinent equipment (if they differ).
q. Staffing levels and/or position assignments with regard to complexity/volume.
r. Training and/or proficiency/experience levels of involved personnel.
s. Automation issues/configuration.
Appendices › Appendix C. Commonly-Asked RAE Follow-Up Questions
Appendix C
Appendix C. Commonly-Asked RAE Follow-Up Questions
1. What was the first indication of the conflict?
2. What was traffic complexity score, on a scale of 1 to 5 (5 being the most complex)?
3. What were factors that affect the traffic complexity score?
4. Was any evasive action or response taken by the pilot(s) due to a Traffic Collision Avoidance System (TCAS) resolution advisory (RA)?
5. Was any conflict alert information available to air traffic control (was it available, did it activate, etc.)?
6. Were there any distractions or environmental conditions that may have contributed to the occurrence?
7. What was the staffing status of radar/radar associate/tracker/hand-off/coordinator position, etc.?
8. What was the status of positions (combined/de-combined)?
9. Was there a controller-in-charge or front line manager-in-charge?
10. Was on-the-job training instruction in progress?
11. What were shift schedules (prior to and day of occurrence)?
12. What was time on position?
13. What was controller(s) status (developmental/certified professional controller)?
NOTE- Facility personnel familiar with the occurrence should be prepared to provide a detailed narrative of the occurrence.