Order JO 7210.634A Air Traffic Organization (ATO) Quality Control
The purpose of quality control, as defined in the Air Traffic Organization (ATO), is to assess the output (whether a product or service) of a particular process or function and identify any deficiencies, risk, or problems that need to be addressed. Within this quality control concept, it is a primary responsibility to take action, particularly at the Service Delivery Point (SDP), to ensure that these products or services meet the requirements of the SDP and the ATO organizationally. This order outlines the processes and steps used to ensure the quality of products and services provided at the SDP level on an ongoing basis.
U.S. DEPARTMENT OF TRANSPORTATION › Distribution: Electronic Initiated By: AJI-0
CHANGE
U.S. DEPARTMENT OF TRANSPORTATION JO 7210.634A
CHG 1 FEDERAL AVIATION ADMINISTRATION
Effective Date: Air Traffic Organization Policy September 1, 2021
SUBJ: Air Traffic Organization (ATO) Quality Control
1. Purpose of This Change. This change updates guidance for the Technical Operations Services Quality Control Program and makes other editorial changes throughout the document.
2. Audience. This change applies to the following Air Traffic Organization (ATO) service units: Technical Operations Services (AJW), Safety and Technical Training (AJI), Mission Support Services (AJV), System Operations Services (AJR), and Air Traffic Services (AJT).
3. Where Can I Find This Change? This order is available on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/ and the Federal Aviation Administration website at http://www.faa.gov/regulations_policies/orders_notices/.
4. Explanation of Policy Change.
a. Cancels the System Service Review Guidance memorandum signed by the Vice President of Technical Operations Services on May 2, 2018, incorporating all pertinent guidance into this Change.
b. Replaces Chapter 8, Technical Operations Quality Control Programs.
c. Adds new Appendix F, Technical Operations Services System Service Review (SSR), Corrective Action Plan (CAP), and Systemic Issue Review (SYSIR) Templates.
d. Updates the definition of Aviation Risk Identification and Assessment (ARIA).
e. Replaces references to Performance Skill Checks with Performance Assessments.
f. Deletes a legacy reference to Electronic Occurrence Report (EOR).
5. Distribution. This change is distributed to the following ATO service units: AJT, AJW, AJR, AJI, and AJV. In addition, the order is distributed to the following: the Air Traffic Safety Oversight Service (AOV), the William J. Hughes Technical Center, the Mike Monroney Aeronautical Center, the National Air Traffic Controllers Association (NATCA), Professional Aviation Safety Specialists (PASS), the National Association of Government Employees (NAGE), and to the interested aviation public.
6. Background. This change updates the Technical Operations Services Quality Control (QC) Program by providing specific procedures and processes for SSR, SYSIR, CAP, and Compliance
Distribution: Electronic Initiated By: AJI-0
Verification used to measure the quality of AJW products and services. Additionally, this change incorporates several minor editorial updates and a change to the ARIA definition.
7. Disposition of Transmittal. Retain this transmittal until superseded by a new basic order.
8. Page Control Chart. See below.
PAGE CHANGE CONTROL CHART
Remove Pages Dated ii and iii 10/1/20 1-1 10/1/20 1-3 through 1-6 10/1/20 2-3 and 2-4 10/1/20 4-11 and 4-12 10/1/20 6-1 through 6-3 10/1/20 8-1 through 8-3 10/1/20 B-1 10/1/20 C-5 10/1/20 D-4 10/1/20 E-1 and E-2 10/1/20
Insert Pages Dated ii and iii 9/1/21 1-1 9/1/21 1-3 through 1-6 9/1/21 2-3 and 2-4 9/1/21 4-11 and 4-12 9/1/21 6-1 through 6-3 9/1/21 8-1 through 8-6 9/1/21 B-1 9/1/21 C-5 9/1/21 D-4 9/1/21 E-1 and E-2 9/1/21 F-1 through F-6 9/1/21
Glenn A. Martin Vice President, Safety and Technical Training Air Traffic Organization
Distribution: Electronic Initiated By: AJI-0
U.S. DEPARTMENT OF TRANSPORTATION
FEDERAL AVIATION ADMINISTRATION
ORDER
JO 7210.634A
Air Traffic Organization Policy Effective date:
10/01/2020
SUBJ: Air Traffic Organization (ATO) Quality Control
The ATO has moved to a systemic outlook of safety within the National Airspace System. This outlook places more value on discovering why adverse safety occurrences happen, and identifying risk, rather than determining who was at fault. Historically, the Federal Aviation Administration (FAA) has experienced external and internal organizational imperatives that have encouraged a safety system focused on the outcome of individual incidents, and required the rapid reporting of preliminary incident information. While the FAA has enjoyed success under this past safety model, our current Safety Management System provides for a more measured and systemic approach to safety risk analysis. Quality control is an essential component of this system.
Glen A. Martin Vice President, Safety and Technical Training Air Traffic Organization
Distribution: Electronic Initiated By: AJI-0
5-4. › 5-2
Table of Contents
Paragraph Page
Chapter 1. General … 1-1
1-1. Purpose of This Order … 1-1
1-2. Audience … 1-1
1-3. Where Can I Find This Order? … 1-1
1-4. Cancellation … 1-1
1-5. Explanation of Policy Changes … 1-1
1-6. Distribution … 1-1
1-7. Organizational Responsibilities … 1-1
1-8. Definitions … 1-3
1-9. Related Publications … 1-6
Chapter 2. Quality Control Model … 2-1
2-1. Quality Control (QC) Model … 2-1
2-2. Facility/District QC Orders … 2-3
2-3. Local (Facility) Safety Reports … 2-4
Chapter 3. Quality Control Monitoring … 3-1
3-1. Introduction … 3-1
3-2. QC Operational Skills Assessment (OSA) … 3-2
3-3. Emphasis Items (EIs) … 3-3
Chapter 4. Service Reviews … 4-1
4-1. Introduction … 4-1
4-2. SSR … 4-2
4-3. TMR … 4-5
4-4. CER … 4-7
4-5. SYSIR … 4-9
Chapter 5. Compliance Verification … 5-1
5-1. Compliance Verifications (CVs) … 5-1
5-2. ICV … 5-1
5-3. ECV … 5-2
5-4. Findings … 5-2
ii
Appendix F. › F-1
Paragraph Page
5-5. Responding to Findings … 5-3
Chapter 6. Quality Control Validations … 6-1
6-1. Quality Control Validations (QCVs) … 6-1
Chapter 7. Corrective Action Plans … 7-1
7-1. Introduction … 7-1
7-2. Develop and Implement CAPs … 7-1
7-3. Monitoring … 7-2
7-4. Documentation … 7-4
Chapter 8. Technical Operations Services Quality Control Program … 8-1
8-1. Purpose … 8-1
8-2. Technical Operations Services System Service Review (SSR) … 8-1
8-3. Technical Operations Services Systemic Issue Review (SYSIR) … 8-4
8-4. Technical Operations Services Corrective Action Plan (CAP) … 8-4
8-5. Technical Operations Services Compliance Verification (CV) … 8-5
Appendix A. Alaska Flight Service Station (FSS) and U.S. NOTAM Office (USNOF) Quality Control Compliance Verification (CV) … A-1
Appendix B. Federal Contract Flight Service Station (FCFSS) Quality Control
Appendix E. Sample Fiscal Year (by Month) QC Activity Plan Large
… B-1
Appendix C. Quality Control Elements … C-1
Appendix D. Generic Facility Quality Control Order Large Tower/TRACON … D-1
Tower / TRACON … E-1
Appendix F. Technical Operations Services SSR, CAP, and SYSIR Templates … F-1
iii
Chapter 1. General
1-1. Purpose of This Order. The purpose of quality control, as defined in the Air Traffic Organization (ATO), is to assess the output (whether a product or service) of a particular process or function and identify any deficiencies, risk, or problems that need to be addressed. Within this quality control concept, it is a primary responsibility to take action, particularly at the Service Delivery Point (SDP), to ensure that these products or services meet the requirements of the SDP and the ATO organizationally. This order outlines the processes and steps used to ensure the quality of products and services provided at the SDP level on an ongoing basis.
(2) › Ensure that their respective organizations comply with the requirements…
1-2. Audience. The chapters of this order apply to ATO service units as follows: Chapters 1–7 and related Appendices: Air Traffic Services, System Operations Services; Chapter 1, Chapter 8, and Appendix F: Technical Operations Services; and this entire order is applicable to Mission Support Services due to their unique support role for the service units.
1-3. Where Can I Find This Order? This order is available on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/ and the Federal Aviation Administration (FAA) website at http://www.faa.gov/regulations_policies/orders_notices/.
1-4. Cancellation. This order cancels FAA Order JO 7210.634, Air Traffic Organization (ATO) Quality Control.
1-5. Explanation of Policy Changes. This revision adds and deletes definitions in paragraph 1-8; incorporates Aviation Risk Identification and Assessment (ARIA), Barrier Analysis Review (BAR), Preliminary ARIA Report (PAR), Referred ARIA Report (RAR), and Combined Safety Barrier Review (CSBR); clarifies organizational responsibilities; adds Chapter 2 and the Quality Control Model; clarifies requirements of the Service Review and Compliance Verification (CV) processes; incorporates ATO Safety Guidance ATO-SG-15-03 and ATO-SG-12-05; incorporates Corrective Action Plans (CAPs) and processes; deletes Quality Control Checks and associated processes; adds Performance Assessment Validations; adds and updates the charts contained in the Appendices; and makes general organizational and editorial updates.
1-6. Distribution. This order is distributed to the following ATO service units: Air Traffic Services (AJT), Technical Operations Services (AJW), System Operations Services (AJR), Safety and Technical Training (AJI), and Mission Support Services (AJV). In addition, the order is distributed to the following: the Air Traffic Safety Oversight Service (AOV), the William J. Hughes Technical Center, the Mike Monroney Aeronautical Center, the National Air Traffic Controllers Association (NATCA), Professional Aviation Safety Specialists (PASS), the National Association of Government Employees (NAGE), and the interested aviation public.
1-7. Organizational Responsibilities.
a. Vice Presidents, or designees, of AJT, AJR, AJW, and AJV must:
(1) Develop all policies and procedures related to quality control.
(2) Ensure that their respective organizations comply with the requirements of this order.
(7) › c. Service Center Directors must:
(3) Provide oversight and support to their respective Director(s) of Operations, Service Center Directors, and Deputy Directors of System Operations to ensure they are able to meet their quality control requirements as described in this order, and ensure they are responding to data indicators of non-compliance and risk.
(4) Continually review available quality control data and information and develop initiatives and/or take actions(s) when appropriate.
(5) Coordinate with other ATO service units, as appropriate, when developing initiatives or taking actions(s) to ensure organizational consistency and effective resource management and communication.
(6) Review safety data analysis provided by ATO Safety and Technical Training, and initiate action(s) when appropriate. Communicate the results of such reviews to ATO Safety and Technical Training.
(7) Respond to requests and/or actions initiated by ATO Safety and Technical Training in their quality assurance role.
b. Directors of Operations must:
(1) Ensure their organization complies with the requirements of this order.
(2) Provide oversight and support to their respective General Managers, Technical Operations District/Group Managers, and/or SDP Managers to ensure they are able to meet and comply with their quality control requirements as described in this order.
(3) Continually review available quality control data and information and develop initiatives and/or take action(s) when appropriate.
(4) Coordinate with other ATO organizations within their service area (Air Traffic Services, Technical Operations Services, the Service Center, ATO Safety and Technical Training, etc.) and service unit headquarters staff as appropriate when developing initiatives or taking actions(s) to ensure organizational consistency and effective resource management and communication.
(5) Examine operational compliance and safety related data provided by Service Center Directors, delivered by the applicable service center group(s), and initiate action(s) when appropriate.
(6) Review data analysis provided by ATO Safety and Technical Training and initiate action(s) when appropriate. Communicate the results of such reviews to ATO Safety and Technical Training.
(7) Respond to requests and/or actions initiated by ATO Safety and Technical Training in their quality assurance role.
c. Service Center Directors must:
e. › Combined Safety Barrier Review Output.
(1) Ensure that their organization complies with the requirements of this order.
(2) Coordinate with the Director(s) of Operations and assist and ensure that the Director(s) of Operations meet their quality control requirements as described in this order.
(3) Ensure that Service Center groups coordinate their actions to support Service Center requirements.
d. SDP Manager, General Manager, Assistant General Managers, and Technical Operations
Services District/Group Managers must:
(1) Ensure that their respective organization complies with the requirements of this order.
(2) Provide oversight and support to their subordinate managers to ensure that they are able to meet their quality control requirements as described in this order.
(3) Continually review available quality control data and information and develop initiatives and/or take actions when appropriate.
(4) Coordinate with their Director of Operations and/or Service Center Quality Control Group, and other SDPs as appropriate, when developing initiatives or taking actions to ensure organizational consistency and effective resource management and communication.
(5) Respond to requests and/or actions initiated by Safety and Technical Training in their quality assurance role.
e. Flight Service Stations (FSSs) / U.S. NOTAM Office / Federal Contract Flight Service Stations (FSSs) must conduct quality control measures according to Appendix A and Appendix B, respectively.
1-8. Definitions.
a. Aviation Risk Identification and Assessment (ARIA). An automated system that helps employ risk-based, data-driven decision-making facilitating better insight into potential risk in the National Airspace System (NAS).
b. Barrier Analysis Review (BAR). The process used to assess severity, likelihood, and barrier effectiveness in Referred ARIA Reports. Barrier analysis is also used to identify and assess factors (mitigating, aggravating, or observed) for air traffic operations where at least one aircraft is receiving Air Traffic Control (ATC) services.
c. BAR Report. The output of the BAR process.
d. Checklists. Checklists are used as minimum guidance in preparing for and conducting Internal Compliance Verifications (ICVs) / External Compliance Verifications (ECVs).
e. Combined Safety Barrier Review Output. The resulting output from the Combined Safety Barrier Review process.
m. › Direct Monitoring.
f. Combined Safety Barrier Review (CSBR). A cooperative process between Quality Assurance (QA) and facilities to gather additional information from subject matter experts and inform all concerned individuals about potential areas of risk in the system. This process utilizes aggregate data from BAR (if available) and includes facility stakeholders in an effort to identify, assess, and mitigate risk present in the operation.
g. Compliance Verification Tool (CVT). A national database that contains information related to the compliance verification process. Information includes checklists, reports, facility information, tracking information, response data, and other statistical information available on the CVT website. Information contained in reports, any corrective action, status reports, and closure is submitted through this database system.
h. Comprehensive Electronic Data Analysis and Reporting (CEDAR). A web-based, comprehensive data reporting, collection, and analysis tool used by both Quality Control (QC) and QA to record data associated with their respective organizational responsibilities.
i. CEDAR Question Tree. An electronic entry method used within the quality control service review processes to capture specific data points regarding information about operational circumstances during the period under review. Question trees are based on a user-friendly question and answer format that guides the user through the electronic “form.”
j. Conformity Index (CI). This item applies to federal and federal contract flight service stations only. Each on-site ICV/ECV conducted by the Flight Service Directorate must include a CI. The CI must essentially be the result of aggregating the weighted indices for each of the functional areas (system safety, system efficiency, and system management) on the national checklist. System safety is weighted more than the other functional areas. Instructions for calculating the CI are available from the CVT. The Flight Service Directorate acknowledges that no two facilities are identical; therefore, CIs are not intended to compare facilities. The intent of the CI is to numerically depict a facility’s overall compliance with directives/regulations and to assist with identifying “at risk” facilities for non-compliance.
k. Corrective Action Plan (CAP). CAPs are collaborative activities enacted to correct non-compliance and areas of risk that have been properly identified, validated, and understood through data collection and analysis. A CAP contains a description of the mitigation actions, the scope of the CAP, a timeframe for completion, a defined monitoring plan, and a defined effectiveness target.
l. Desk Audit. A desk audit is an off-site method of assessing checklist and off-checklist items. It is accomplished through discussion with facility personnel and/or review of requested or obtained recordings, data, and/or documentation.
m. Direct Monitoring. Monitoring an operational position real-time from the same position (for example, monitoring arrival east while physically sitting at arrival east).
y. › Significant Safety Risk.
n. External Compliance Verification (ECV). An externally initiated assessment of a facility, conducted primarily by the Quality Control Group (QCG) and/or additional personnel, in response to data-driven indicators of potential risk and/or practices. An ECV may be conducted on-site, using a customized checklist, to assess a facility’s overall performance. ECVs are conducted on an as-needed basis as determined via indicators of potential risk and non-compliance.
o. Internal Compliance Verification (ICV). A facility’s self-evaluation that is conducted by the facility / designated personnel using the checklists contained in the CVT and procedures outlined in this directive.
p. Off-Checklist Item. An assessed item that is not specifically identified on a national checklist.
q. Operational Skills Assessment (OSA) Worksheet. An electronic worksheet in CEDAR used to document an individual’s technical performance.
r. Playback Monitoring. Indirectly monitoring an operational position at any date/time other than live/real-time using available playback tools (Falcon, etc.) synchronized with a voice file.
s. Preliminary ARIA Report (PAR). An initial report of an air traffic operation identified by ARIA for further review by QA personnel.
t. Quality Assurance Group (QAG). The office in each service area responsible for conducting occurrence classification, identifying and categorizing air traffic incidents, identifying reports for barrier analysis, and conducting barrier analysis reviews in support of the ATO Top 5 and other safety processes. The mission of the QAGs may encompass other duties, including identifying significant safety risk trends and identifying potential significant events.
u. Quality Control Group (QCG). The office in each service center responsible for conducting data-driven external compliance verifications, identifying and delivering SDP non-compliance/risk data to the Directors of Operation and SDPs, assessing the effectiveness of SDP corrective actions, facilitating the significant event process, and monitoring SDP QC programs while ensuring that they are in compliance and completed in accordance with directives.
v. Referred ARIA Report (RAR). Subset of Preliminary ARIA Reports identified for BAR.
w. Remote Monitoring. Indirectly monitoring an operational position in real-time from a remote location (watch desk, operations supervisor desk, other operational position, etc.).
x. Service Delivery Point (SDP). An air traffic control facility, flight service station, or staffed/unstaffed technical operations facility.
y. Significant Safety Risk. Identified ATO exposure to risk that has a high likelihood of occurrence and/or severity.
l. › FAA Order 6000.15, General Maintenance Handbook for National Airspace…
z. Special Evaluation. This item applies to federal contract flight service stations only and assesses specific areas, programs, offices, or organizations as directed by the Flight Service Directorate.
aa. Systemic. An identified safety concern that has the potential to introduce risk into the national airspace system at the local, district, regional, service area, or national level.
bb. Voluntary Safety Reporting Program (VSRP). A voluntary program that provides a confidential, non-punitive mechanism for employees to report safety events and problems. VSRPs use employee input to identify leading indicators and significant safety concerns and issues, operational deficiencies, non-compliance with regulations, deviations from policies and procedures, and potential risk in the system. Qualitative data received through this program helps identify indicators of potential hazards and areas of risk in the NAS.
1-9. Related Publications.
a. FAA Order JO 7210.632, Air Traffic Organization Occurrence Reporting.
b. FAA Order JO 7210.633, Air Traffic Organization (ATO) Quality Assurance (QA).
c. FAA Order JO 3400.20, Individual Performance Management (IPM) for Operational Personnel.
d. FAA Order JO 1030.3, Initial Event Response.
e. FAA Order JO 7200.21, Partnership for Safety Program.
f. FAA Order JO 7200.20, Voluntary Safety Reporting Programs.
g. FAA Order JO 3120.4, Air Traffic Technical Training.
h. FAA Order JO 7110.65, Air Traffic Control.
i. FAA Order JO 1000.37, Air Traffic Organization Safety Management System.
j. FAA Order JO 7110.10, Flight Services.
k. FAA Order JO 6040.6, National Airspace System Technical Evaluation Program.
l. FAA Order 6000.15, General Maintenance Handbook for National Airspace System (NAS) Facilities.
Figure 2-1: The Five-Step Process/5 QC Elements › Figure 2-1: The Five-Step Process/5 QC Elements
Chapter 2. Quality Control Model
2-1. Quality Control (QC) Model.
a. A strong QC program integrates five QC elements: the five-step process, Local (facility) Safety Reports, the Local Safety Council (LSC), local QC orders, and a QC activity plan. The core of these five elements is the five-step process. Together, the five elements and the five-step process form a dynamic “5 x 5” QC program model. The five-step process is a continuous and cyclical process. The five-step process and additional QC elements are listed below and displayed in Figure 2-1. More information about QC Elements are contained in Appendix C.
(1) Collect data (supports initial identification of non-compliance or risk, and monitors implemented corrective actions).
(2) Validate and understand potential facility problems/issues (assess/analyze).
(3) Develop and implement Corrective Action Plans (CAPs).
(4) Document findings/actions.
(5) Review data for integrity.
Figure 2-1: The Five-Step Process/5 QC Elements
(2) › Validate and Understand Facility Problems (Assess/Analyze).
b. It is important to recognize that Individual Performance Management (IPM) is not a component of QC. IPM is a separate but parallel process that contributes to a strong proactive safety management system. Field facilities must operate QC processes as a systemic (deidentified) data collection method and perform IPM as a separate process following the specific steps outlined in FAA Order JO 3400.20.
(1) QC processes, procedures, and policies that use direct employee (identified) observation as a foundation are not in compliance.
(2) Facilities must not establish a QC program that requires identified observation of employees outside of approved IPM OSA processes.
(3) Facilities must not use QC activities to generate, or as the source of, IPM follow-up activities.
(4) QC activities collect and analyze systemic compliance, risk, and drift and are not to be attributed to or identify an individual employee.
c. Below is an outline of the five-step process.
(1) Collect Data. Facilities must collect/review data through the following:
(a) QC OSAs
(b) Emphasis Items
(c) Compliance Verifications
(d) Partnership for Safety (PFS) Safety Data Portal (if applicable; must have a LSC)
(e) Mandatory Occurrence Report (MOR) and Barrier Analysis Review (BAR) data (if applicable) to identify potential patterns or trends that may be indicators of facility noncompliance and risk. The review should not exclusively focus on loss of separation occurrences. It is expected that MOR and BAR data be reviewed on a regular basis, however, based on the volume of potential occurrences that can be generated at some facilities, this review is not intended to create a workload that consumes all QC efforts or resources.
(f) MORs and BAR data must not be the primary basis of a facility QC program. Facilities must ensure they comply with the requirements prescribed in the NATCA/FAA Collective Bargaining Agreement when using MOR data to support IPM activities.
(g) Facilities should review aggregate groupings or trends of factors identified in barrier analysis and use them in conjunction with other QC data to provide additional information that will enable them to better focus their service reviews or to create Emphasis Items to collect additional data.
(2) Validate and Understand Facility Problems (Assess/Analyze). Facilities must ensure they properly understand potential identified problems by rigorously assessing collected
U.S. DEPARTMENT OF TRANSPORTATION › (1) QC OSA sampling plan (see Appendix D).
data through standardized processes. Before developing a CAP or invalidating a potential systemic issue, facilities must assess potential issues by using one of the following processes:
(a) Service reviews.
(b) Compliance verifications.
(c) CSBR. Requests for CSBR are made through the respective service area QA office.
REFERENCE– FAA Order JO 7210.633, Chapter 3, Requesting Combined Safety Barrier Review.
NOTE– It is understood that with some significant events or compliance issues, facilities may need to implement corrective actions prior to conducting an assessment. This should happen only under unusual circumstances. When this occurs, a service review must be conducted as soon as possible to validate or modify the issue and CAP.
(3) Develop and Implement CAPs. Once a problem is understood (assessed and analyzed), facilities must collaboratively develop a CAP. CAPs must be designed to address the specific problem and be implemented throughout the facility or applicable operational area. In addition, CAPs must include how the effectiveness of implemented mitigations will be assessed. Facilities must monitor implemented CAPs as they continue to collect data. This can be done through performing QC OSAs, designing specific Emphasis Items to assess a specific CAP, performing ICVs, reviewing reported/detected occurrence data, reviewing BAR/CSBR data, or analyzing data available in the PFS Portal through the LSC.
(4) Document. Facilities must document CAPs within CEDAR to maintain a record of implemented corrections for mitigation monitoring and effectiveness determinations. CAPs resulting from the ICV and ECV processes are documented in the Compliance Verification Tool. LSC corrective actions, safety information, and problems should be documented in CEDAR and shared in ATC InfoHub.
(5) Data Integrity. Facilities must ensure that data collected through QC OSAs and Emphasis Items accurately reflect demonstrated technical performance. In addition, facilities must validate documentation associated with On-the-Job Training (OJT) and Certification Skill Checks / Performance Assessments to ensure that these processes accurately reflect facility performance. This ensures a solid foundation of data upon which CAPs are built.
2-2. Facility/District QC Orders.
a. Facilities and/or districts must not create QC orders that either duplicate or contradict the national order or collective bargaining agreement.
b. Facility/district QC orders (see Appendix D) may only contain the following elements:
(1) QC OSA sampling plan (see Appendix D).
v. › QC Validation information (include numbers of items reviewed and…
(2) Plan for conducting random/scheduled System Service Reviews (SSRs) (and Traffic Management Reviews (TMRs) for facilities with Traffic Management Units).
(3) Designation of points of contact for Systemic Issue Reviews (SYSIRs).
(4) QC OSA Validation sampling plan to be documented in CEDAR.
(5) Certification Skill Check and Performance Assessment Validation process to be documented in CEDAR.
(6) OJT Documentation Validation process to be documented in CEDAR.
(7) Requirements for recurring reports on performed QC processes, results of analyses of safety data, implemented corrective action plans, and data monitoring activities.
2-3. Local (Facility) Safety Reports.
a. Overview.
(1) Facilities are encouraged to create regular reports of collected data to support the identification of non-compliance and risk, provide visibility into facility performance, and ensure that the facility’s QC program is operating as intended. Reports can be generated monthly, quarterly, or at some other frequency depending on the available resources, size, and complexity of the facility; the amount of available data; and identified compliance issues. A sample of possible facility safety report data is contained in Appendix D, and an example facility QC Activity Plan is contained in Appendix E.
(2) Facilities are encouraged to include the data listed below in their recurring safety reports.
(a) Status: This section should list the status of required QC processes. The following should be considered:
i. QC OSAs conducted during the reporting period.
ii. Emphasis Items (include pre-existing Emphasis Items, new Emphasis Items created during the reporting period, and Emphasis Items closed during the reporting period).
iii. Service reviews conducted during the reporting period (include each different service review (SSR, Covered Event Review, TMR, SYSIR) and the reason for each).
iv. Status of current fiscal year ICV (include the percentage of items completed and remaining).
v. QC Validation information (include numbers of items reviewed and whether this is in accordance with the facility plan for each).
U.S. DEPARTMENT OF TRANSPORTATION › Some of this information can be obtained via automated report in the…
(b) Facility reporting data: This section is intended to provide information on the health of the proactive safety culture of the facility and assist in trend identification when used with QC data and analyzed through the service review process. The following data should be considered:
i. Trends found through review of MORs reported during a specific time period
ii. Trends found through the BAR process.
(c) Findings: This section should provide information on validated instances of systemic facility non-compliance and areas of risk, derived from available data sources. The following findings should be considered:
i. Findings from service reviews (the report should include all topic areas from the CEDAR question tree in which the service review team validated an issue and include a brief synopsis on selected (most serious) identified issues)
ii. Findings from the ICV (include any items rated non-compliant and their current status (open/closed))
iii. Findings from Quality Control Validations (QCVs) (include any compliance issues identified)
iv. Findings from CSBR
(d) CAPs. This section should include information on all facility CAPs regardless of the triggering process (for example, CAPs created from service reviews, CSBR, ICV/ECVs, and/or QCVs). The following types of CAPs should be considered.
i. New CAPs created during the reporting period (include the associated monitoring plans)
ii. CAPs closed during the reporting period (include data indicating how the CAP data target was met)
iii. Status of pre-existing CAPs not closed during the reporting period (include the status of the associated monitoring plan(s) and any preliminary findings, if available)
NOTE– Some of this information can be obtained via automated report in the ATO Manager Dashboard (some Service Delivery Points may be excluded). The report is accessed via the ATO Application Portal.
U.S. DEPARTMENT OF TRANSPORTATION › Quarterly targets for fully certified operational personnel and…
Chapter 3. Quality Control Monitoring
3-1. Introduction.
a. Quality Control (QC) monitoring is a key data-gathering component of an SDP QC program. Through QC monitoring, an SDP can identify areas of success and system risk by measuring compliance to requirements in FAA directives. QC monitoring collects technical performance data that measures facility performance. These data support other quality control processes that assess training, procedures, airspace, directives, equipment, and the technical performance of personnel.
b. QC processes are separate from individual performance management. The purpose of data collected through QC monitoring is to measure organizational performance.
3-2. QC Operational Skills Assessment (OSA). A QC OSA is a QC sampling method used to collect data by assessing the technical performance demonstrated by operational personnel. A QC OSA must not be used as the basis of, or to support, IPM.
a . When to Perform a QC OSA. Sufficient QC OSAs must be performed to provide a valid quality control sample of the various SDP positions and functions. Each SDP must establish a sampling plan that includes random and scheduled activities. QC OSA sampling plans must assess each type of operational/control position including watch supervision functions and traffic management functions.
NOTE– The facility sampling plan for QC OSAs must ensure that all operational positions are assessed, and is not based on individuals. There is no requirement to ensure that individuals have received a QC OSA. OSAs are only conducted on positions being worked by position certified individuals.
EXAMPLE– 1. An ATC-12 TRACON conducts a random QC OSA on the departure east position from 1620-1700UTC.
2. An ARTCC schedules a minimum of six (6) QC OSAs each week on specific positions, in a specific area, and for defined time periods.
(1) Facilities are required to conduct a number of QC OSAs that will ensure an accurate assessment of facility performance. The quarterly targets, listed in Figure 3-1 and Figure 3-2 below, provide a 95% confidence level that QC OSA data collected will meet this accuracy threshold. The Air Traffic Control System Command Center (ATCSCC) will use the same target OSA numbers as noted for the ATCSCC/Terminal Quarterly QC OSA Plan by Facility, Figure 3-1. Quarterly targets were developed by analyzing facility types and the associated numbers of fully certified operational personnel or operational areas. Applying this analysis to a statistical model provides the resultant QC OSA target value.
NOTE– Quarterly targets for fully certified operational personnel and operational areas, Figures 3-1 and 3-2, are determined at the beginning of each fiscal year. Once determined, that quarterly target is to be used for the remainder of the fiscal year.
(1) › Review.
Figure 3-1: ATCSCC/Terminal Quarterly QC OSA Plan by Facility
ATCSCC/Terminal Quarterly QC OSA Plan by Facility CPC Count Only OSA Target
0-50 Half of Staff
51-75 31
76-100 41
101-125 52
126-150 61
151-175 70
176-200 79
201-225 87
226-250 95
Figure 3-2: En Route Quarterly QC OSA Plan by Facility
En Route Quarterly QC OSA Plan by Facility
Number of Operational Areas Quarterly OSA Target (25 x # of
Areas)
8 200
7 175
6 150
5 125
4 100
ZAN, ZSU, ZUA Half of Staff
b. How to Perform a QC OSA. QC OSAs may be conducted via real-time remote monitoring, real-time direct monitoring, or via playback tools (playback monitoring). Facilities are encouraged to use available playback tools/monitoring as their primary method of conducting QC OSAs. QC OSAs contain the following steps:
(1) Review. Employees conducting the review segment of a QC OSA must identify and record all technical performance demonstrated during the session, whether exemplary, meeting standards (compliant), and/or deficient (needs improvement). Reviewers are also expected to be mindful of, and identify, potential systemic issues associated with training, efficiency, airspace,
(9) › Potential systemic non-compliance/risk identified in QC OSAs must be…
procedures, directives, and equipment, when applicable. Potential systemic issues are addressed through the systemic issue review process.
(2) Documentation. Results of QC OSAs must be documented in the OSA module in CEDAR. All identified items must be documented in the OSA. Any potential systemic issues are collected within the OSA entry. Select the applicable systemic issue option, and enter a brief summary. Items not observed during the OSA should be left blank. For observed items, select the following category:
(a) Exemplary: the QC OSA item was completed in accordance with FAA Orders and environmental conditions or other factors made it very challenging, or the employee went above and beyond what was required.
(b) Compliant: the QC OSA item was completed in accordance with FAA Orders.
(c) Needs Improvement: the QC OSA item was not completed in accordance with FAA Orders.
NOTE– Comments are not required to close out the OSA form, however, comments greatly assist in finalizing a full understanding of what happened during the skill assessment. If checking or not checking an item category creates possible ambiguity for an individual reviewing the OSA, use the comments section to assist in a full understanding of what was observed during the QC OSA related to a specific item.
(3) QC OSAs are de-identified samplings of individual technical performance and may not be used for or to trigger IPM actions.
(4) Each QC OSA must accurately capture the technical performance demonstrated during that session. Data captured during QC OSAs are the basis for many other QC processes; accurate documentation is essential to ensure proper identification of facility performance issues.
(5) QC OSAs may be conducted by both management and non-management staff since they will only be used for identifying facility systemic issues and not used for IPM.
(6) QC OSAs must only be used to identify facility/organizational systemic noncompliance and/or monitor the effectiveness of CAPs.
(7) All QC OSAs require a bookmark of the reviewed session (if the capability exists) to be attached to the OSA.
(8) Facilities must regularly review aggregate results from completed QC OSAs and identify potential facility systemic non-compliance/risk. The “OSA Report” in CEDAR displays aggregate information on QC OSA results, for a user selected date range.
(9) Potential systemic non-compliance/risk identified in QC OSAs must be analyzed through the service review or the Compliance Verification processes in order to validate and understand the potential issue.
Facilities should regularly review both custom sub-tasks and EIs in… › -
3-3. Emphasis Items (EIs). EIs are prioritized custom sub-tasks in OSAs, and are used to collect data on specific focus items for a period of time defined by the facility. These items may be identified to be observed, documented, and tracked as a quality control area of emphasis.
a. Who Identifies EIs. Facilities can choose to develop an EI, or may be directed to do so by their district, Director of Operations, or other service unit personnel.
b. How to Identify an EI. An EI can be developed for any issue beyond the standard subtasks in the CEDAR OSA forms. There are three primary purposes of EIs:
(1) Monitoring CAPs: Collecting data to assess the effectiveness of corrective actions implemented to mitigate a specific hazard.
(2) Collecting additional data to validate suspected trends or issues: Potential facility problems identified through other means (e.g., service reviews, Combined Safety Barrier Review, Compliance Verifications, Partnership for Safety Portal data, Local Safety Council identification) often require additional data to determine the validity and scope of the issue. Facilities are encouraged to use EIs as a method of collecting such data.
(3) Emphasizing special focus issues: Facilities, Districts, Service Areas, and/or Service Units may use Emphasis Items to place special focus on specific issues by requiring a topic to be assessed in every OSA conducted within the applicable location/organization. Collected data must be used to assess compliance with the requirement under review.
c. How to Create and Document EIs. EIs must be entered into CEDAR to ensure documentation and organizational visibility into collected data, and is done so in the following manner:
(1) Facilities can create an EI by first creating a custom sub-task in their OSA module. Once a custom sub-task is created, checking the “emphasis item” radio button for that custom sub-task will move it to the top of the OSA form and identify it as an EI.
(2) Each EI must have a defined start and stop time period associated with it. Facilities should select a time period that will ensure sufficient data are collected on the sub-task (a minimum of 60 days is recommended).
(3) Facilities should regularly review both custom sub-tasks and EIs in their local OSAs to revalidate the need for each, and make revisions as necessary. Large numbers of custom subtasks can overload OSA evaluators resulting in items being overlooked or not assessed. Conversely, small numbers of custom sub-tasks can result in data not being collected on items important to a facility. Routine review of these custom items ensures the list of sub-tasks remains fresh and relevant to the data needs of the facility, and allows OSA reviewers to concentrate on evaluating the most concise and relevant list of items.
a. Who May Request an SSR. Personnel performing watch supervision… › a. Who May Request an SSR. Personnel performing watch supervision…
Chapter 4. Service Reviews
4-1. Introduction.
a. Service reviews are tools to be used for the on-going evaluation and improvement of SDP services and to validate suspected systemic issues or best practices. Service reviews should normally be conducted outside of the operational quarters.
b. There are four different types of service reviews: System Service Reviews (SSR), Traffic Management Reviews (TMR), Covered Event Reviews (CER), and Systemic Issue Reviews (SYSIR). Service reviews provide a methodical manner to take a broad look at all facets that affect the delivery of air traffic services. While some service reviews may be conducted postevent (SSR), some are required on a regular basis (TMR) and some are required after accidents (CER). It is expected that most other reviews will be originated absent a specific event (SSR and SYSIR). Service reviews not associated with an event should be originated on a regular schedule or conducted randomly to ensure that an SDP does not wait for an event to actually occur.
c. When a service review is originated and an associated Air Traffic Safety Action Program (ATSAP) report has been accepted, SDP personnel must follow the requirements of ATSAP. The ATSAP report must be considered to be an accepted report unless advised by the Event Review Committee (ERC) that the report is excluded. If Quality Control (QC) Operational Skills Assessments (OSAs) are performed in association with a service review, and an involved specialist(s) indicates that he/she has filed an ATSAP report, properly annotate the OSA (check the “ATSAP” box).
d. Service reviews must be conducted in a proactive manner and are intended to identify areas for facility improvement, including systemic risk, as well as exemplary performance and best practices, all striving to continually improve air traffic services. Multiple points of view are an asset to the service review process. The results of service reviews should be determined by concurrence of the personnel involved in each review.
4-2. SSR. The intent of an SSR is to review the air traffic services provided in any situation at any time under any circumstances in an in depth, comprehensive, and collaborative manner. The absence of a defined list of instances when an SSR must be conducted allows a SDP the latitude to use judgment and discretion in the determination of when to perform an SSR. SSRs may be specifically focused and limited in scope or may eventually evolve into a larger more indepth review.
NOTE– The filing or acceptance of an ATSAP report does not preclude an SDP from conducting an SSR. SSRs shall not be used solely to focus on individuals, their performance, or as a tool to conduct Individual Performance Management.
a. Who May Request an SSR. Personnel performing watch supervision (Operations Supervisor (OS), Operations Manager (OM), Controller-in-Charge (CIC), etc.) may request an SSR whenever they identify circumstances in the operating quarters that they believe warrant such a review. Personnel performing watch supervision must give first priority to determining what immediate actions (if any) need to be taken to ensure the continued quality of SDP services
There is no recommended target number of SSRs to conduct, however,… › There is no recommended target number of SSRs to conduct, however,…
for circumstances that trigger a request for an SSR. SDPs may be required to conduct an SSR, as noted in paragraph 4-2d.
b. Review of SSR Requests. Personnel designated by the SDP manager must review each request for an SSR. This review must include appropriate documentation associated with each request to determine whether an SSR should be conducted. If an SSR is not conducted, document the findings in CEDAR and close out the request. If an SSR is conducted, follow the procedures for conducting an SSR in this section.
c. Who May Conduct an SSR. SDP staff managers or other personnel as designated by the SDP staff manager(s). SSRs would not normally be conducted at the ATCSCC, rather, the TMR process is primarily utilized.
d. When to Conduct an SSR. SDPs must continually review services provided and initiate SSRs on a regular basis. SSRs may be random, scheduled, due to public inquiries, or as a follow up to known operational activities (post-event). In addition, SDPs may be required to conduct a SSR by service unit management (ATM, General Manager, Director of Operations, Director of Safety and Operations Support, Vice President, and/or their designees) by the QCG on behalf of the Director of Operations, or CSBR.
(1) In addition to random and scheduled SSRs, SSRs may be triggered for a variety of reasons. Primary triggers for conducting an SSR are:
(a) To validate a suspected systemic issue identified through a review of collected QC data.
(b) In response to potential systemic compliance issues associated with a single reported/detected safety occurrence.
(c) To validate a suspected systemic issue identified through the CSBR process.
(2) SDPs are encouraged to develop local plans, contained in the facility QC order, for conducting random or scheduled SSRs. When conducting random or scheduled SSRs, SDPs should focus on specific positions or operational areas (e.g., the arrival positions in a Terminal Radar Approach Control (TRACON), ground control in an Airport Traffic Control Tower (ATCT), an operational area in an Air Route Traffic Control Center (ARTCC), arrival sectors to a common approach control in an ARTCC) when determining when and where to focus the SSR.
(3) SSRs must be conducted post-event for any non-fatal accident or occurrence colorcoded yellow or red, in reference to the FAA Order JO 1030.3, Initial Event Response, and the NATCA/FAA Collective Bargaining Agreement.
NOTE– There is no recommended target number of SSRs to conduct, however, facilities are encouraged to conduct SSRs as often as possible, resources permitting. A deficiency in quality or quantity of SSRs can be an indicator of an unhealthy QC program. Conducting comprehensive SSRs increases the opportunity to identify systemic facility issues and best practices. In the absence of unlimited personnel resources, conducting fewer but more thorough and comprehensive service reviews is preferred to conducting
(d) › Procedures
multiple service reviews that result in high counts of service review numbers, with no resultant identification of systemic issues or best practices.
EXAMPLE–
1. An ATC-11 TRACON conducts a random SSR to review services on the arrival east position from 1625-1700UTC.
2. An ARTCC conducts a minimum of three (3) SSRs each week. These SSRs are conducted on specific positions for defined time periods and are rotated through areas of operation on a regular basis.
3. An ATC-7 Tower/TRACON facility receives a call from a pilot questioning services provided five (5) days earlier. The manager/staff manager requests that a SSR be conducted.
4. An operations manager at an ATC-11 Tower facility requests an SSR as a result of a runway incursion involving an air carrier and a vehicle on an active runway. The SDP reviews the request and conducts an SSR.
5. A CSBR team is convened, at the request of the Service Area QA office, based on past SDP BAR data indicating a potential issue with Instrument Flight Rules and Visual Flight Rules conflicts. As a result, the team requests a SSR be completed to review any issues in need of mitigation. The SSR is conducted and a need for training is identified. As a result, a CAP is developed and implemented.
e. How to Conduct an SSR.
(1) Decision point. Based on available information and local SDP or district policy, determine if an SSR should be conducted. Determinations of when to conduct a SSR must include the criteria described in paragraph 4-2d. SSRs directed by service unit management, or that are mandated via other process, are not subject to this decision point and must be conducted.
(2) Data review. Determine the time period and position(s) to be reviewed based on known circumstances. Review as much data as needed (QC OSAs, BAR output, etc.) to develop an understanding of what occurred during the timeframe in question or to assess/analyze the potential systemic issue being considered. Use available tools (Traffic Flow Management System (TFMS), replay, voice, discussions, etc.) as appropriate to support this process. When conducting SSRs based on requests by personnel performing watch supervision, review all documentation associated with the request. This must include immediate issues identified and/or addressed, what (if any) data were reviewed, and any other issues that may have been identified. When conducting any SSR, consider the following areas as necessary:
(a) Training: Classroom, OJT hours, Mandatory Briefing Items (MBI), air traffic simulation, etc.
(b) Efficiency: Traffic Management Initiatives (TMI), delays, traffic volume/complexity, Airport Arrival Rate (AAR)/Airport Departure Rate (ADR), runway configuration, etc.
(c) Airspace/Airport: Special Use Airspace (SUA), noise abatement procedures, etc.
(d) Procedures: Letters of Agreement (LOAs), waivers, Standard Terminal Arrival Routes (STAR)/Standard Instrument Departure (SID), deicing, Line-Up and Wait (LUAW), etc.
(e) Directives: Standard Operating Practices/Procedures (SOP), FAA Order JO 7110.65, Code of Federal Regulations (CFR) procedures, Notices to Airmen (NOTAM), etc.
f. › Employee Interviews.
(f) Technical operations: Equipment configuration, performance, outages, etc.
(g) Customer feedback: ATCSCC logs, pilot/airline comments/inquiries, airport operator comments/inquiries, etc.
(h) Resource management: Operational position combinations/configurations, personnel assignments, available staffing, position relief management, leave approvals, etc.
(i) Individual Performance: Actions of involved personnel (specialist, manager, etc.).
(3) Issue identification. After concluding the data review, isolate all items of recognized risk in the above identified areas and/or determine if a potential issue is valid or not.
(a) Once issues are identified, or an item is validated, work with operational staff, facility management, etc., to develop and enact a corrective action plan to address the issue(s) through the corrective action plan process.
(b) Additional issue identification. Each section in the CEDAR question tree includes the option for the identification of a potential systemic issue. If during the course of service review systemic issues are identified (either related to the original service review purpose or not), those issues may be validated within the current SSR being conducted or they may be identified as potentially systemic, and handled via the SYSIR process (select the applicable systemic issue option and enter a brief synopsis).
(4) Documentation. All identified/validated issues and actions taken must be documented through the use of the SSR question tree in CEDAR for analysis and SSR closure. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree, including the possibility of any systemic issues. If CEDAR is unavailable, retain the results, and enter them in CEDAR as soon as it becomes available.
(5) SSR closure. Once all issues have been identified, or forwarded via the SYSIR process, and all documentation is completed, close the SSR in CEDAR by selecting the “Close SSR” option. Issues identified through a SSR must result in either a CAP that addresses the specific issue, or be identified as a potential SYSIR.
(a) SSRs that were generated as the result of a services rendered teleconference must be reviewed by the respective QCG prior to the SDP closing the SSR. This review is intended to be an assessment of the quality of the SSR conducted by the SDP.
(b) Resultant SSR CAPs must be documented in CEDAR, and are generated via the “Create CAP” function within the SSR. CAPs must be developed, and mitigations initiated, within 30 calendar days of SSR closure.
f. Employee Interviews. Interviews with employees are an essential element used to gain an understanding about what occurred during the period under review. Personnel conducting
There is no recommended target number for TMRs to conduct. › There is no recommended target number for TMRs to conduct.
SSRs are encouraged to obtain multiple perspectives that may indicate what occurred and should consider all employees regardless of status, e.g., developmental, as potential interviewees. Employee participation is mandatory; however, interviews with any employee must be conducted in an atmosphere of shared concern that is designed to gain a better understanding about the operational environment. Any information obtained during such interviews should be used within the SSR process to aid in identifying risk that may require corrective action.
g. Other SDPs. When elements of the SSR indicate another SDP may benefit from reviewing the information, the SSR must be forwarded to the other SDP staff for review, comment, and/or action, as appropriate.
4-3. TMR. TMRs are in-depth, comprehensive, and collaborative facility operational reviews. The intent of a TMR is to perform an evaluation of SDP traffic management operations with a focus on identifying issues that may impact system efficiency. This should include items that need correcting as well as identification and recognition of best practices. The TMR is the primary service review process for the ATCSCC.
a. When to Initiate a TMR. TMRs may be random, scheduled, or specific (post-event). SDPs must ensure that services provided by Traffic Management Units (TMU)/ATCSCC are continually reviewed and initiate TMRs on a regular basis. TMRs may be initiated at any time, due to random or scheduled reviews, because of significant delay events (due to weather, equipment outages, etc.), associated with a special event (sporting event, fly-in, etc.), or at the request of management.
(1) In addition to random and scheduled TMRs, the primary triggers for conducting a TMR are:
(a) Significant delay events (due to weather, equipment outages, TMIs, etc.)
(b) Special event activities (e.g., sporting events, fly-ins, Temporary Flight Restrictions)
(c) A request from operational management.
(d) CSBR
(2) When conducting random or scheduled TMRs, facilities should concentrate on specific traffic management positions or functions when determining when and where to focus the TMR.
(3) TMRs must be conducted when requested by Air Traffic Services or System Operations organizational leadership.
NOTE– There is no recommended target number for TMRs to conduct. However, applicable facilities are encouraged to conduct TMRs often, resources permitting. Conducting comprehensive TMRs increases the opportunities to identify potential systemic issues. A deficiency in quality or quantity of TMRs can be an indicator of an unhealthy QC program. In the absence of unlimited personnel resources, conducting fewer
(b) › These metrics should be evaluated against those values that measure…
but more thorough and comprehensive TMRs is preferred to conducting multiple TMRs that results in high count totals, with no resultant identification of systemic issues or best practices.
b. Who May Initiate a TMR.
(1) SDP TMR. Traffic Management Officer (TMO), SDP/General Managers, National Operations Manager (NOM), National Traffic Management Officer (NTMO), Supervisory Traffic Management Coordinator (STMC), Traffic Management Coordinator-in-Charge, OS, OM, CIC, etc., quality control managers/staff, or other personnel as designated by the SDP Manager. In addition, personnel within the SDP’s service unit (General Manager, Director of Operations, Vice President, and/or their designees), may require a SDP to initiate a TMR. Finally, the Deputy Director of System Operations (DDSO) and/or the QCG may also require an SDP to initiate a TMR on behalf of the Director of Operations at the service area.
(2) ATCSCC. Any request made by an SDP to the ATCSCC to initiate a TMR must be routed through either the OM, TMO, SDP Manager (or designee), General Manager, or Assistant General Manager for awareness and approval.
(3) CSBR panel.
c. Who Must Perform a TMR. TMRs should only be conducted at SDPs with a TMU or at the ATCSCC. TMRs must be performed by SDP-designated staff.
NOTE– Due to the national perspective of the ATCSCC, the ATCSCC may request individual SDP TMRs to be completed in support of a particular TMR being conducted by the ATCSCC.
d. How to Perform a TMR.
(1) Data review. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). Personnel conducting a TMR must review traffic management performance metrics for the date(s) under review.
(a) This review must include, at a minimum, the following metrics for that SDP (if applicable):
i. TMIs, to include miles-in-trail restrictions, ground delay program/airspace flow program/ground stop.
ii. Operations Network (OPSNET) delays.
iii. Terminal area efficiency rating.
(b) These metrics should be evaluated against those values that measure system capacity and availability in order to provide an overall picture of available capacity as measured against actual performance. When reviewing capacity consider, as a minimum, the following values:
U.S. DEPARTMENT OF TRANSPORTATION › 4-4. CER.
i. National Traffic Management Log runway accuracy.
ii. Average daily capacity.
iii. Monitor alert parameters.
iv. AAR/ADR.
v. Runway configuration.
(c) In addition, reviewers should consider traffic management data from adjacent facilities when such data would be pertinent to the review being conducted.
(d) When appropriate, reviewers may interview operational personnel (National Traffic Management Specialist, NTMO, NOM, Traffic Management Coordinator, STMC, OS, Air Traffic Control Specialist, CIC, OM, etc.) to obtain an operational perspective regarding certain aspects of the date(s) under review. In addition, reviewers should consider contacting adjacent SDPs (ARTCC, TRACON, ATCT, ATCSCC, etc.) and/or other sources (airspace operators, National Weather Service, etc.) to gain a perspective on what was or was not effective and what impact that date’s operations may have had on those SDPs.
(2) Issue identification. When comparing metrics to available capacity, reviewers should assess whether those initiatives that were implemented were thorough, performed as intended, and implemented following FAA orders. In addition, reviewers must assess whether those triggers used to initiate any TMIs were appropriate, coordinated, and communicated to the appropriate end user. Reviewers must then identify performance gaps, best practices, and/or successes primarily at the SDP level. However, such items may be identified at other organizational levels as well. Once the issue(s) is identified, SDPs must work with operational staff, facility management, etc., to develop and enact a corrective action strategy to address the issue(s) in a timely fashion. For those issues identified at higher organizational levels, service units (at the appropriate organizational level) must develop strategies to overcome those gaps.
e. Documentation. TMRs must be documented in CEDAR. Any issues confirmed through a TMR must result in a CAP that addresses the specific issue and is documented in CEDAR via the “Create CAP” function. TMR CAPs must be developed, and mitigations initiated, within 30 calendar days of TMR closure. When elements of the TMR involve another facility, the TMR must be forwarded to the appropriate SDP for review, comment, and/or action, as appropriate. If issues are identified above the SDP level, forward that information via CEDAR and/or email to the appropriate DDSO for follow-up.
4-4. CER. The intent of a CER is to supplement and document the existing required review of air traffic services rendered during an aircraft accident. CERs are an in-depth, comprehensive, and collaborative facility operational review. A CER must include a review of all aspects of service (individual performance, equipment issues, weather, etc.) and identify any issues that cannot be ruled out as potentially contributing to the accident. Instances of identified exemplary performance should be identified and documented as well. A CER may be used as a tool to identify systemic risk as well as a method to improve air traffic services following an aircraft accident. The filing of an ATSAP report does not preclude a SDP from conducting a CER.
d. › Documentation and Closure.
a. When to Initiate a CER. The Director/Deputy Director of Operations, Senior Advisor, or the QCG on behalf of the Director of Operations in each respective service area determines when facilities must conduct a CER. CERs are a supplement to other post-accident activities and occur after required notifications and services rendered teleconferences.
(1) A CER must never be initiated until all search and rescue activities, required notifications, and services rendered teleconferences have been completed.
(2) CERs must be conducted after any aircraft accident involving fatalities in which air traffic services were provided.
(3) A CER must be completed within five administrative days of the fatal accident under review.
b. Who Must Initiate a CER. SDP management, or their designated staff, must initiate a CER when directed by the Director of Operations, Senior Advisor, the QCG on behalf of the director(s) of operations, or when mandatory.
c. How to Conduct a CER.
(1) Use available tools (TFMS, replay, voice, discussions, etc.) to develop an understanding of what occurred during the timeframe in question. Include any previous findings from any review of services data compiled as required by FAA Order JO 1030.3, Initial Event Response. When completing the CER, assess each of the following:
(a) Training: Classroom, OJT hours, MBIs, air traffic simulation, etc.
(b) Efficiency: TMIs, delays, traffic volume/complexity, AAR/ADR, runway configuration, etc.
(c) Airspace/Airport: Operational position combinations, SUA, noise abatement procedures, etc.
(d) Procedures: LOAs, STARs/SIDs, deicing, LUAW, etc.
(e) Directives: SOPs, FAA Order JO 7110.65, CFR procedures, NOTAMs, etc.
(f) Technical operations: Equipment configuration, performance, outages, etc.
(g) Resource management: Operational position combinations/configurations, personnel assignments, available staffing, position relief management, leave approvals, etc.
(h) Individual Performance: Actions of involved personnel (specialist, manager, etc.).
d. Documentation and Closure. SDPs must document the findings of a review of individual performance as a QC OSA for all involved employees and check the “ATSAP filed” box, if applicable. In addition, record any systemic findings in CEDAR in the CER module under the applicable category in the CER question tree.
U.S. DEPARTMENT OF TRANSPORTATION › a. Each SDP must identify designees for the systemic areas of training,…
(1) Employee(s) last providing ATC services to an aircraft involved in a fatal accident must be relieved from the operational position as soon as feasible and must remain relieved from operational duties until the employees’ performance can be reviewed and associated training, if assigned, is completed. When an ATSAP report has been submitted, facilities must only administer ERC-assigned training. Documentation of ERC-assigned training must comply with existing requirements.
NOTE– If involved employees do not submit an ATSAP report and the employee’s supervisor believes training is warranted, the supervisor will assign appropriate training and ensure completion and documentation in accordance with FAA Order JO 3120.4, Air Traffic Technical Training. If involved employee(s) submit an ATSAP report and the employee’s supervisor believes training is warranted, the facility must submit its training recommendation to the ATSAP ERC for consideration; joint, collaborative management/NATCA submissions are always preferred, however separate recommendations will be accepted by the ERC. ERC review of facility training recommendations as a result of a CER is required in accordance with FAA Order JO 7200.20, Voluntary Safety Reporting Program (VSRP).
(2) The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). Each section in the CER question tree includes the option for the identification of a potential systemic issue. Any potential systemic issue found must be indicated as such in the CER and handled via the SYSIR process, or if already validated within the applicable CER, mitigated through the CAP process.
(3) CERs must be documented in CEDAR. Any facility non-compliance or systemic issues validated through a CER must result in a corrective action plan that addresses the identified issue, and the CAP must developed, and mitigations initiated, within 30 calendar days of CER closure. A CAP is generated through the “Create CAP” function within the CER.
(4) A CER generated due to a fatal accident must be reviewed by the respective QCG prior to the SDP closing the CER. This review is not intended to be a part of the SDP review of the applicable employee’s performance, but is a review of the quality of the overall CER conducted by the SDP.
4-5. SYSIR. SYSIRs are a comprehensive and collaborative review of potential systemic issues identified during OSAs, SSRs, CSBR, and CERs. SYSIRs are a method for SDP personnel that conduct an OSA, SSR, CSBR, or CER to identify areas that have potential system impact, whether at the SDP, service area, or service unit level. SYSIRs can also be generated directly through the Air Traffic SYSIR function in CEDAR if a suspected systemic issue is found outside of the OSA, SSR, CSBR, or CER process. Individuals who identify potential systemic issues initiate the process of identifying previously unrecognized risk. When conducting an OSA, SSR, or CER, SDP personnel must continually look for underlying issues beyond the initial “what” that may have occurred and look for the “why.” Asking “why” will help lead personnel to the initial identification of a systemic issue.
a. Each SDP must identify designees for the systemic areas of training, efficiency, airspace/airport, procedures, directives, and technical operations. The designees may be permanent, or rotated, at the discretion of the SDP. SDPs with limited or no support staff will be supported by their district in addressing the above systemic areas.
(3) › The SYSIR collaborative team must examine the potential systemic issue…
b. SDPs must conduct SYSIRs in the following manner:
(1) Assess all available information of each OSA, SSR, CSBR, or CER conducted, looking for possible underlying issues beyond those that are operationally immediate. If such an issue is identified, select in CEDAR one of the following options for further review:
(a) Training.
(b) Efficiency.
(c) Airspace/airport.
(d) Procedures.
(e) Directives.
(f) Resource Management.
(2) In the SYSIR comments section of the OSA, SSR, or CER, describe the item that is a potential systemic issue. Once the OSA, SSR, or CER is saved, CEDAR will forward the SYSIR to the appropriate designees.
EXAMPLE– 1. Any individual conducting an OSA can identify a potential systemic issue. When conducting a QC OSA, the reviewer identifies the incorrect application of vectors to intercept the final approach course. The reviewer believes this may be more widespread beyond one QC OSA and that a possible systemic training issue may exist in the facility. The reviewer enters a potential systemic training issue in the OSA and describes the issue. The SYSIR is then forwarded to the facility designee for all Training SYSIRs for review.
2. A SSR or CER collaborative team may determine that the scope of an issue warrants review under the SYSIR process. When conducting a SSR, the collaborative team determines via interviews that the application of tower visual separation is not being properly applied by facility personnel due to knowledge gaps and poor understanding of the language in FAA Order JO 7110.65, Air Traffic Control. As a result, the team believes a potential systemic issue of directives and training exists in the facility. The team enters both a potential systemic directives and training issue in the SSR and respectively describes both issues. Each SYSIR is then forwarded to the facility designee for Directives and Training SYSIRs for review. The two SYSIRs will be linked through the original SSR to ensure they are reviewed in tandem.
3. A collaborative CSBR team has convened, at the request of the facility Local Safety Council, to review suspected systemic risk associated with helicopter operations in close proximity to the departure and arrival corridors. As a result of the review, and the analysis of facility QC data and Barrier Analysis Review (BAR) data, the team determines there is risk that is in need of mitigation. A SYSIR is submitted under the Training SYSIR category for review and processing by the facility designee.
(3) The SYSIR collaborative team must examine the potential systemic issue to the extent necessary to determine its validity. It is the responsibility of the team to ensure each issue is objectively reviewed and that no issue is pre-judged prior to the team’s analysis. Systemic issue validation must employ a variety of techniques. When validating potential systemic issues, consider utilizing some or all of the following methods:
(b) › Concur and Elevated
(a) Review of collected data (e.g., QC OSA data, MOR data, BAR output, compliance verification data, and Emphasis Item data).
(b) Interviews of operational and/or staff personnel.
(c) An examination of training materials and documentation (e.g., OJT, recurrent training, refresher training, and MBIs).
(d) Review of operational replays and/or voice recordings.
(e) Review of local/national directives.
(f) Review of operational reference material (approach plates, charts, etc.).
(g) Review of sector/airspace/runway configuration/usage.
(h) Review of equipment installation, operation, configuration, and availability.
(i) Review of traffic management initiatives, procedures, and compliance.
(j) Review of Partnership for Safety Portal data.
(k) Collection of additional data through Emphasis Items or the facility ICV.
NOTE– This list is not all-inclusive. There are other items that may need examination. Teams should ensure that all pertinent avenues are explored prior to closing the SYSIR.
(4) Once the examination is complete, the designee must close the item in CEDAR with one of the following three responses:
(a) Concur: This response is appropriate if the SYSIR identifies/validates a previously unrecognized risk or non-compliance. The designee must concisely document the systemic issue identified in the SYSIR comments and close the SYSIR. The SDP must then employ corrective action. A CAP must be created, and mitigations initiated, via the “Create CAP” function within the SYSIR within 30 calendar days of SYSIR closure. The corrective action does not have to be completed to close the SYSIR.
(b) Concur and Elevated: This response is appropriate if the SYSIR collaborative team identifies an issue, but the means to mitigate the issue is above the SDP level. The designee must concisely document the reason(s) the SDP cannot correct the identified issue in the SYSIR comments, select “Concur and Elevated,” and close the SYSIR in CEDAR. Before submitting an issue as “Concur and Elevated,” the SDP must consult with applicable district staff. “Concur and Elevated” SYSIRs are forwarded automatically through CEDAR. They will be routed as follows:
U.S. DEPARTMENT OF TRANSPORTATION › (c) Does Not Concur
i. To the QCG to be worked collaboratively with the NATCA Service Area Safety Representative. If the QCG is able to coordinate mitigations for the item, they must document actions taken in the elevated SYSIR prior to closure. If the QCG non-concurs, they must document the reasoning for making this determination, inform the SDP, and then close the item. If the QCG is unable to facilitate a solution but agrees there is an issue, they must concisely document the reason(s) they cannot mitigate the identified issue in the SYSIR comments, select “concur and elevated,” and close the SYSIR in CEDAR.
ii. The “concur and elevated” SYSIR will then be routed to the AJI Safety Services Group, AJI-15, for resolution. AJI-15 will work the item to closure. AJI-15 will either concur, and the item will be mitigated at the national level, or non-concur, and the item will be closed after documenting why this determination was made. The appropriate service center QCG must be notified by AJI-15 of the non-concur determination.
NOTE– AJI-15 will contact applicable service unit(s) with the identified issue(s) and preliminary data to support the topic. AJI-15 will work with the applicable service unit(s) to identify any additional supporting data and develop potential mitigation(s).
NOTE– If CEDAR is temporarily unavailable, retain the results and enter them in the SYSIR as soon as it becomes available.
(c) Does Not Concur: This response is appropriate if the collaborative team has examined the issue identified in the SYSIR but cannot identify any unrecognized risk or non-compliance. The designee must document the steps taken to examine the issue and the reason(s) for selecting “Does Not Concur,” and close the SYSIR in CEDAR.
c. ICVs must evaluate all items on the Compliance Verification Tool… › c. ICVs must evaluate all items on the Compliance Verification Tool…
Chapter 5. Compliance Verification
5-1. Compliance Verifications (CVs). CVs are a way of assessing SDP performance and identifying areas for improvement. CVs must be conducted in accordance with Article 51 of the FAA/NATCA Collective Bargaining Agreement, which defines union participation. ICVs are planned assessments accomplished through the use of a checklist and random sampling methods such as, but not limited to, direct operational observation, discussions with SDP personnel, review of voice or radar/surveillance data, equipment parameters, certification parameters, and examination of other documentation. ECVs are assessments of SDPs that are conducted on an as-needed basis as determined by the service unit, Director of Operations, and/or the QCG. Determinations to conduct ECVs will be based on data analysis that identifies potential risk within specific SDPs. ECVs are not intended to be conducted at regular time intervals or on a regularly scheduled basis. ECVs may be conducted through various methods that may include developing a custom checklist, review of available data (CEDAR, quality control (QC) and quality assurance data, barrier analysis data, etc.), direct observation, interviews with personnel, and other means as appropriate. There are unique requirements for compliance verification for Alaska Flight Service Station (FSS) and U.S. NOTAM Office (USNOF), these are covered in Appendix A.
a. Facilities should use compliance verifications in three (3) ways:
(1) CAP monitoring – use the results of ICV and ECV checklist items as indicators of the effectiveness of mitigations.
(2) As an indicator of potential target areas for QC Operational Skills Assessments, nonevent driven service reviews (System Service Reviews (SSRs) and Traffic Management Reviews), and possible Emphasis Items.
(3) Identification of potential systemic issues – potential systemic non-compliance identified through the evaluation of checklist items must be analyzed through the SSR process or by the CV team conducting the CV in order to validate and understand the potential facility noncompliance. Facilities with established Local Safety Councils are encouraged to use their Local Safety Council to review, identify and mitigate facility systemic non-compliance and/or risk.
NOTE– FSSs, USNOF, and Federal Contract FSSs will follow the applicable procedures outlined in Appendix A and B to complete CVs.
5-2. ICV.
a. ICVs are expected to be conducted by an ICV team designated by the SDP Manager or General Manager following all applicable national collective bargaining agreements.
b. All FAA air traffic control SDPs and federal contract towers must conduct an ICV annually each fiscal year. The schedule for conducting an ICV will be determined by the facility in order to produce the most comprehensive evaluation results within the annual time period.
c. ICVs must evaluate all items on the Compliance Verification Tool (CVT) checklist
U.S. DEPARTMENT OF TRANSPORTATION › a. The ICV and ECV teams will assess items using the following…
except those not applicable to the specific facility type. Use of this standardized checklist requires facilities to assess requirements that may not normally be reviewed through other QC processes. Any Air Traffic Organization (ATO) entity may propose the addition or deletion of a checklist item(s). Requests for checklist modification must be forwarded to the appropriate service unit for evaluation and possible incorporation.
(1) Facilities should use a variety of methods to assess compliance with checklist items. This includes monitoring of operations (live and via playback tools), reviewing collected data in CEDAR, and interviewing facility personnel.
(2) It is imperative that facilities document all non-compliance identified through the ICV process. This ensures an accurate record of facility compliance and correction.
(3) Facilities may use SSRs to better understand an issue identified through the ICV process. SSR findings must be documented in CEDAR.
5-3. ECV.
a. The Directors of Operations, within each service area, must ensure that ECVs are conducted in response to data-driven indicators of potential systemic risk and/or noncompliance. ECVs may also be initiated by the Service Unit. ECVs will primarily be conducted by Service Center QCGs. A service unit or a Director of Operations, at the service area, may assign additional personnel to any ECV team. The QCG may request resources from the service unit/area to augment ECV staff.
b. The QCG must collaborate with the initiating organization (applicable service unit) to determine the scope of the ECV. ECVs may be broad in scope and utilize a customized checklist, or may focus narrowly on just a few items. When defining ECV scope, the service center QCG must consider trend analysis from the ATO Safety and Technical Training Quality Assurance Group. An ECV may be conducted as a site visit or a desk audit.
c. The Air Traffic Control Systems Command Center ECV team will be comprised of representatives from any of the service center QCGs.
d. At any time during an ECV, the team may observe operational items that represent a significant safety risk. After advising the SDP manager, the ECV team should then focus on those items whether or not they were defined within the original scope of the ECV.
e. The ECV team lead will ensure the visit is coordinated as needed, and the team will conduct an in-briefing with the SDP manager. The briefing will include an introduction of team members and ECV expectations.
5-4. Findings.
a. The ICV and ECV teams will assess items using the following categories and enter applicable details into the CVT:
(1) › NL
(1) Exemplary (E): This finding is assigned to items that demonstrate exemplary performance in quality and efficiency.
(2) Compliant (C): This finding is assigned to items that are completed in compliance with national, service area, and local requirements. Details are not required to be entered into the CVT.
(3) Non-compliant Low Risk (NL): This finding is assigned to items that are noncompliant, but do not represent a moderate or significant safety risk to the National Airspace System (NAS).
(4) Non-compliant Medium Risk (NM): This finding is assigned to items that are noncompliant and represent a moderate safety risk to the NAS.
(5) Non-compliant High Risk (NH): This finding is assigned to items that are noncompliant and represent a significant safety risk to the NAS.
(6) Not Observed (NO): This item is assigned to items that are not observed during the verification. The reason the item was not observed must be documented.
(7) Informational (I): For ECV use only. This finding is assigned to any item or topic that is of interest or significance, or used for awareness of information.
b. Prior to leaving the SDP, or concluding the desk audit, the ECV team lead must brief the SDP manager on all items that will be rated as non-compliant.
c. QCGs will forward findings of the ECV pertaining to training to ATO Safety and Technical Training.
d. ECV reports must be completed and submitted into the CVT within 10 administrative work days. The report must list all items rated and include associated details for all items rated “E,” “NL,” “NM,” or “NH.”
5-5. Responding to Findings.
a. CAP. Items assessed as non-compliant require a corrective action plan as described in Chapter 7 of this order. CAPs must be documented in the CVT. The CAP communicates how specific risk/non-compliance will be resolved. The plan must specifically describe actions taken to effectively mitigate and monitor the identified issue(s).
b. The SDP must respond to items identified as introducing risk or non-compliant in the following manner:
(1) NL: Develop corrective action and enter the plan in the CVT within 30 calendar days of the finding. To close the item, document the processes used to ensure the effectiveness of the mitigation.
(2) NM: Develop corrective action and enter the plan in the CVT within 20 calendar days of the finding. To close the item, document the processes used to ensure the effectiveness of the mitigation.
U.S. DEPARTMENT OF TRANSPORTATION › c. No action is required for checklist items rated “E,” “C,” “I,” or…
(3) NH: Due to the severity of the finding and subsequent risk to the NAS, SDPs must, within seven calendar days of being notified of the rating by the ICV/ECV team, develop a corrective action plan and obtain associated Director of Operations concurrence at the applicable service center (through the QCG). To close the item, document the processes used to ensure the effectiveness of the mitigation and enter it into the Compliance Verification Tool.
c. No action is required for checklist items rated “E,” “C,” “I,” or “NO.”
(5) › Follow-up process
Chapter 6. Quality Control Validations
6-1. Quality Control Validations (QCVs). QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all QC processes to ensure the effective identification of non-compliance and associated corrective actions. QCVs are the primary method that facilities must use to ensure the integrity of data collected in QC processes. QCVs require facilities to review samplings of QC OSAs, Certification Skill Checks / Performance Assessments, and OJT documentation to validate accuracy and completeness within each process. The QCV process is a statistical sampling of a particular process or task to ensure compliance and accuracy. This process may also identify potential systemic issues associated with training, efficiency, airspace, procedures, directives, and equipment. If risk is determined to exist, facilities must determine the appropriate mitigation strategy and document a CAP in CEDAR within 30 calendar days.
a. SDPs must develop a local validation plan for each of the three required QCVs. Local validation plans must contain the following:
(1) Target number of validations to be performed: Targets may be defined by any calendar unit (e.g., monthly, quarterly, annually). Validations must be conducted in sufficient numbers (a representative sampling) to ensure an accurate assessment of facility performance in conducting each of the processes being validated. It is recommended that SDPs validate a minimum of 15% of all QC OSAs, Certification Skill Checks / Performance Assessments, and OJT Instructor (OJTI) documentation each quarter at each SDP.
(2) Sampling method: While each process should be sampled randomly, local validation plans should include selection methods that ensure a cross-section of sectors/positions are reviewed. For example, a local validation plan could include a requirement that at least one QC OSA for each operational position be validated each fiscal year.
(3) Replay tools to be used: SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer/OJTI. Validations must be conducted within the maximum data retention periods for the facility to ensure availability of required data.
(4) Feedback process: SDPs may provide feedback to individual non–bargaining unit personnel for QCVs when appropriate. Individual feedback must always come through the individual’s immediate supervisor. SDPs must only provide facility-wide feedback to bargaining unit employees for all QCVs.
(5) Follow-up process: SDPs must follow up on issues identified through validations to ensure that the feedback or corrective action was effective in improving the respective process (QC OSAs, Certification Skill Checks / Performance Assessments, and OJT documentation). Follow-up processes related to non-corrective action plans should include a review of the previously identified feedback issue, a defined period for follow-up review, and closure if the issue is resolved. If the issue still exists, additional feedback must be provided. For matters that
(2) › Follow-up mechanisms to ensure that the feedback was effective if skill…
resulted in a CAP, the monitoring and effectiveness goals will indicate the success of any implemented corrective action.
b. QCVs must be conducted in accordance with Article 51 of the FAA/NATCA Collective Bargaining Agreement, which defines union participation. Facilities are encouraged to establish collaborative teams to conduct QCVs.
c. QCVs must only be used to identify organizational or systemic issues.
d. OSA Validations. SDPs must establish a validation process whereby representative samplings of OSAs are evaluated to ensure accuracy and completeness. At a minimum, SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer. The purpose of this initiative is to ensure that the reviewer has captured the performance of the trainee accurately in the OSA. These reviews provide an opportunity to identify exemplary performance that can be shared in the facility as a best practice, as well as any performance deficiencies or risky behaviors that otherwise may have been overlooked. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement. The validation process must include:
(1) Feedback to the person conducting the OSA through their immediate supervisor, except as noted in paragraph 6-1a.(4); and
(2) Follow-up mechanisms to ensure that the feedback was effective if OSA improvement is required.
e. Certification Skill Check and Performance Assessment Validations. SDPs must establish a validation process whereby representative samplings of Certification Skill Checks and Performance Assessments are evaluated to ensure accuracy and completeness. At a minimum, SDPs must utilize both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer. The purpose of this initiative is to ensure that the OS/STMC/NTMO has accurately captured the performance of the developmental, Certified Professional Controller-in-Training (CPC-IT), or the Traffic Management Controller-in-Training (TMC-IT) / Traffic Management Specialist-in-Training and/or Developmental/OS/STMC/NTMO during a Certification Skill Check or Performance Assessment. These reviews provide an opportunity for an SDP to ensure consistency in their skill checks / assessments as well as identify performance deficiencies or risky behaviors that otherwise may have been overlooked. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement. The validation process must include:
(1) Feedback to the person conducting the skill check / assessment through their immediate supervisor and
(2) Follow-up mechanisms to ensure that the feedback was effective if skill check / assessment improvement is required.
f. On-the-Job Training Documentation (OJTD) Validations. SDPs must establish a validation process whereby they assess the accuracy and completeness of the documentation provided by OJT instructors. The purpose of the OJTD is to ensure that OJTIs accurately captured the performance of the CPC-IT/TMC-IT/NTMS-in-training/developmental/OS/STMC/ NTMO during OJT. These reviews provide an opportunity for an SDP to ensure consistency in the conduct and documentation of OJT and to identify performance deficiencies on the part of OJTIs. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement.
U.S. DEPARTMENT OF TRANSPORTATION › (c) Follow-up mechanisms to ensure that feedback and training were…
(1) When conducting OJTD validations, SDPs must review a representative sampling of the OJT documentation for the year under review using available replay tools.
(a) SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer.
(b) Tower-only facilities without surface radar are expected to use voice files to the maximum extent possible to assess any OJT sessions reviewed in this process.
(2) The validation process must include:
(a) Feedback to the SDP OJTIs;
(b) Refresher training, if appropriate, to SDP OJTIs. If assigned, document refresher training in the employee’s FAA Form 3120-1, Training and Proficiency Record, using CEDAR; and
(c) Follow-up mechanisms to ensure that feedback and training were effective.
(2) › c. The following steps are required to complete a CAP.
Chapter 7. Corrective Action Plans
7-1. Introduction. Finding and fixing problems is the fundamental purpose of our safety management system and the quality control process. CAPs are the method SDPs must use to document implemented corrections to validated systemic non-compliance or identified risk.
7-2. Develop and Implement CAPs. CAPs are actions taken by SDPs to correct non-compliance or risk, which has been properly identified, validated, and understood through data collection and analysis. Once a problem is understood (in scope and causal factors), SDPs must develop corrective actions to address the identified issue(s). CAPs must be designed to correct the specific problem, and then implemented throughout the applicable area. In addition, CAPs must include how the effectiveness of implemented mitigations will be measured. Facilities must monitor implemented CAPs as they continue to collect data. This can be accomplished through performing QC OSAs, designing specific EIs to measure a specific CAP, performing ICVs, reviewing reported/detected occurrence data, or analyzing data available in the PFS Portal through the LSC.
a. CAPs may be generated from any of the following.
(1) Service Reviews
(2) Compliance Verifications
(3) QC Validations
(4) Significant events/investigations (require a System Service Review (SSR) to validate the issue, possibly at a later date)
(5) LSC analysis (validation of issues through SSRs is encouraged)
(6) Runway Safety Action Teams (RSATs)
(7) Combined Safety Barrier Review
(8) For issues identified externally such as; AOV audits, NTSB investigations, and/or AJI audits.
b. CAP development teams should ensure that they gather input from relevant facility personnel to ensure all information is considered in creating a CAP. Facility personnel could include the following depending on the specific issue.
(1) Facility staff personnel (e.g., QC, airspace/procedures, training)
(2) Operational staff (e.g., Developmentals, Certified Professional Controllers, Supervisors, Operations Managers, Traffic Management Coordinator, STMC)
(3) Facility management (e.g., Support Managers, Facility Manager)
c. The following steps are required to complete a CAP.
An En Route facility has completed training of all radar-certified… › An En Route facility has completed training of all radar-certified…
(1) Describe the specific corrective actions that will mitigate the facility non-compliance/risk. Examples of corrective actions include, but are not limited to:
(a) Training (must target the specific knowledge gap)
(b) Changes to local procedures and/or processes
(c) Realignment of airspace
(d) Changes to letters of agreement with adjacent facilities, airport operators, etc.
(2) Identify the scope of the correction (e.g., facility-wide or certain operational areas within a facility).
(3) Identify a timeframe for completion of the action(s) taken.
(4) All CAPs must include a specific monitoring plan. Identify a monitoring plan for determining the effectiveness of the implemented corrective actions. Monitoring plans should include a frequency of data review, what data must be reviewed, and an assigned focal within the facility/organization for ensuring monitoring is conducted.
(5) Identify the target value for mitigation effectiveness.
EXAMPLE– A collaborative SSR team validates systemic facility non-compliance with the application of vectors to intercept the final approach course. This was based on QC OSA and Barrier Analysis Review (BAR) data. The team develops a CAP requiring training for all radar certified personnel on the issue. Once the training is completed within the required 60 days, the facility will include a new Emphasis Item in all QC OSAs on vectoring to the final approach course to be collected for 90 days from the completion of all training. The facility will consider the non-compliance mitigated if compliance with this Emphasis Item meets or exceeds 90%.
d. If the mitigation effectiveness target is not met, a revised CAP must be developed, documented, and enacted. This must include a new monitoring plan to assess effectiveness of the revised CAP.
7-3. Monitoring. Facilities must monitor all implemented CAPs in order to determine their effectiveness. Monitoring will result in CAP closure or a revised CAP if effectiveness targets are not met. CAP monitoring is primarily accomplished by assessing collected/reported data.
a. There are several possible methods for collecting data for a monitoring plan:
(1) Emphasis Items: Develop an Emphasis Item for all OSAs that collects data specific to the non-compliance/risk and associated mitigation.
EXAMPLE– An En Route facility has completed training of all radar-certified personnel on the use of speed control with terminal arrivals, as required by a CAP. The facility subsequently implements a new Emphasis Item for all OSAs. This Emphasis Item requires each OSA conducted for the next 90 days to specifically assess
U.S. DEPARTMENT OF TRANSPORTATION › EXAMPLE–
compliance with the issue. At the end of that time period the data will be assessed to determine the level of compliance with the requirement. Compliance will be used to determine CAP effectiveness.
(2) Log entries: Require entries on the Facility Operations Log (Form 7230-4) in CEDAR. Log entries must be for a specific type operation/occurrence and must include a specific keyword to support the word search function in CEDAR.
EXAMPLE– A terminal facility has revised and implemented local policies regarding combining/decombining arrival and final positions in the Terminal Radar Approach Control, as required by a CAP. The facility will require a log entry each time any of these positions are combined/decombined for the next 60 days to determine the level of compliance with the new requirement. Each log entry must include the keyword “FINCAP” to flag the entry for word searches. At the end of that time period the data will be assessed to determine the level of compliance with the requirement. Compliance will be used to determine CAP effectiveness.
(3) Compliance Verifications: Use the facility ICV process. There must be a checklist item directly related to the non-compliance and mitigation.
EXAMPLE– A terminal facility has completed training on its local QC OSA sampling plan in response to a CAP (QC OSAs were not being conducted in sufficient numbers to meet the facility defined quarterly target). The facility will assess compliance with the CAP during its upcoming ICV under the checklist item “Quality Control Monitoring.” The facility will consider the non-compliance mitigated if the target is met in each quarter remaining in the fiscal year.
(4) PFS Portal: Use data available through the PFS Portal to assess mitigation effectiveness. Facilities must have an LSC to use the PFS Portal.
EXAMPLE– A terminal facility has completed training of all operational personnel on vectoring and speed control techniques when sequencing arrivals for final approach, as required by a CAP. The CAP was developed in response to a high number of go-arounds due to unstable approaches. After 120 days, the facility’s LSC will review track data and Air Traffic Safety Action Program reports (accessed through the PFS Portal) to assess the effectiveness of the CAP. The LSC will specifically compare go-arounds during this time period to the time period used in the CAP.
(5) MOR data: Use a specific MOR that is directly related to the non-compliance and associated mitigation.
EXAMPLE– An En Route facility has completed training of all operational personnel on the handling of formation flights, as required by a CAP. After 180 days, the facility will review all “Suspected Loss Involving Formation Flight” MORs to assess compliance with the requirements of joining and separating formation flights. Compliance will be used to determine CAP effectiveness.
Continue reading ↓ 120 of 241 passages shown