Order JO 7010.14 Air Traffic Organization Audits and Assessments Program
This document is cancelled. Cancelled by JO 2900.2. Check the official source before relying on it.
Text read by OCR from a scanned file. Check the original, as some words may be misread.
U.S. DEPARTMENT OF TRANSPORTATION ORDER FEDERAL AVIATION ADMINISTRATION NUMBER Air Traffic Organization Policy JO 7010.14
Effective Date: 09/30/2013
SUBJ:Air Traffic Organization Audits and Assessments Program
1. Purpose of This Order. This order establishes requirements for the Air Traffic Organization (ATO) Safety and Technical Training (AJI) Audits and Assessments Program. Audits and assessments are an integral part of the ATO's Safety Management System and a component of safety assurance. This order defines the policy and responsibilities for conducting audits and assessments, including independent operational assessments, of the ATO operational services, designated solutions, safety programs and initiatives, and technical training. This order describes the procedures for conducting audits and assessments, reporting results, and ensuring corrective actions are implemented when National Airspace System (NAS) equipment, operations, and procedures do not perform as expected or required. AJI uses audits and assessments to identify instances of noncompliance with requirements, safety hazards, operational concerns, and other safety issues. AJI audits and assessments are conducted at the national, service area, district, and operating unit levels. The appendices of this order contain supplemental guidance and procedures; however, supplements may not subtract from the policy described herein.
2. Audience. This order applies to all ATO personnel and contract service providers.
3. Where Can I Find This Order? This order is available on the MyFAA employee website: https://employees.faa.gov/tools resources/orders notices/. This order can also be found on the Air Traffic Publications website: http://www.faa.gov/air_traffic/publications.
4. Policy. AJI conducts audits and assessments in accordance with Federal Aviation Administration Acquisition Management System (AMS) policy, JO 1000.37, Air Traffic Organization Safety Management System; JO 7210.633, Air Traffic Organization Quality Assurance Program (QAP), and the requirements contained in this order. This order does not cancel or supercede any current FAA or ATO orders. The provisions of this order do not override the provisions of agency collective bargaining agreements.
5. Audit and Assessment Responsibilities.
a. AJI Responsibilities. AJI must:
(1) Determine ATO operational service, NAS equipment, safety program or initiative, and technical training initiatives to audit and/or assess based on, but not limited to:
(a) Input from ATO management (internal AJI and ATO service units)
(b) Suspected risk/safey-related trends identified through analysis
(c) Designated solutions (e.g., NAS system acquisitions, modifications, etc.)
Distribution: Electronic Initiated By: AJl-32
09/30/13 JO 7010.14
(d) Safety documentation (e.g., safety risk management documents, corrective action plans, and other applicable sources)
(e) Previous audit and/or assessment results
(2) Establish an audit and/or assessment team to conduct each audit and/or assessment. The team size and composition is based on the scope of the audit and/or assessment.
(a) Assign a team lead to manage, oversee and lead each audit and/or assessment.
(b) May assign internal AJI personnel or request external ATO personnel to serve as team members. Team member selection must be in accordance with collective bargaining agreements, when applicable.
• Note 1: Additional personnel may serve as advisors or subject matter experts; however, they are not team members.
• Note 2: Air Traffic Safety Oversight Service (AOV) personnel may observe the conduct of an audit and/or assessment, when AOV requests to do so. However, AOV personnel are not team members.
(3) Prepare for an audit and/or assessment by reviewing relevant safety data (e.g., mandatory occurrence report, electronic occurrence reports, previous audit/assessment results, mitigation or corrective action plans, etc.), requirements, procedures or operations, and developing any documents (e.g., procedures, checklists, datasheets, questionnaires, surveys, etc.) needed to conduct the audit and/or assessment.
(4) Notify the affected service unit at least 90 days in advance of a scheduled audit and/or assessment, and at least 30 days in advance of an ad-hoc audit and/or assessment, unless otherwise coordinated.
(a) At a minimum, the notification must include the objective, scope, team information, and audit and/or assessment dates.
(b) AJI may conduct a no-notice audit and/or assessment, with the concurrence of the AJI Vice President and the affected Service Unit Vice President, where advance notification as described above will not be provided.
(5) Conduct an audit and/or assessment using various methods and techniques including, but not limited to, reviewing documents, records, and data (e.g., radar, voice recordings, etc.); conducting interviews with management personnel, or designee; and observing operations and procedures. AJI may conduct an audit and/or assessment on-site and may collect some or all data remotely. When an audit and/or assessment is conducted on-site, the team lead must:
(a) Conduct an opening meeting with the manager, or designee, to explain the objective and scope of the audit and/or assessment and any planned activities.
(b) Conduct a closing meeting with the manager, or designee, to discuss the audit and/or assessment.
09/30/13 JO 7010.14
(6) Document and report results.
(a) Prepare a cover memo and report to document the results of each audit and/or assessment. Distribute the cover memo and report to the affected service unit or decision maker within five business days of the AJI Vice President, or designee, approval.
(b) Notify the affected service unit when a response is requested to address an identified finding.
(1) Initiate a Corrective Action Request (CAR) when it becomes unlikely that a safety related finding will or should be resolved informally. The CAR process requirements can be found in JO 7210.633, and supplemental guidance is included in Appendix B of this order.
(2) Request a response, via the report cover memo, for any finding that will not result in a CAR, but require resolution. The procedures for initiating and responding to these findings are included in Appendix B of this order.
(3) A response to independent operational assessment findings are addressed as part of the in-service decision action plan. See Appendix C-2 for additional information.
b. ATO Service Unit Responsibilities. ATO service units must:
(1) Submit a request for an audit and/or assessment to AJI via official correspondence. At a minimum, the request must include a point of contact, the service, safety program or initiative, and/or procedure to be audited and/or assessed, background information, and a description of the safety concern(s).
(2) Designate personnel to participate in an AJI audit and/or assessment as team member(s), subject matter expert(s), point of contact, etc., when applicable. Participation of bargaining unit personnel must be in accordance with collective bargaining agreements.
(3) Ensure the affected manager, or designee, attends the opening and closing meetings. The manager, or designee, may invite service unit, service area, service center, district, or facility personnel, etc., to attend the opening and closing meetings.
(4) Respond to findings resulting from an AJI audit and/or assessment by the response due date when required. The affected service unit must:
(a) Submit a Corrective Action Plan (CAP) in response to any CARs initiated by AJI. CAR process requirements can be found in JO 7210.633. Supplemental guidance is included in Appendix B of this order.
(b) Submit a response to AJI regarding any findings that will not enter the CAR process, but require resolution. The procedures for initiating and responding to these findings are included in Appendix B of this order.
Joseph Teixeira Vice President, ATO Safety and Technical Training
09/30/13 JO 7010.14 Appendix A
Appendix A: Definitions
Appendix A: Definitions
a. Ad-Hoc Audit and/or Assessment. An audit and/or assessment directed by AJI management or requested by ATO service unit management. An ad hoc audit and/or assessment is conducted on an as-needed basis and is prioritized based on the potential risk to the NAS or the safety concern. An ad hoc audit and/or assessment may take priority over a scheduled audit and/or assessment.
b. Assessment. A process of measuring or judging the value or level of something.
c. Audit. A review of an organization's safety programs or initiatives to verify completion of tasks and determine an organization's compliance with FAA directives and procedures.
d. Corrective Action. The initial action or series of actions taken to correct a discrepancy.
e. Corrective Action Plan (CAP). Documentation prepared in response to a corrective action request that describes the action(s) to be taken to correct a safety-related finding.
f. Corrective Action Request (CAR). A formal document identifying a nonconformance that is systemic in nature and requires a root cause analysis and modification.
g. Finding: Results of the analysis of collected data against audit and/or assessment criteria.
h. Scheduled Audit and/or Assessment. An audit and/or assessment that's normally conducted on a recurring basis and is scheduled and planned in advance of the fiscal year.
i. Team Lead. An AJI employee assigned to manage, oversee, and lead an audit and/or assessment.
j. Team Member. Personnel (internal AJI or external) assigned to conduct an AJI audit and/or assessment.
k. Non-safety Related Finding. A concern or noncompliance that does not indicate a potential safety risk to the NAS.
1. Safety Related Finding. A safety issue or noncompliance with a safety related requirement that indicates a significant hazard or potential safety risk to the NAS.
09/30/13 JO 7010.14 Appendix B
Appendix B: Resolution of Audit and/or Assessment Findings
Appendix B: Resolution of Audit and/or Assessment Findings
1. Corrective Action Requests. AJI will initiate a CAR to begin the top-down (from the AJI Vice President to the affected Service Unit Vice President), formal process of informing ATO service units of an identified safety related finding and requesting action be taken to correct the issue. The CAR process requirements can be found in JO 7210.633, and supplemental guidance is provided below. The audit and/or assessment team lead may recommend the initiation of a CAR when a safety related finding requires immediate action to correct, requires root cause analysis to eliminate the hazard or correct the noncompliance to prevent recurrence, etc.
a. Initiating a CAR. AJI will send a memo from the AJI Vice President to the affected Service Unit Vice President to initiate a CAR. AJI must include all available information regarding the safety related finding, a request for a CAP and point of contact, and a response due date in the memo. AJI must attach any available supporting documentation AJI will normally request an initial response within 30 business days of receipt of a CAR.
b. Submitting a Response. The affected service unit must submit a response by the response due date. The initial response should include:
(1) A CAP describing the action the service unit plans to take to correct the safety related finding. The service unit may be required to conduct Safety Risk Management (SRM) of proposed corrective actions; see JO 1000.37 for SRM requirements. The elements of an acceptable CAP are described below. The person or office affected for completing each action and the expected start and completion dates should be included for each element.
(a) Short-Term Corrective Action (if applicable). Describe the immediate action(s) that will be taken to correct the deficiency (i.e., safety issue or noncompliance.)
(b) Factors. Identify the factors (root, causal or contributing) that led to the safety issue or noncompliance and the methodology used to identify them.
(c) Corrective Action Strategy. Describe the actions to be taken to address the factors and to prevent recurrence.
c. Closing a CAR. If AJI determines that the CAP and/or actions taken are sufficient, AJI will notify the affected Service Unit Vice President, via memo, that the CAR is closed.
09/30/13 JO 7010.14 Appendix B
2. Resolution Request.
a. AJI will notify the affected service unit, via the report cover memo, regarding any findings that will not result in a CAR, but require resolution. AJI will provide information regarding the finding and will assign a response due date. AJI will normally request a response within 45 days of receipt of the request. Independent operational assessment findings are addressed as part of the inservice decision action plan.
b. Submitting a Response. The affected service unit must submit a response by the response due date. The initial response should describe the actions to be taken to resolve the finding and a point of contact. If additional time is needed, the service unit's initial response should indicate when a response will be provided.
Note: The service unit may submit documents, records, data, etc. as evidence that the noncompliance or safety concern is not valid. If the evidence is sufficient, AJI will close the issue.
c. If AJI determines the action(s) proposed does not appropriately address the finding, AJI will work with the service unit point of contact to try to resolve the finding. If the finding is not resolved, AJI may elevate the issue to the service unit's next level of management to determine action.
d. AJI will track the status of findings in an AJI safety management tracking tool. AJI may request that the service unit provide periodic (monthly, quarterly, etc.) updates regarding the status of open findings.
e. If AJI determines the action(s) proposed or taken is sufficient, AJI will notify the affected service unit, via memo or electronic mail, that the finding is closed.
Appendix C: Audit and Assessment Supplemental Procedures
C-1: Operational Peer Assessments
C-2: Independent Operational Assessments
Appendix C: Audit and Assessment Supplemental Procedures
C-3: Safety Management System Audits and/or Assessments
Appendix C-1: Operational Peer Assessments
1. Purpose and Scope. An Operational Peer Assessment (OPA) is conducted by AJI, in collaboration with the affected service unit, based on a suspected risk, safety related trend, or other special emphasis item or concern. An OPA is conducted to evaluate a facility’s operational and technical adequacy in delivering air traffic control services. The primary focus of an OPA is management's involvement in and oversight of facility staffing and scheduling, operations and procedures, training, and quality control efforts. An OPA may also focus on the management communication effectiveness, team building and collaboration (internal and external), facility safety culture, etc. During the conduct of an OPA, the assessment team may provide coaching, guidance, and tools to the local management team.
Appendix C-1: Operational Peer Assessments
2. Initiating an OPA. An OPA may be directed by the AJI Vice President or management official or requested by the Service Unit Vice President or management official. ATO service units may submit a request for an OPA to AJI via official correspondence. The request must include a point of contact for the requesting service unit, the facility to be assessed, any background information, and a description of the suspected risk, safety related trend, or special emphasis item or concern. AJI and/or the affected service unit may also identify OPAs to be conducted throughout the fiscal year.
3. Assessment Team: At a minimum, the team will consist of a team lead and two peer managers.
a. AJI will assign a team lead to manage, oversee, and lead the OPA.
b. The assessed service unit will designate at least two peer managers to conduct the OPA. For the purposes of this process, a peer manager is a management official who has similar experience and expertise as the air traffic manager at the facility being assessed. A peer manager participates in the OPA as a team member and may also serve as an advisor to the team lead.
c. Depending on the size and complexity of the facility being assessed, additional personnel may be needed to conduct the OPA. AJI may assign, and/or the director of operations may designate, additional team members to conduct the OPA.
4. Notification. AJI will notify the headquarters director of operations within 90 days of a scheduled OPA or 30 days of an ad hoc OPA, unless otherwise coordinated. The headquarters director of operations will ensure that the notification is sent to the affected service area diretor of operations, quality control group manager, district manager, facility manager, etc. At a minimum, the notification must include the objective, scope, the facility to be assessed, team information, and the dates the team will be on-site.
5. Planning and Conducting.
a. The team lead will oversee the development of any documents (e.g., checklists, data sheets, questionnaires, etc.) needed to conduct the assessment. The team lead or peer managers may contact the service unit or facility to request information needed in preparation for an OPA.
b. The team lead will develop the team's schedule. The schedule should include the team's estimated arrival and departure times to and from the facility, opening and closing meetings, any planned activities (e.g., interviews, data gathering, observations, etc.) and team assignments for each activity. The schedule may also include information regarding team meetings. Team members are requested to attend all team meetings.
c. On-site activities.
(1) The team lead will conduct an opening meeting with the facility manager, or designee, to explain the objective and scope of the assessment and the planned activities. The manager, or designee, may invite headquarters, service area, service center, district, facility personnel, etc., to attend the opening meeting.
(2) When applicable, bargaining unit personnel may participate in an OPA in accordance with the collective bargaining agreement. Participation of bargaining unit or additional personnel (e.g., facility, service area, district, quality control group, etc.) will be at the direction of the team lead.
(3) The assessment team will review pertinent documents and/or records, conduct interviews with management personnel (or designee), and may observe operational procedures. Team members must keep the team lead informed of information discovered during the assessment. The team will meet to determine preliminary results.
(4) The team lead will conduct a closing meeting with the facility manager, or designee, to discuss the preliminary assessment results. The facility manager, or designee, may invite service unit, service area, service center, district, facility personnel, etc. to attend the closing meeting.
6. Finalize Results: The team will finalize the assessment results. The results may include nonsafety related and safety related findings. The peer managers will make recommentations regarding any findings requiring resolution. The team lead will brief AJI management officials and then brief the assessed service unit management officials.
7. Assessment Report. The team lead, in collaboration with the team, will prepare the cover memo and report. The report will include the final assessment results and peer managers’ recommendations. AJI will send the report to the headquarters Director of Operations within five business days of the AJI Director's approval. The headquarters Director of Operations will ensure that the report is sent to the affected Service Area Diretor of Operations, Quality Control Group Manager, District Manager, Facility Manager, etc.
8. Resolution Requests. AJI must notify the affected service unit, via the cover memo, when a response is requested to address a finding. AJI will record, and may track, the status of OPA findings in an AJI safety management tracking tool.
a. AJI typically will not initiate a CAR to resolve OPA findings. However, the team will elevate any significant safety hazards or risk to the NAS to the Facility Manager and the District Manager immediately.
09/30/13 JO7010.14 Appendix C
b. The team may identify non-safety related findings associated with management effectiveness and efficiencies. The District Manager must provide a 90-day follow-up to the Service Area Director of Operations regarding these issues or concerns.
Appendix C-2: Independent Operational Assessments
Appendix C-2: Independent Operational Assessments
1. Purpose and Scope. The Federal Aviation Administration (FAA) is committed to verifying that new solutions are operationally effective, suitable, and safe before deployment. An Independent Operational Assessment (IOA) is a system-level assessment conducted in an operational environment to assess the operational readiness of a system before it is incorporated into the National Airspace System. An IOA provides decision-makers with an independent determination of operational readiness in support of production and In-Service Decisions (ISD) as required by the FAA Acquisition Management System policy.
2. Initiating an IOA. The Chief Operating Officer, through the Vice President of Air Traffic Organization (ATO) Safety and Technical Training (AJI), designates solutions on which to conduct an IOA. The decision to designate a solution for IOA is based on such factors as complexity, operational criticality, lifecycle cost, interoperability, and safety risk. The IOA Designation Board makes recommendations as to which programs should be designated.
3. Team Members: AJI personnel serve as the IOA Team Lead and is responsible for defining IOA team membership based on the intended operational use of the system to be assessed and the IOA strategy. A request for IOA team members will be submitted through Labor Management Relations for coordination with the national union offices at least 90 days prior to Program Management Office (PMO) Operational Testing and at least six months prior to the scheduled commencement of any IOA conduct activity. There may be occasional exceptions to this timeframe depending on the Acquisition Team strategy and delays in the program schedule.
IOA teams are typically composed of subject matter experts from the field and headquarters. IOA team member disciplines include Air Traffic Control Specialists, Air Traffic Supervisors, Airway Transportation Systems Specialist, Technical Operations Management, Second-level Suport Specialists, Air Traffic and Technical Operations headquarters personnel and other system users (e.g., Department of Defense, airlines, the National Weather Service, and general aviation.) Team members representing the FAA or other government agencies must be federal employees. Additional personnel may serve as advisors during an IOA, but they are not team members and may not participate in the determination of operational readiness.
4. Notification. AJI will notify facility management at least six months prior to IOA activities of the planned assessment. AJI and local team members will provide facility management and the Service Center Manager a courtesy briefing that serves as a general introduction to the process and the particular program to be assessed. At the completion of the system test activities, the Vice President of the PMO will declare to the Vice President of AJI,via an IOA Readiness Declaration (IOARD), the readiness of the system to enter IOA and operational use at the key site. The Vice President of AJI will direct the commencement of an IOA after accepting the IOARD.
5. Planning and Conducting
5. Planning and Conducting
a. Preparations for an IOA will begin at least six months before conduct at the key site. AJI will develop an IOA Plan and a will lead the team in the development of an IOA Procedure document for
each IOA to be conducted. These documents address the scope of the IOA, along with any limitations to the scope. The procedures are the final refinement of the IOA strategy defined in the plan.
b. AJI will lead the team in the development of questionnaires and interview questions requiring bargaining unit review. The development of questionnaires and interview questions will be completed at least 60 days before the assessment is scheduled. AJI will coordinate questionnaires and interview questions through Labor Management Relations for national union offices to review and comment. After 30 days, the questionnaires and interview questons will be coordinated with the union representative at the key site. AJI will meet with facility management and local union representative to determine the best method for distribution of questionnaires and conduct of interviews at the site.
c. AJI will schedule courtesy inbriefs and outbriefs with appropriate facility managers and local union representatives at the beginning and at the end of the IOA conduct period.
d. AJI will conduct IOA's while system personnel perform normal operations. The team will obtain quanitative and qualitative data through observations in a manner that will not impact operations of a facility. Participants questionnaires and interview responses will also be obtained.
e. After IOA conduct, AJI will conduct data reduction and analysis, followed by a team caucus to determine assessment results.
6. Assessment Report: The IOA report will provide the Vice President of AJI and the production or ISD authority the team's assessment of the hazards and operational concerns and the team's determination of operational readiness for national deployment.
7. Reporting Results/Briefings: The results of IOAs are reported and briefed to the Vice President of AJI, the Vice President of the PMO, appropriate stakeholders and acquisition team members, and the production decision or ISD authority.
8. ISD Action Plans for IOA: As part of the ISD briefing, the PMO will submit action plans that address IOA hazards and operational concerns. AJI will review ISD action plans and attend the ISD stakeholder meeting to address any concerns related to the IOA report. The PMO will provide a monthly status of action plans to the ISD Executive Secretariat, and all actions are tracked until they are closed.
Appendix C-3: Safety Management System Audits and/or Assessments
Appendix C-3: Safety Management System Audits and/or Assessments
Reserved