Notice N JO 3330.71 Operational Assessment Program (OAP)
This document is cancelled. It may have been replaced. Check the official source before relying on it.
Text read by OCR from a scanned file (1 of 15 pages). Check the original, as some words may be misread.
NOTICE N JO 3330.71 U.S. DEPARTMENT OF TRANSPORTATION
NOTICE N JO 3330.71 U.S. DEPARTMENT OF TRANSPORTATION
FEDERAL AVIATION ADMINISTRATION
Air Traffic Organization Policy Effective Date: 04/09/14
Cancellation Date: 04/09/15
SUBJ: Operational Assessment Program (OAP)
1. Purpose of this Notice.
a. This notice reestablishes the Operational Assessment Program (OAP) at the three key sites adds a forth key site and enables the expansion of the program to additional key sites yet to be determined. Using the OAP—a program used to assess the knowledge and skills of Air Traffic Controller (ATC) Candidates seeking promotion to level 10 and higher terminal radar approach controls (TRACON) —we will improve our selection process of candidates bidding to the current four key sites and all additional key sites identified.
FEDERAL AVIATION ADMINISTRATION
b. This notice conveys instructions, standards, and guidance for administering the OAP. All management and staff involved in the OAP must be familiar with, and comply with, this notice.
c. A technical report, TRACON Operational Assessment Validation under Contract #DTFAWA-09-A80027, documents the validation of the OAP and is maintained in the Office of the Director of Resource Enterprise, AJG-R.
2. Audience. Key Site Air Traffic Managers (ATMs), FAA Electronic Learning Management System (eLMS) managers, candidates and their ATMs, examination proctors and raters.
3. Where Can I Find This Notice? You can find this order on the MyFAA Employee website. Select "Tools & Resources," then select "Orders & Notices."
4. Scope. This notice designates Chicago (C90) and New York (N90) Terminal Radar Approach Control (TRACON) facilities as key sites for the OAP. AJG-R will assist with the evaluation of additional key sites for participation in the OAP process. During the notice period, we will continue to evaluate the OAP, including previous key sites, Atlanta TRACON (A80) and Southern California TRACON (SCT), analyze data collected, determine if key facilities can maintain the OAP, and consider alternatives for improving the OAP.
5. Office of Primary Interest (OPI). The Director of Resource Enterprise, AJG-R, is the OPI for this notice.
Distribution: C90, N90, A80, SCT, AJG-0, AJO-0, AJT-0, AGC-1, Initiated By: AJG-R4 AHR-0, AAM-001
6. OAP Processes and Requirements.
a. Under the OAP a key site ATM initiates a request to open a vacancy announcement for qualified air traffic control specialists. If the ATM chooses to use the OAP, the vacancy announcement will include a reference to the OAP as part of the evaluation process. The FAA Office of Human Resource Management will provide the key site ATM a certified list of eligible candidates. The key site ATM' will provide a list of candidates to the Manager, Technical Requirements and Forecasting Group (AJG-R4), who will ensure that the knowledge examination is loaded into each candidate's eLMS learning plan. AJG-R4 will generate a unique proctor code login for use in accessing the test and will provide these codes to the key site ATM.
NOTE: At their discretion, key site and/or the candidate's ATMs may delegate any portion of OAP responsibilities to non-bargaining unit employees.
b. The key site ATM will notify the candidate's ATM using the OAP Proctor's Checklist (appendix A), which provides instructions for the administration of the OAP knowledge examination. In a separate email, the key site ATM will provide the candidate's ATM with a unique code for the test proctor to use to login into the eLMS knowledge exam.
c. This multiple-choice examination measures the fundamental knowledge necessary to be successful at the key sites. Candidates must achieve the minimum cut score on the knowledge examination prior to further assessment under the vacancy announcement.
d. Upon notification from the key site ATM, candidates have ten business days to complete the knowledge examination. When the ten-day period has expired, AJG-R4 will gather the candidates' results from eLMS and notify the key site ATM.
e. If the candidate's ATM is unable to proctor the examination according to the guidelines in appendix A, and a non-bargaining unit designee is unavailable, the candidate's ATM will notify the key site ATM. The key site ATM is responsible for coordinating an alternate test location.
f. A candidate has one attempt to successfully complete this examination per vacancy announcement. Successful knowledge examination results are valid for one year, and candidates may apply the results to subsequent OAP vacancy announcements at selected key sites covered by this notice.
g. When the examination is over, the checklist in appendix A must be completed, signed by the proctor and candidate, and returned to the key site ATM. The original is kept in accordance with applicable agency policy related to employee hiring.
h. The key site ATM notifies the candidate of their results using the OAP Candidate Notification (appendix F), which includes additional OAP instructions and information.
i. If the candidate achieves the minimum cut score for the knowledge examination, they will be scheduled for further familiarization and evaluation using practical radar scenarios at the key site.
j . If the candidate does not achieve the minimum cut score for the knowledge examination, the candidate will be given no further consideration in the application process for that particular vacancy announcement. The candidate may bid and re-test for subsequent vacancy announcements.
7. Familiarization Package.
a. The key site ATM will send a familiarization package (see Checklist of Familiarization Package Contents (Appendix E) to those candidates who achieve the minimum cut score on the knowledge examination and are eligible for further assessment.
b. This familiarization package helps candidates prepare for the radar scenarios on which they are going to be rated.
8. Radar Scenario Familiarization at the key site.
a. Candidates will receive three hours of familiarization prior to running six practice radar scenarios. This includes review of the airspace, local procedures, and basic radar techniques.
b. The practice scenarios familiarize the candidate with the designated airspace and pertinent procedures.
9. Radar Scenarios.
a. Nine OAP radar scenarios are developed and maintained according to the Radar Skills Scenario Development Checklist (Appendix B). Assessment of OAP candidates only employs radar scenarios developed within these guidelines.
b. Candidates receive six practice and three rated radar scenarios. A rater—using the Operational Assessment Rating Form (Appendix C)—assesses the candidate on the three rated scenarios.
c. If, for any reason (e.g. on-site equipment failure, software or lab problem), any candidate(s) cannot complete the radar scenarios, the key site ATM will determine the appropriate course of action to ensure all candidates receive an equal opportunity to demonstrate performance. If necessary the ATM will consult with the appropriate policy, Human Resource, or Labor Management Resource point of contact.
10. OAP Raters.
a. Key site ATMs are responsible to ensure that raters are qualified in accordance with the OAP Rater Qualification Checklist (Appendix D). The key site ATMs will maintain a list of current raters. Raters should be recertified every two years.
b. OAP raters at the key site will provide the candidates with three hours of familiarization and six practice scenarios.
c. For every candidate, an OAP rater at the key site will complete the Operational Assessment Rating Form for each of the three rated scenarios.
11. Key site ATMs.
a. Assessment data collected using the rating forms in Appendix E shall be tabulated according to every candidate's cumulative obtained for all three rated scenarios.
b. Key site ATMs will review OAP assessment data collected and information contained in the candidate's application prior to making any selections.
c. In accordance with applicable agency policies related to employee hiring and records retention, the key site ATM keeps all performance results from the OAP for every candidate.
12. Evaluating the OAP.
a. AJG-R4 will evaluate the OAP throughout the duration of the program. The ability to maintain the program within established guidelines at each key site is vital to the long-term direction of this initiative. At any time during the OAP selection process, the Director of Resource Enterprise, AJG-R, may review the process at any key site or knowledge examination site. The review ensures guidelines and testing procedures are followed in accordance with this notice.
b. AJG-R4 will issue a summary program review report with program recommendations to the Vice President of Management Services, AJG-0, before the expiration of this notice and after collaborating with:
. Assistant Administrator for Human Resource Management, AHR-001 . Office of Personnel Selection Research Team, AAM-520 (CAMI); • Manager, Air Traffic Services Field Support and Requirements, AJT-6; and . Key site ATMs. All current and future key sites , Management Services Unit (AJG-0); Air Traffic Services Unit (AJT-0); Office of Chief Counsel (AGC-0); and Human Resource Management (AHR-0), Office of Personnel Selection Research Team, AAM-520 (CAMI).
Teri L. Bristol Chief Operating Officer Air Traffic Organization
Appendix A
Appendix A
Appendix A. Operational Assessment Program (OAP)
Proctor's Checklist
The OAP requires candidates to complete a knowledge based examination through eLMS within ten (10) business days of being notified by the Key Site ATM. The OAP proctor is responsible for verifying that testing is completed according to this notice. Proctor will document the testing process using this checklist. Once the process is complete the proctor will return this form, signed and dated, to the Key Site ATM.
Appendix A. Operational Assessment Program (OAP)
Task
Date
Completed Candidate's ATM received instructions to proctor the examination
Advised candidate they won't be allowed to leave test area once login is complete
Candidate advised they have two hours to complete examination
Candidate logged in using proctor code provided by Key Site ATM
Candidate completed knowledge examination in presence of proctor
No reference materials or electronic devices were used during knowledge examination
Examination completed in one continuous session at a location free of distractions
By signing this form, proctor and candidate acknowledge that the examination was conducted according to this checklist.
(Candidate printed name) (Candidate signature) (date)
(Proctor printed name/title) (Proctor signature) (date)
04/09/14 N JO 3330..71
Appendix B
Appendix B
Appendix B. Radar Skills Scenario Development Checklist
The Key Site ATM is responsible for ensuring the practice and rated scenarios used in the OAP conform to this checklist. Scenario Guidelines (checkmark) • Airspace designed to be sufficiently explained within three (3) hours
Appendix B. Radar Skills Scenario Development Checklist
Three (3) or less airspace shelves
Radar display of airspace is 45 miles or less
Ten (10) or fewer airspace fixes
Three (3) or fewer departure gates
Scenarios do not exceed 45 minutes
Scenarios do not contain any runway configuration changes
Scenarios do not contain any standard instrument departures (SIDs)
Scenarios do not have more than two instrument approaches at each primary airport
Scenarios are comprised of at least two, but no more than three events
All events last 15 minutes
All events have 6 to 15 aircraft that require control decisions or actions
Distracter aircraft, if used, limited to 5 or fewer per scenario
All events use a mix of aircraft weight classes
All aircraft used are in small, large, or heavy weight classes
No aircraft with speeds less than 170 knots on the final
Only call signs of major air carriers are used
Only November used for general aviation call signs
No foreign civilian registry aircraft or foreign air carriers
No military call signs or military aircraft types
Vectoring not required to more than one (1) runway at a major airport and one (1) runway at a satellite airport
Key Site ATM / Date (Printed Name) (Signature) (Key Site ATM acknowledges all radar scenarios developed in compliance with program guidelines)
04/09/14 NJO 3330.71 Appendix C
Appendix C. Operational Assessment Rating Form
Appendix C. Operational Assessment Rating Form
OPERATIONAL ASSESSMENT
RATING FORM
Appendix C
Effective Communication
Uses clear, concise transmission to pilots and outside sources (i.e., controllers) uses standard phraseology; effectively uses resources; remains calm and maintains professional manner.
Appendix C
• Repeats transmissions due to lack of clarity
• Frequently misses Readback/hearback errors
• Does not combine transmissions
• Regularly transposes words or numbers
• Uses excessive information in a single transmission
® ®
• Minor phraseology errors
• Uses consistent, but nonstandard phraseology that is effective (e.g., approach clearance items out of sequence)
• Misses some readback/hearback errors
• On occasion, pilot questions a transmission
(D <D
• Maintains clear and concise communications
• Ensures all transmissions are understood
• Readback/hearback accurate
• Good frequency management
• Adheres to established standard operating procedures and FAA Order 7110.65, Air
Traffic Control
(D ©
Appendix C
Working Speed
Maintains the appropriate pace of work; is deliberate in approach to work; adjusts working speed as required.
• Is reactive versus proactive
• Poor pattern, irregular traffic flow
• Communication lacks confidence or slow to communicate given the situation
• Slows down all aircraft to deal with increased volume/complexity (no clear plan)
• Departures not released in a timely manner
• Fails to pace control actions and associated tasks at an acceptable rate
® ©
• Control decisions are sometimes/occasionally late
• Handoffs and point outs are occasionally late • Transmissions are behind (i.e., behind the power curve)
• Aircraft patterns are not precise or have errors in pattern
• Experienced few delays
© ®
• Makes control decisions in a timely manner
o Pattern
o Speed control
o Handoffs and point outs
• Organized strips
• Responds appropriately to increases in traffic volume/complexity
• Acts rapidly to correct errors
© ©
Appendix C
Planning and Projection
Maintains constant situational awareness by planning and prioritizing approach to work, vigilantly scans the situation including assigned airspace and outside your area (i.e., other sectors); balances workload efficiently; mentally projects future actions and events based on current conditions; recognizes items that can effect planning, such as aircraft types, speeds, weather conditions, controller abilities, training, and distractions.
Lacks a disciplined, structured pattern (e.g., inconsistent or excessive vectoring)
Does not fill slots or gaps; does not execute arrival/departure sequences
Repeated vectoring inside approach gate
Uses reactionary control actions to resolve unforeseen situations
Uses unnecessary altitude/speed or vector assignments
Experiences repeated loss of separation
Allows intercepting localizer at an altitude
above the glide scope
Issues approach clearances with inappropriate intercept angle
Allows aircraft to go through final/inefficient headings
Unnecessarily tunnels departures (when airspace is clear)
Has arrival/departure delays
Unable to project and detect conflicts
®
Experiences Infrequent loss of separation
Occasionally uses excessive spacing
Has infrequent airspace violations
Occasionally late with coordination (e.g., point outs when others are entering adjacent airspace)
Sometimes violates SOP
Seldom requires use of other airspace to manage traffic flow
• Efficiently vectors aircraft
• Determines an initial plan/or sequence; takes proactive measures to implement
plan
• Applies speed control to build and maintain intervals for spacing and/or flow control
• Provides safe, orderly, and efficient traffic flow and spacing
• Uses effective strip board management; accurate strip/pad marking
• Adheres to established standard operating procedures and Order 7110.65
© ©
Appendix C
Prioritizing
Determines and responds to the most critical tasks needing to be accomplished within specified time limits; recognizes and responds to appropriate issues.
• Poor execution of planned control actions
• Regularly increases workload through lack of prioritization
(e.g., making extra transmissions)
• Actions negatively impacts adjacent sectors
• Allows distractions to take away from necessary tasks (e.g., landline calls)
© ©
• Occasionally addresses secondary task when should be focused on more critical tasks
• Timing of control instructions sometimes late
• Misses characteristics of performance/does not recognize or consider
• Recognizes need for an alternate plan, but unable to develop and execute it
• Occasionally impacts adjacent sector
© ©
• Recognizes aircraft performance
• Maintains focus on critical tasks despite distractions
• Flexible order of duty, able to deal with required changes
• Identifies priorities of transmissions (who you should talk to first, etc)
• Works aircraft in order of importance
• Adjusts the timing of control instructions to meet requirements
© ©
Appendix C
Speed Control
Speed control is based on control decisions and used to achieve appropriate spacing in the pattern and appropriate spacing intervals on final, taking compression into account.
• Unnecessarily delays aircraft
• Spacing is inconsistent or separation is lost
• Assigns non-standard and/or inconsistent speeds
• Uses continued and excessive speed adjustments
• Incorrectly uses speed reduction for purposes other than to maintain or achieve an
interval
© ©
• Experiences occasional delays
• Uses inconsistent spacing
• Experiences occasional unrecognized compression
• Occasionally issues unnecessary speed adjustments
© ©
• Maintains appropriate and consistent spacing on final and out departure gates
• Experiences no delays
• Recognizes intervals and uses speed control to achieve it
• Plans speed for controlled turns
• Recognizes the max speed lead aircraft should be at in descent areas
• Adheres to established standard operating procedures and Order 7110.65
© ©
Appendix D
Appendix D. OAP Eater Qualification Checklist
Appendix D. OAP Eater Qualification Checklist
The Key Site ATM is responsible for ensuring that raters have completed all the items on this checklist.
Rater tasks/qualifications Review the candidate's familiarization package
Review this notice (N JO 3330.71)
Run all practice radar scenarios
Run all rated radar scenarios
Complete Fundamentals of ATC On-The-Job-Instructor (OJTI) Course 55049
Assess a sample performance from each of the three rated scenarios using Appendix C of N JO 3330.71, Operational Assessment Rating Form
checkmark V
Rater's name/title
(Printed Name) (Signature)
Date Qualified
ATM signature
(Printed Name) (Signature)
Appendix E
Appendix E
Appendix E. Checklist of Familiarization Package Contents
Key Site ATMs are responsible for ensuring the familiarization package contains all of the items on the checklist.
Item
Schedule of activities at the key site
Description of airspace
Description of scenario procedures
Appendix E. Checklist of Familiarization Package Contents
Types of aircraft in the scenarios
Aircraft call signs used in the scenarios
Description of the five (5) performance dimensions in Appendix C, Operational Assessment Rating Form
Map of airspace
Approach plates for arrival airports
checkmark *
Key Site ATM
(Printed Name) (Signature)
Date
Appendix F
Appendix F
Appendix F. OAP Candidate Notification
Candidate
(Printed name)
This is official notification of the results of the OAP knowledge examination associated with vacancy announcement
Your results have been recorded as: Achieved cut score Did not achieve cut score Incomplete
Appendix F. OAP Candidate Notification
If your results are noted above as "Achieved cut score" you will receive further instructions from the facility associated with this vacancy announcement with information regarding further participation in the selection process.
If your results are noted above as "Did not achieve cut score" or "Incomplete" you will not be given further consideration under the associated vacancy announcement. If you have questions, please contact the Key Site ATM or his/her designee.
Key Site ATM
(Printed Name) (Signature)
Date
Key Site Phone Number