Notice N JO 3330.70 Operational Assessment Program (OAP)
This document is cancelled. It may have been replaced. Check the official source before relying on it.
Text read by OCR from a scanned file. Check the original, as some words may be misread.
NOTICE U.S. DEPARTMENT OF TRANSPORTATION N JO 3330.70 FEDERAL AVIATION ADMINISTRATION Air Traffic Organization Policy Effective Date: 05/14/12
Cancellation Date: 05/13/13
SUBJ: Operational Assessment Program (OAP)
1. Purpose of this Notice.
a. This notice extends the Operational Assessment Program (OAP) currently implemented at three key sites, and enables the expansion of the program to additional key sites yet to be determined. Using the OAP—a program used to assess the knowledge and skills of air traffic controller (ATC) candidates seeking promotion to level 10 and higher terminal radar approach controls (TRACON) —we will improve our selection process of candidates bidding to the current three key sites and all additional key sites identified.
b. This notice conveys instructions, standards, and guidance for administering the OAP. All management and staff involved in the OAP must be familiar with, and comply with, this notice.
c. A technical report, TRACON Operational Assessment Validation under Contract #DTFAWA-09-A80027, documents the validation of the OAP and resides in the Office of the Director of Community Enterprise, AJG-P.
2. Audience. This notice affects key site Air Traffic Managers (ATMs), Electronic Learning Management System (eLMS) Managers, candidates and their ATMs, and examination proctors and raters.
3. Where to Find This Notice. You can find this notice on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/.
4. Scope. Three key sites currently use the OAP: Chicago Terminal Radar Approach Control (TRACON) (C-90), Atlanta TRACON (A-80), and Southern California TRACON (SCT). AJG-P will assist with the evaluation of additional key sites for participation in the OAP process. During the notice period we will continue to evaluate the OAP, analyze data collected, determine if key facilities can maintain the OAP, and consider alternatives for improving the OAP.
5. Office of Primary Interest (OPI). Director of Community Enterprise, AJG-P, is the OPI for this notice.
6. OAP Processes and Requirements.
a. Under the OAP, a key site ATM initiates a request to open a vacancy announcement for qualified air traffic control specialists. The vacancy announcement will include a reference to
Distribution: Chicago TRACON C90, Southern California Initiated By: AJG-P TRACON SCT, Atlanta TRACON A80, all additional key sites, AJG-0, AJN-0, AJT-0, AGC-0, AHR-0
the OAP as part of the evaluation process. The FAA Office of Human Resource Management sends the key site ATM a certified list of eligible candidates. The key site ATM provides the list of candidates to the Manager, Technical Workforce Services Team (AJG-P33), whose responsibility includes loading the knowledge examination into each candidate's eLMS learning plan. AJG-P33 will generate a unique proctor code login for use in accessing the test and will provide these codes to the key site ATM.
NOTE: At their discretion, key site and/or the candidate's ATMs may delegate any portion of OAP responsibilities to non-bargaining unit employees.
b. The key site ATM will notify the candidate's ATM using the OAP Proctor's Checklist (Appendix A), which provides instructions for the administration of the OAP knowledge examination. In a separate email, the key site ATM will provide the candidate's ATM with a unique code for the test proctor to use to login into the eLMS knowledge exam.
c. This multiple-choice examination measures the fundamental knowledge necessary to be successful at the key sites. Candidates must achieve the minimum cut score on the knowledge examination prior to further assessment under the vacancy announcement.
d. Upon notification from the key site ATM, candidates have ten business days to complete the knowledge examination. When the ten-day period has expired, AJG-P33 will gather the candidates' results from eLMS and notify the key site ATM.
e. If the candidate's ATM is unable to proctor the examination according to the guidelines in Appendix A, and a non-bargaining unit designee is unavailable, the candidate's ATM will notify the key site ATM. The key site ATM is responsible for coordinating an alternate test location.
f. A candidate has one attempt to complete this examination successfully per vacancy announcement. Successful knowledge examination results are valid for one year, and candidates may apply the results to subsequent OAP vacancy announcements at selected key sites covered by this notice.
g. When the examination is over, the Proctor's Checklist (Appendix A) must be completed, signed by the proctor and candidate, and returned to the key site ATM. The original is kept in accordance with applicable agency policy related to employee hiring.
h. The key site ATM notifies the candidate of their results using the OAP Candidate Notification (Appendix B), which includes additional OAP instructions and information.
i. If the candidate achieves the minimum cut score for the knowledge examination, they will be scheduled for further familiarization and evaluation using practical radar scenarios at the key site.
j. If the candidate does not achieve the minimum cut score for the knowledge examination, the candidate will be given no further consideration in the application process for that particular vacancy announcement. However, the candidate may bid and re-test for subsequent vacancy announcements.
7. Familiarization Package. a. The key site ATM will send a familiarization package (see Checklist of Familiarization Package Contents (Appendix C)) to those candidates who achieve the minimum cut score on the knowledge examination and are eligible for further assessment.
b. This familiarization package helps candidates prepare for the radar scenarios on which they are going to be rated.
8. Radar Scenario Familiarization at the key site.
a. Candidates will receive three hours of familiarization prior to running six practice radar scenarios. This includes review of the airspace, local procedures, and basic radar techniques.
b. The practice scenarios familiarize the candidate with the designated airspace and pertinent procedures.
9. Radar Scenarios.
a. Nine OAP radar scenarios are developed and maintained according to the Radar Skills Scenario Development Checklist (Appendix D). Assessment of OAP candidates only employs radar scenarios developed within these guidelines.
b. Candidates receive six practice and three rated radar scenarios. A rater—using the Operational Assessment Rating Form (Appendix E)——assesses the candidate on the three rated scenarios.
c. If, for any reason (e.g. on-site equipment failure, software or lab problem), any candidate(s) cannot complete the radar scenarios, the key site ATM will determine the appropriate course of action to ensure all candidates receive equal opportunities to demonstrate performance.
10. OAP Raters.
a. Key site ATMs are responsible to ensure that raters are qualified in accordance with the OAP Rater Qualification Checklist (Appendix F). The key site ATMs will maintain a list of current raters.
b. OAP raters at the key site will provide the candidates with three hours of familiarization and six practice scenarios.
c. For every candidate, an OAP rater at the key site will complete the Operational Assessment Rating Form for each of the three rated scenarios.
11. Key site ATMs.
a. Assessment data collected using the rating forms in Appendix E shall be tabulated according to every candidate's cumulative or average score obtained for all three rated scenarios.
b. Key site ATMs will review OAP assessment data collected and information contained in the candidate's application prior to making any selections.
c. Key site ATMs may use any other information relevant to a candidate's qualifications prior to making a selection. However, rationale should be documented if a candidate is selected that had an overall lower score on the rated scenario portion of the OAP than another candidate on the same referral list.
d. In accordance with applicable agency policies related to employee hiring and records retention, the key site ATM keeps all performance results from the OAP for every candidate.
12. Evaluating the OAP.
a. AJG-P33 will evaluate the OAP throughout the duration of the program. The ability to maintain the program within established guidelines at each key site is vital to the long-term direction of this initiative. At any time during the OAP selection process, the Director of Community Enterprise, AJG-P, may review the process at any key site or knowledge examination site. The review ensures guidelines and testing procedures are followed in accordance with this notice.
b. AJG-P33 will issue a summary program review report with program recommendations to the Vice President of Management Services, AJG-0, before the expiration of this notice and after collaborating with:
• Office of Human Resources Management Programs and Policies, AHP-001; • Office of Human Resources Field Operations, AHF-001; Office of Personnel Selection Research Team, AAM-520 (CAMI); Manager, Terminal Field Support and Requirements, AJT-6; and • Key site ATMs.
13. Distribution. Distribution of this notice is to Chicago TRACON C90, Southern California TRACON SCT, Atlanta TRACON A80, all additional key sites, Management Services Unit, (AJG-0); Operations Service Unit (AJN-0); Terminal Services Unit (AJT-0); Office of Chief Counsel (AGC-0); and Human Resource Management (AHR-0); Office of Personnel Selection Research Team, AAM-520 (CAMI).
SSavid bepple J. David Grizzle
Chief Operating Officer Air Traffic Organization
05/14/12 N JO 3330.70 Appendix A
Appendix A: Operational Assessment Program (OAP) Proctor's Checklist
Appendix A: Operational Assessment Program (OAP) Proctor's Checklist
The OAP requires candidates to complete a knowledge-based examination through eLMS within ten (10) business days of notification by the key site ATM. The OAP Proctor is responsible for verifying that testing complies with the terms in this notice. The Proctor will document the testing process using this checklist. Once the process is complete, the Proctor will return this form, signed and dated, to the key site ATM.
Date Task Completed Candidate's ATM received instructions to proctor the examination
Advised candidate they won't be allowed to leave test area once login is complete
Candidate advised they have two hours to complete examination
Candidate logged in using proctor code provided by key site ATM
Candidate completed knowledge examination in presence of proctor
No reference materials or electronic devices were used during knowledge examination
Examination completed in one continuous session at a location free of distractions
By signing this form, proctor and candidate acknowledge that the examination was conducted according to this checklist.
(Candidate printed name) (Candidate signature) (Date)
(Proctor printed name/title) (Proctor signature) (Date)
Appendix B
Appendix B: OAP Candidate Notification
Appendix B: OAP Candidate Notification
Candidate (Printed name)
This is official notification of the results of the OAP knowledge examination associated with vacancy announcement
Your results have been recorded as: Achieved cut score Did not achieve cut score Incomplete
If your results are noted above as “Achieved cut score," you will receive further instructions from the facility associated with this vacancy announcement with information regarding further participation in the selection process.
If your results are noted above as “Did not achieve cut score," or “Incomplete," you will not be given further consideration under the associated vacancy announcement. If you have questions, please contact the key site ATM or his/her designee.
Key site ATM (Printed Name) (Signature)
Date
Key site telephone number
N JO 3330.70 Appendix C
Appendix C: Checklist of Familiarization Package Contents
Appendix C: Checklist of Familiarization Package Contents
Key site ATMs are responsible for ensuring the familiarization package contains all of the items on
the checklist.
Item Schedule of activities at the key site
Description of airspace
Description of scenario procedures
Types of aircraft in the scenarios
Aircraft call signs used in the scenarios
check mark
Description of the five (5) performance dimensions in Appendix E,
Operational Assessment Rating Form
Map of airspace
Approach plates for arrival airports
Key site ATM (Printed Name)
Date
(Signature)
05/14/12 N JO 3330.70 Appendix D Appendix D: Radar Skills Scenario Development Checklist
The key site ATM is responsible for ensuring the practice and rated scenarios used in the OAP conform to this checklist. Scenario Guidelines (check mark) Airspace designed to be sufficiently explained within three (3) hours Three (3) or less airspace shelves
Radar display of airspace is 45 miles or less
Ten (10) or fewer airspace fixes
Three (3) or fewer departure gates
Scenarios do not exceed 45 minutes
Scenarios do not contain any runway configuration changes Scenarios do not contain any standard instrument departures (SIDs)
Scenarios do not have more than two instrument approaches at each primary airport
Scenarios are comprised of at least two, but no more than three events
All events last 15 minutes
All events have 6 to 15 aircraft that require control decisions or actions
Distracter aircraft, if used, limited to 5 or fewer per scenario
All events use a mix of aircraft weight classes
All aircraft used are in small, large, or heavy weight classes
No aircraft with speeds less than 170 knots on the final
Only call signs of major air carriers are used
Only November used for general aviation call signs
No foreign civilian registry aircraft or foreign air carriers
No military call signs or military aircraft types Vectoring not required to more than one (1) runway at a major airport and one (1) runway at a satellite airport Key site ATM Date (Printed Name) (Signature) (Key site ATM acknowledges all radar scenarios were developed in compliance with program guidelines)
05/14/12 N JO 3330.70 Appendix E
Appendix E: Operational Assessment Rating Form
OPERATIONAL ASSESSMENT
RATING FORM
Communication
Uses clear, concise transmission to pilots and outside sources (i.e., controllers) uses standard phraseology; effectively uses resources; remains calm and maintains professional manner.
RATING FORM
Candidates name (printed)
Repeats transmissions due to lack of clarity
Frequently misses Readback/hearback errors
Does not combine transmissions
Regularly transposes words or numbers
Uses excessive information in a single transmission
1 2
Rater Ops Initials
Minor phraseology errors
Uses consistent, but nonstandard phraseology that is effective (e.g., approach clearance items out of sequence)
Misses some readback/hearback errors
On occasion, pilot questions a transmission
3 4
Date
Maintains clear and concise communications
Ensures all transmissions are understood
Readback/hearback accurate
Good frequency management
Adheres to established standard operating procedures and FAA Order 7110.65, Air Traffic Control
5 6
Working Speed
Maintains the appropriate pace of work; is deliberate in approach to work; adjusts working speed as required.
Candidates name (printed)
Is reactive versus proactive
Poor pattern, irregular traffic flow
Communication lacks confidence or slow to communicate given the situation
Slows down all aircraft to deal with increased volume/complexity (no clear plan)
Departures not released in a timely manner
Fails to pace control actions and associated tasks at an acceptable rate
1 2
Rater Ops Initials
Control decisions are sometimes/occasionally late
Handoffs and point outs are occasionally late Transmissions are behind (i.e., behind the power curve)
Aircraft patterns are not precise or have errors in pattern
Experienced few delays
3 4
Date
Makes control decisions in a timely manner
Pattern
o Speed control
Handoffs and point outs
Organized strips
Responds appropriately to increases in traffic volume/complexity
Acts rapidly to correct errors
5 6
Planning and Projection
Maintains constant situational awareness by planning and prioritizing approach to work, vigilantly scans the situation including assigned airspace and outside your area (i.e., other sectors); balances workload efficiently; mentally projects future actions and events based on current conditions; recognizes items that can effect planning, such as aircraft types, speeds, weather conditions, controller abilities, training,
and distractions.
Candidates name (printed)
Rater Ops Initials
Lacks a disciplined, structured pattern (e.g., inconsistent or excessive vectoring) Does not fill slots or gaps; does not execute arrival/departure sequences Repeated vectoring inside approach gate Uses reactionary control actions to resolve unforeseen situations Uses unnecessary altitude/speed or vector assignments Experiences repeated loss of separation Allows intercepting localizer at an altitude above the glide scope Issues approach clearances with inappropriate intercept angle Allows aircraft to go through final/inefficient headings Unnecessarily tunnels departures (when airspace is clear) Has arrival/departure delays Unable to project and detect conflicts
1 2
Experiences Infrequent loss of separation Occasionally uses excessive spacing Has infrequent airspace violations Occasionally late with coordination (e.g., point outs when others are entering adjacent airspace) Sometimes violates SOP Seldom requires use of other airspace to manage traffic flow
3 4
Date
Efficiently vectors aircraft
Determines an initial plan/or sequence; takes proactive measures to implement plan • Applies speed control to build and maintain intervals for spacing and/or flow control Provides safe, orderly, and efficient traffic flow and spacing • Uses effective strip board management; accurate strip/pad marking • Adheres to established standard operating procedures and Order 7110.65
5 6
Prioritizing
Determines and responds to the most critical tasks needing to be accomplished within specified time limits; recognizes and
responds to appropriate issues.
Candidates name (printed)
Poor execution of planned control actions
Regularly increases workload through lack of prioritization (e.g., making extra transmissions)
Actions negatively impacts adjacent sectors
Allows distractions to take away from necessary tasks (e.g., landline calls)
1 2
Rater Ops Initials
Occasionally addresses secondary task when should be focused on more critical tasks
Timing of control instructions sometimes late
Misses characteristics of performance/does not recognize or consider
Recognizes need for an alternate plan, but unable to develop and execute it
Occasionally impacts adjacent sector
3 4
Date
Recognizes aircraft performance
Maintains focus on critical tasks despite distractions
Flexible order of duty, able to deal with required changes
Identifies priorities of transmissions (who you should talk to first, etc.)
Works aircraft in order of importance
Adjusts the timing of control instructions to meet requirements
5 6
Speed Control
Speed control is based on control decisions and used to achieve appropriate spacing in the pattern and appropriate spacing intervals on final, taking compression into account.
Candidates name (printed)
Rater Ops Initials
Unnecessarily delays aircraft
Spacing is inconsistent or separation is lost
Assigns non-standard and/or inconsistent speeds
Uses continued and excessive speed adjustments
Incorrectly uses speed reduction for purposes other than to maintain or achieve an interval
1 2
Experiences occasional delays
Uses inconsistent spacing
Experiences occasional unrecognized compression
Occasionally issues unnecessary speed adjustments
3 4
Date
Maintains appropriate and consistent spacing on final and out departure gates
Experiences no delays
Recognizes intervals and uses speed control to achieve it
Plans speed for controlled turns
Recognizes the max speed lead aircraft should be at in descent areas
Adheres to established standard operating procedures and Order 7110.65
5 6
05/14/12 N JO 3330.70 Appendix F
Appendix F: OAP Rater Qualification Checklist
Appendix F: OAP Rater Qualification Checklist
The key site ATM is responsible for ensuring that raters have completed all the items on this checklist.
Rater tasks/qualifications check mark Review the candidate's familiarization package
Review this notice (N JO 1xxxx.x)
Run all practice radar scenarios
Run all rated radar scenarios
Complete Fundamentals of ATC On-The-Job-Instruction (OJTI) Course 55049
Review information provided by AJG-P33 regarding common rater problems and biases Assess a sample performance from each of the three rated scenarios using Appendix E of N JO 1xxxx.x, Operational Assessment Rating Form
Rater's name/title (Printed Name) (Signature)
Date Qualified
ATM signature (Printed Name) (Signature)