Notice N JO 3330.69 Operational Assessment Program (OAP)
This document is cancelled. It may have been replaced. Check the official source before relying on it.
Distribution: Electronic Initiated By: AJG-B
NOTICE U.S. DEPARTMENT OF TRANSPORTATION
FEDERAL AVIATION ADMINISTRATION N JO 3330.69
Air Traffic Organization Policy
Effective Date:
05/13/2011
Cancellation Date:
05/14/2012
SUBJ: Operational Assessment Program (OAP) 1. Purpose of this Notice. a. This notice introduces an Operational Assessment Program (OAP) at three Key Sites. Using this OAP, we intend to improve our selection process of candidates bidding to one of the three Key Sites from other air traffic facilities by developing a program to assess their skills. b. This notice explains how the OAP will work and conveys instructions, standards, and guidance for administering the OAP. All management and staff involved in the OAP must be familiar with and comply with this notice. 2. Audience. Key Site Air Traffic Managers (ATMs), FAA Electronic Learning Management System (eLMS) managers, candidates and their ATMs, examination proctors and raters.
FEDERAL AVIATION ADMINISTRATION N JO 3330.69
3. Where to Find This Notice. You can find this order on the MyFAA Employee website: https://employees.faa.gov/tools_resources/orders_notices/ 4. Scope. We will start the OAP at three Key Sites - Chicago Terminal Radar Approach Control (TRACON) (C-90), Atlanta TRACON (A-80), and Southern California TRACON (SCT). During the notice period we will evaluate the OAP, analyze lessons learned, determine if key facilities will be able to maintain the OAP, and consider alternatives for improving the selection process.
5. Office of Primary Interest (OPI). ATO Administration and Talent Management, AJG-B 6. OAP Processes and Requirements. a. Under the OAP, a Key Site ATM initiates a request to recruit for air traffic specialists. The vacancy announcement will include a reference to the OAP as part of the evaluation process. The FAA Office of Human Resource Management sends the Key Site ATM a certified list of eligible candidates. The Key Site ATM provides the list of candidates to the Manager, Technical Workforce Services Team (AJG-B14), whose responsibility includes loading the knowledge examination into each candidate’s eLMS learning plan. AJG-B14 will generate a unique proctor code login to be used to access the test and will provide these codes to the Key Site ATM.
Note: Key Site and candidate’s ATMs may delegate any portion of OAP responsibilities to nonbargaining unit employees at their discretion. b. The Key Site ATM will notify the candidate’s ATM using Appendix A, OAP Proctor’s Checklist which provides instructions for the administration of the OAP knowledge examination and includes the unique login proctor code. c. This multiple-choice examination measures the candidate’s foundational knowledge necessary to be successful at the Key Sites. Candidates must achieve the minimum cut score on the knowledge examination prior to further assessment under the vacancy announcement. d. A candidate is allowed only one attempt at this examination per vacancy announcement. Successful knowledge examination results are valid for one year and candidates may apply the results to subsequent OAP vacancy announcements at any Key Site. e. If the candidate’s ATM is unable to proctor the examination according to the guidelines in Appendix A, and a non-bargaining unit designee is unavailable, the candidate’s ATM will notify the Key Site ATM.
The Key Site ATM is responsible for coordinating an alternate test location. f. Candidates will have ten (10) business days to complete the knowledge examination once notified by the Key Site ATM. Once the ten day period has expired, AJG-B14 will gather candidate results from eLMS and notify the Key Site ATM. g. When the examination is over, the checklist (Appendix A, OAP Proctor’s Checklist) must be completed, signed by the proctor and candidate, and returned to the Key Site ATM. The original must be kept in accordance with applicable agency policy related to employee hiring. h. The Key Site ATM notifies the candidate of their results using Appendix F, OAP Candidate Notification, which includes additional OAP instructions and information. i. If the candidate achieves the minimum cut score for the knowledge examination, they will be scheduled for further familiarization and evaluation using practical radar scenarios at the Key Site. j.
If the candidate does not achieve the minimum cut score for the knowledge examination, the candidate will not be given further consideration in the application process for that particular vacancy announcement. However, the candidate may bid and re-test for subsequent vacancy announcements. 7. Familiarization Package. a. The Key Site ATM will send a familiarization package (see Appendix E, Checklist of Familiarization Package Contents) to those candidates that achieve the minimum cut score on the knowledge examination and are scheduled for further assessment. b. This familiarization package is intended to help candidates prepare for the radar scenarios on which they will be rated.
8. Radar Scenario Familiarization at the Key Site. a. Candidates will receive three hours of familiarization prior to running six practice radar scenarios. This includes review of the airspace, local procedures and basic radar techniques. b. The practice scenarios are intended to familiarize the candidate with the designated airspace and pertinent procedures. 9. Radar Scenarios. a. OAP radar scenarios are developed and maintained according to Appendix B, Radar Skills Scenario Development Checklist. Only radar scenarios developed within these guidelines may be used to assess OAP candidates. b. Candidates will receive three rated radar scenarios. They are assessed by a rater using Appendix C, Operational Assessment Rating Form. c. If the radar scenarios cannot be completed for any reason (i.e. on-site equipment failure, software or lab problem) the Key Site ATM will determine the appropriate course of action to ensure all applicants receive equal opportunities to demonstrate performance. 10. OAP Raters. a. Key Site ATMs are responsible to ensure that raters are qualified in accordance with Appendix D, OAP Rater Qualification Checklist. The Key Site ATMs will maintain a list of current raters. b. OAP raters at the Key Site will provide the candidates with the three hours of familiarization and the six practice scenarios. c. An OAP rater at the Key Site will complete the form in Appendix C, Operational Assessment Rating Form, for each of the three rated scenarios for every candidate. 11. Key Site ATMs.
a. Key Site ATMs will review all OAP assessment data collected for every candidate prior to making any selections.
b. All performance results from the OAP for every candidate must be kept by the Key Site ATM in accordance with applicable agency policies related to employee hiring and records retention. 12. Evaluating the OAP. a. The OAP will be evaluated throughout the duration of the program. The ability to maintain the program within established guidelines at each Key Site is vital to the long term direction of this initiative. At any time during the OAP selection process, the Director of ATO Administration and
Talent Management, AJG-B, may review the process at any Key Site or knowledge examination testing site. The intent of the review is to ensure guidelines and testing procedures are followed in accordance with this notice.
b. Before the expiration of this notice and after collaborating with the following: Office of Human Resources Management Programs and Policies, AHP-001; Office of Human Resources Field Operations, AHF-001; Manager, Terminal Field Support and Requirements, AJT-6; Key Site ATMs, AJG-B 14 will issue a summary program review report with program recommendations to the Director of ATO Administration and Talent Management, AJG-B.
13. Distribution. This notice is distributed to the three Key Site Facilities: Chicago TRACON C90, Southern California TRACON SCT, Atlanta TRACON A80, Strategy and Performance Business Unit, AJG-0, Operations Business Unit, AJN-0, Terminal Services, AJT-0, Office of Chief Counsel, AGC-0 and Human Resource Management, AHR-0.
\ Ol J. David Grizzle Acting Chief Operating Officer Air Traffic Organization
05/13/2011 N JO 3330.69 Appendix A
Appendix A. Operational Assessment Program (OAP) Proctor’s Checklist
Appendix A. Operational Assessment Program (OAP) Proctor’s Checklist
The OAP requires candidates to complete a knowledge based examination through eLMS within ten (10) business days of being notified by the Key Site ATM. The OAP proctor is responsible for verifying that testing is completed according to this notice. Proctor will document the testing process using this checklist. Once the process is complete the proctor will return this form, signed and dated, to the Key Site ATM. Task
Date Completed Candidate’s ATM received instructions to proctor the examination
Advised candidate they won’t be allowed to leave test area once login is complete
Candidate advised they have two hours to complete examination
Candidate logged in using proctor code provided by Key Site ATM
Candidate completed knowledge examination in presence of proctor
No reference materials or electronic devices were used during knowledge examination
Examination completed in one continuous session at a location free of distractions
By signing this form, proctor and candidate acknowledge that the examination was conducted according to this checklist. … … … (Candidate printed name) (Candidate signature) (date) … … … (Proctor printed name/title) (Proctor signature) (date)
05/13/2011 N JO 3330.69 Appendix B
Appendix B. Radar Skills Scenario Development Checklist
Appendix B. Radar Skills Scenario Development Checklist
The Key Site ATM is responsible for ensuring the practice and rated scenarios used in the OAP conform to this checklist. Scenario Guidelines (check mark) Airspace designed to be sufficiently explained within three (3) hours
Three (3) or less airspace shelves
Radar display of airspace is 45 miles or less
Ten (10) or fewer airspace fixes
Three (3) or fewer departure gates
Scenarios do not exceed 45 minutes
Scenarios do not contain any runway configuration changes
Scenarios do not contain any standard instrument departures (SIDs)
Scenarios do not have more than two instrument approaches at each primary airport
Scenarios are comprised of at least two, but no more than three events
All events last 15 minutes
All events have 6 to 15 aircraft that require control decisions or actions
Distracter aircraft, if used, limited to 5 or fewer per scenario
All events use a mix of aircraft weight classes
All aircraft used are in small, large, or heavy weight classes
No aircraft with speeds less than 170 knots on the final
Only call signs of major air carriers are used
Only November used for general aviation call signs
No foreign civilian registry aircraft or foreign air carriers
No military call signs or military aircraft types
Vectoring not required to more than one (1) runway at a major airport and one (1) runway at a satellite airport
Key Site ATM … / … Date … (Printed Name) (Signature) (Key Site ATM acknowledges all radar scenarios developed in compliance with program guidelines)
05/13/2011 N JO 3330.69
Appendix C
Appendix C
Appendix C. Operational Assessment Rating Form
OPERATIONAL ASSESSMENT
RATING FORM
7
05/13/2011 N JO 3330.69
Appendix C
Communication
Uses clear, concise transmission to pilots and outside sources (i.e., controllers) uses standard phraseology; effectively uses
resources; remains calm and maintains professional manner.
Candidates name (printed) … Rater Ops Initials … Date …
Repeats transmissions due to
lack of clarity
Frequently misses
Readback/hearback errors
Does not combine
transmissions
Regularly transposes words or
numbers
Uses excessive information in
a single transmission
Minor phraseology errors
Uses consistent, but
nonstandard phraseology that
is effective (e.g., approach
clearance items out of
sequence)
Misses some
Readback/hearback errors
On occasion, pilot questions a
transmission
Maintains clear and concise
communications
Ensures all transmissions are
understood
Readback/hearback accurate
Good frequency management
Adheres to established
standard operating procedures
and FAA Order 7110.65, Air
Traffic Control
8
05/13/2011 N JO 3330.69
Appendix C
Working Speed
Maintains the appropriate pace of work; is deliberate in approach to work; adjusts working speed as required.
Candidates name (printed) … Rater Ops Initials … Date …
Is reactive versus proactive
Poor pattern, irregular traffic
flow
Communication lacks
confidence or slow to
communicate given the
situation
Slows down all aircraft to deal
with increased
volume/complexity (no clear
plan)
Departures not released in a
timely manner
Fails to pace control actions
and associated tasks at an
acceptable rate
Control decisions are
sometimes/occasionally late
Handoffs and point outs are
occasionally late
Transmissions are behind
(i.e., behind the power curve)
Aircraft patterns are not precise
or have errors in pattern
Experienced few delays
Makes control decisions in a
timely manner
o Pattern
o Speed control
o Handoffs and point outs
Organized strips
Responds appropriately to
increases in traffic
volume/complexity
Acts rapidly to correct errors
9
05/13/2011 N JO 3330.69
Appendix C
Planning and Projection
Maintains constant situational awareness by planning and prioritizing approach to work, vigilantly scans the situation including assigned
airspace and outside your area (i.e., other sectors); balances workload efficiently; mentally projects future actions and events based on
current conditions; recognizes items that can effect planning, such as aircraft types, speeds, weather conditions, controller abilities, training,
and distractions.
Candidates name (printed) … Rater Ops Initials … Date …
Lacks a disciplined, structured pattern (e.g.,
inconsistent or excessive vectoring)
Does not fill slots or gaps; does not execute
arrival/departure sequences
Repeated vectoring inside approach gate
Uses reactionary control actions to resolve
unforeseen situations
Uses unnecessary altitude/speed or vector
assignments
Experiences repeated loss of separation
Allows intercepting localizer at an altitude
above the glide scope
Issues approach clearances with
inappropriate intercept angle
Allows aircraft to go through final/inefficient
headings
Unnecessarily tunnels departures (when
airspace is clear)
Has arrival/departure delays
Unable to project and detect conflicts
Experiences Infrequent loss of
separation
Occasionally uses excessive spacing
Has infrequent airspace violations
Occasionally late with coordination
(e.g., point outs when others are
entering adjacent airspace)
Sometimes violates SOP
Seldom requires use of other
airspace to manage traffic flow
Efficiently vectors aircraft
Determines an initial plan/or sequence;
takes proactive measures to implement
plan
Applies speed control to build and maintain
intervals for spacing and/or flow control
Provides safe, orderly, and efficient traffic
flow and spacing
Uses effective strip board management;
accurate strip/pad marking
Adheres to established standard operating
procedures and Order 7110.65
10
05/13/2011 N JO 3330.69
Appendix C
Prioritizing
Determines and responds to the most critical tasks needing to be accomplished within specified time limits; recognizes and
responds to appropriate issues.
Candidates name (printed) … Rater Ops Initials … Date …
Poor execution of planned
control actions
Regularly increases workload
through lack of prioritization
(e.g., making extra
transmissions)
Actions negatively impacts
adjacent sectors
Allows distractions to take
away from necessary tasks
(e.g., landline calls)
Occasionally addresses
secondary task when should
be focused on more critical
tasks
Timing of control instructions
sometimes late
Misses characteristics of
performance/does not
recognize or consider
Recognizes need for an
alternate plan, but unable to
develop and execute it
Occasionally impacts adjacent
sector
Recognizes aircraft
performance
Maintains focus on critical tasks
despite distractions
Flexible order of duty, able to
deal with required changes
Identifies priorities of
transmissions (who you should
talk to first, etc)
Works aircraft in order of
importance
Adjusts the timing of control
instructions to meet
requirements
11
05/13/2011 N JO 3330.69
Appendix C
12
Speed Control
Speed control is based on control decisions and used to achieve appropriate spacing in the pattern and appropriate spacing
intervals on final, taking compression into account.
Candidates name (printed) … Rater Ops Initials … Date …
Unnecessarily delays aircraft
Spacing is inconsistent and/or
separation is lost due to
compression on final or speed
over-takes
Assigns non-standard and/or
inconsistent speeds
Uses continued and excessive
speed adjustments
Incorrectly uses speed
reduction for purposes other
than to maintain or achieve an
interval
Experiences occasional
delays
Uses inconsistent spacing
Experiences occasional
unrecognized compression
Occasionally issues
unnecessary speed
adjustments
Maintains appropriate and
consistent spacing on final and
out departure gates
Experiences no delays
Recognizes intervals and uses
speed control to achieve it
Plans speed for controlled turns
Recognizes the max speed
lead aircraft should be at in
descent areas
Adheres to established
standard operating procedures
and Order 7110.65
05/13/2011 N JO 3330.69 Appendix D
Appendix D. OAP Rater Qualification Checklist
The Key Site ATM is responsible for ensuring that raters have completed all the items on this checklist. Rater tasks/qualifications check mark Review the candidate’s familiarization package
Review this notice N JO 3330.69
Appendix D. OAP Rater Qualification Checklist
Run all practice radar scenarios
Run all rated radar scenarios
Complete Fundamentals of ATC On-The-Job-Instruction (OJTI) Course 55049
Assess a sample performance from each of the three rated scenarios using Appendix C of N JO 3330.69, Operational Assessment Rating Form
Rater’s name/title … / … (Printed Name) (Signature) Date Qualified … ATM signature … / … (Printed Name) (Signature)
05/13/2011 N JO 3330.69 Appendix E
Appendix E. Checklist of Familiarization Package Contents
Appendix E. Checklist of Familiarization Package Contents
Key Site ATMs are responsible for ensuring the familiarization package contains all of the items on the checklist. Item check mark Schedule of activities at the key site
Description of airspace
Description of scenario procedures
Types of aircraft in the scenarios
Aircraft call signs used in the scenarios
Description of the five (5) performance dimensions in Appendix C, Operational Assessment Rating Form
Map of airspace
Approach plates for arrival airports
Key Site ATM … / … (Printed Name) (Signature) Date …
05/13/2011 N JO 3330.69 Appendix F
Appendix F. OAP Candidate Notification
Appendix F. OAP Candidate Notification
Candidate … (Printed name) This is official notification of the results of the OAP knowledge examination associated with vacancy announcement … Your results have been recorded as: … Achieved cut score … Did not achieve cut score … Incomplete If your results are noted above as “Achieved cut score” you will receive further instructions from the facility associated with this vacancy announcement with information regarding further participation in the selection process. If your results are noted above as “Did not achieve cut score” or “Incomplete” you will not be given further consideration under the associated vacancy announcement. If you have questions, please contact the Key Site ATM or his/her designee. Key Site ATM … / … (Printed Name) (Signature) Date … Key Site Phone Number …