Order 8110.107A Monitor Safety/Analyze Data
This document is cancelled. Cancelled by 8110.107B. Check the official source before relying on it.
U.S. DEPARTMENT OF TRANSPORTATION
U.S. DEPARTMENT OF TRANSPORTATION
FEDERAL AVIATION ADMINISTRATION
National Policy
ORDER 8110.107A
10/1/2012
SUBJ: Monitor Safety/Analyze Data
This order contains the Monitor Safety/Analyze Data (MSAD) process, designed to promote an improved continued operational safety (COS) methodology by incorporating a data-driven, riskbased approach for safety assurance and safety risk management. MSAD supports aviation products throughout their life cycle.
ORDER 8110.107A
This order describes how the Aviation Safety organization (AVS) staff uses MSAD within the Aircraft Certification Service’s (AIR) Safety Management System (SMS) to identify and manage risk in aviation products.
MSAD uses product defined hazard criteria to surface potential hazards from aviation safety data. MSAD uses a standard taxonomy for organizing COS data and promotes quick identification of emerging safety trends.
In addition, MSAD establishes a causal analysis approach. This approach may identify underlying contributing factors, such as process breakdowns, which we then communicate to the appropriate AVS oversight business process owner.
The MSAD process is heavily based on existing industry best practices. MSAD builds a safety risk management model that sets the example for how we expect industry to evolve, in taking responsibility for the safety of their aviation products.
Susan J. M. Cabler Acting Manager, Aircraft Engineering Division Aircraft Certification Service
Distribution: Electronic Initiated By: AIR-150
I I
REVISION HISTORY
REVISION HISTORY
Rev Description of Change Effective Date
0 Original 3/12/2010
1 Major Changes:
Page 1, Para 1-2: Clarified that references to “ACO” may
apply to other offices in AIR when those offices have responsibility for continued operational safety oversight Page 1, Para 1-3: Changed all references of AIR-140 to AIR
0 Original 3/12/2010
150, the new business process owner Page 4, Fig. 2: Revised MSAD process flow to enhance
readability and reflect changes to process Page 6, Para 2-6: More guidance for marking events as “false
positives” Page 7, Para 2-7: Added guidance for minimizing single
point safety decisions during the preliminary risk assessment Page 8, Para 2-9: Clarified responsibilities for TSO articles
and standard parts installed on multiple product types that span purview of more than one directorate Page 9, Para 2-9a: Added information encouraging
involvement of AEG Page 10, Para 2-9c: Added a third risk value, “Time until
control program risk guideline is reached,” which must be calculated as part of the risk analysis Page 12, Fig. 4: Added a diagram outlining the corrective
action timeline Page 13, Para 2-10: Moved CARB section (previously para
2-15) for improved readability and added responsibility for CARB when dealing with MCAIs Page 15, Para 2-11: Clarified when a “structured” root causal
analysis must be performed Page 17, Para 2-15b: Added guidance for determining the
appropriate corrective action Page 26, Chapter 4: Added chapter on applying the MSAD
process to foreign products Page 33, Chapter 7: Added chapter detailing situations where
exceptions may be made to the MSAD process
10/1/2012
ii
Table of Contents
Paragraph Page
Chapter 1. General Information … 1
Chapter 1. General Information … 1
1-1. Purpose of this Order … 1 1-2. Audience … 1 1-3. Certificate Holder Engagement … 1 1-4. Where to Find This Order … 1
Chapter 2. MSAD Process Overview … 2
2-1. Purpose of the MSAD Process … 2 2-2. Range of the MSAD Process … 2 2-3. MSAD Overview … 2 2-4. The MSAD Process Flow … 3 2-5. Acquire Data (Step 1.0) … 6 2-6. Hazard Criteria Analysis (Step 2.0) … 6 2-7. Perform Preliminary Risk Assessment (Step 3.0) … 7 2-8. Initiate Immediate Corrective Action (Step 4.0) … 8 2-9. Record Risk Analysis Results (Step 5.0) … 8 2-10. CARB … 13 2-11. Safety Issue Cause(s) (Step 6.0) … 15 2-12. Perform Causal Analysis (Step 7.0) … 16 2-13. Document the Cause(s) (Step 8.0) … 16 2-14. Identifying Causes and Contributing Factors … 16 2-15. Evaluate and Select Corrective Action for a Fleet Issue (Step 9.0) … 17 2-16. Submit to Process Owner for Further Analysis (Step 10.0) … 21 2-17. Submit Cause to Certificate Oversight Process (Step 11.0) … 22 2-18. Document and Submit Issue to External Organization (Step 12.0) … 22 2-19. Initiate AD, SAIB or other Corrective Action Process (Step 13.0) … 22 2-20. Prepare Internal Feedback to MSAD Process Owner (Optional) (Step 14.0) … 22 2-21. Prepare Lessons Learned (Step 15.0) … 22
Chapter 3. Follow On and Trending … 23
Chapter 3. Follow On and Trending … 23
3-1. Monitor and Validate … 23 3-2. Trending … 23
Chapter 4. Applying MSAD to Foreign Products … 26
4-1. Introduction … 26 4-2. Addressing MCAI … 26 4-3. Addressing Events … 27
Chapter 5. Roles and Responsibilities … 29
5-1. Process Owner Responsibilities … 29 5-2. AIR-110 Responsibilities … 29
Chapter 5. Roles and Responsibilities … 29
iii
Table of Contents (Continued)
5-3. AIR-120 and AIR-130 Responsibilities … 29 5-4. Directorate Responsibilities … 29 5-5. Responsibilities of offices responsible for COS oversight … 30
Chapter 6. Background … 31
6-1. AIR SMS … 31 6-2. Industry Interface and Applicability … 31 6-3. Benefits of the MSAD Process … 32 6-4. MSAD Tool Support … 32
Chapter 7. Exceptions … 33
Chapter 7. Exceptions … 33
7-1. Exceptions to Some Requirements of this Order … 33
Appendix A. Definitions and Acronyms … A-1
Appendix B. Administrative Information … B-1
Appendix C. Directive Feedback Information … C-1
iv
Chapter 1. General Information
1-1. Purpose of this Order. This order explains how you will use the MSAD process to analyze COS data and monitor safety in aircraft fleets. In this order, we describe the steps of the process, the tasks within those steps, and the responsibilities incumbent on all process users for all product types.
Chapter 1. General Information
1-2. Audience. We, the FAA, wrote this order for aviation safety engineers (ASE), aviation safety inspectors (ASI), aircraft certification offices (ACO*) staffs, directorate standards staffs, and all Aircraft Certification Service (AIR) and Flight Standards Service (AFS) personnel responsible for monitoring and addressing product safety risks.
*Note: For the purposes of this order, the term “ACO” is used as defined in FAA Order 8100.5, Aircraft Certification Service, paragraph 1-8(c). The term “ACO” also applies to an FAA office responsible for COS oversight.
1-3. Certificate Holder Engagement. Compatible certificate holder processes that are acceptable to the FAA ACO can be used in lieu of functions identified for FAA personnel throughout this order. The ACO, Safety Management Design and Analysis Branch (AIR-150) and the appropriate directorates should coordinate to ensure these processes provide an adequate oversight role for the ACO in ensuring that the objectives of this order are met.
1-4. Where to Find This Order. You can find this order on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/ or the FAA’s Regulatory and Guidance Library (RGL) website at http://rgl.faa.gov.
Chapter 2. MSAD Process Overview
2-1. Purpose of the MSAD Process. We designed the MSAD process to filter, review, analyze and trend aviation safety data. The MSAD process helps us identify safety issues in the in-service aircraft fleets, and identify corrective actions to mitigate safety risks across the fleet. The process also identifies other causes of safety issues that cannot be addressed by fleet (product/part) corrective actions. MSAD users should submit these causes to the appropriate organization and/or process owner (whether inside or outside AIR) for further analysis and action.
Chapter 2. MSAD Process Overview
2-2. Range of the MSAD Process.
a. We intend the MSAD process to analyze in-service data to determine corrective action for COS issues. The MSAD process covers everything from receiving data to determining fleet corrective action. Issuing the corrective action, including the notice of proposed rulemaking (NPRM) process, is outside MSAD. It is part of the airworthiness directive (AD), special airworthiness information bulletin (SAIB), and/or other FAA actions or recommendations processes.
b. Certain certificate holders have their own processes to filter, review, analyze and trend aviation safety data on their products. We expect the ACOs will continue to foster cooperative COS agreements which integrate the certificate holders’ processes with the MSAD process in a manner that is compatible with this order. In those instances, the certificate holder will accomplish many of the steps defined in this order to address the safety of their products with the ACO ASE performing an oversight role.
c. MSAD may also interface with other AIR processes, non-AIR FAA processes, and industry, to help identify non-fleet-based problems. For example, ASEs and ASIs who oversee the certificate holder may analyze product design, production, operations, and maintenance process data to identify certificate holder risks and corrective actions that should reduce aircraft fleet risks.
2-3. MSAD Overview. As an ASE following the MSAD process, you perform both a risk analysis of the potential safety issue and a causal analysis. Following the MSAD process, you initiate the AD or SAIB (or other corrective actions or recommendations) as required. When you complete the process, you store event data, safety issues, risk analysis, causal analysis and corrective action data for future use. Figure 1 is a high-level view of the entire process. Although we display and describe the components sequentially, you may encounter situations where portions of the process are worked concurrently or out of sequence.
+
I L---------
Figure 1. High-Level View of MSAD
Acquire
data
Conduct
hazard criteria analysis
Perform preliminary
risk assessment
Perform
risk analysis
Aviation
data triggers
MSAD
Identify
causes
Risk is reduced
Select corrective
action
Aviation data, Aviation data Senior COS ASE(s) ASE evaluates ASE identifies ASE works with in a standard is filtered, conducts certificate holder causes that other AIR and taxonomy, is using qualitative risk analysis or contributed to the AVS experts to collected by automation assessment to filter performs event. ASE identify and automation, and/or the data or issues quantitative risk identifies candidate evaluate provided by manually, to quick disposition analysis to product/part candidate certificate identify (Emergency ADs, identify safety solutions and corrective actions holders or event data Final rule; request issues for submits nonto reduce risk. captured by likely to for comments and corrective action. product/part cause May lead to the other identify obvious noninformation to initiation of an methods. safety issues). applicable parties. AD or nonissues. mandatory corrective action.
Monitor and validate corrective actions
2-4. The MSAD Process Flow. In figure 2, the entire MSAD process is given in fifteen steps. Subsequent paragraphs explain every step. We will expand some steps, like risk analysis, into more detail as we discuss them.
Link B Monitor &
validate (paragraph
3-1) 1.0
Event
/-------
data
Hazard 2.0
criteria
---------- --------------------------------------------
Directorate
COS Rep Criteria Hazard criteria not met ~
sampling
review
4.0 ASE 3.0 Initiate Emergency Emergency Unsafe & AD or AD or urgent .-- "Final Rule; "Immediate
START []
Automation/ D
manual entry D
Acquire data 0 E] Automation CJ Conduct hazard B criteria analysis
-----+ --+ Criteri not met …
Criter met ·········-!>
Senior COS ASE
Perform preliminary risk
assessment
Key Data: white = input shaded = output
Off page sending connector
Off page arriving connector
Decision
Task with actor
Start or end of process
Database
Process flow
Data flow
Page link flow
As required/ optional flow
MSAD record
database
request for Adopted Potentially Non-safety
NOTE: All event data, decisions, calculations, and information related to the MSAD process will be stored by event in the MSAD record database.
comment" information
Link C Trend data
(figure 7)
Guidelines
Rule" process unsafe issue
5.0 ASE
Record risk ~isk ------------------ - d t analysis results a a (figure 3)
Is risk at or above mandatory corrective action risk guideline?
(CARB Decision)
No
.. r
END
Link A MSAD (figure
2, page 2) ~----~
Figure 2. MSAD Process Flow Page 1
Note: See Chapter 3 for details on link C. See figure 3 for details on step 5.0.
-CJ
[J-
Figure 2. MSAD Process Flow - Page 2
External (non-AIR)
issues documentation
AD, SAIB
AD, SAIB
or other optional corrective
action information
ASE Initiate AD, SAIB* or other corrective action
process
7.0 8.0 ASE
Perform causal
analysis
ASE Evaluate & select
corrective action
for a fleet issue (CARB Decision) ASE
Submit cause
to certificate
oversight
process
Cause/effect
chart
AD, SAIB
or other optional corrective
action decision
6.0
12.0
14.0
13.0
15.0
ASE Prepare internal
feedback to MSAD process
owner (optional)
Feedback
ASE
Prepare COS lesson learned,
as necessary
Lessons
learned END
ASE Submit to process owner
for further
analysis
9.0
Is the safety issue cause(s)
unclear?
People/ process issue documentation
Process issue documentation
11.0 10.0
Product/part
cause? Internal AIR process or people
cause? Yes
Yes
Design or Production Certificate holder
cause?
Yes
Maintenance or operations
cause?
Yes
Cause(s) ASE
Document the
cause(s)
Link A
MSAD (figure 2,
page 1)
Link B Monitor &
validate (paragraph
3-1)
ASE Document & submit issue
to external
(non-AIR) organization
Was previous corrective action
ineffective?
Does CARB recommend
RCA?
No No
Yes Yes Yes
No
2-5. Acquire Data (Step 1.0). Data is acquired from FAA databases using an automated batch process. Acquired data can also be manually input, such as a report satisfying the requirement found in Title 14 of the Code of Federal Regulations (14 CFR) 21.3 and 183.63. In both cases, the data goes to an electronic form called the MSAD record. Not all event data will be analyzed through the MSAD process. Hazard criteria (described in paragraph 2-6) are used to filter out the events that do not present potential safety issues. However, the events that are filtered out can still be used for trending purposes.
a. Responsible Office. MSAD software sends event data to the responsible office for the applicable product-type. When event data applies to more than one product-type, MSAD software sends it to all responsible offices based on the make and model. If all offices indicate they are not responsible for the event data, the software will send the event data to AIR-150 for resolution. When the event data does not contain any product-type data that is directly traceable to an aircraft, engine, or propeller, such as an appliance, the software sends it to all offices to determine whether the part/appliance is installed on any of the aircraft for which the office has type certificate (TC)/supplemental type certificate (STC) oversight. An MSAD record will not show that it is fully “closed” until all offices involved have completed their review and taken appropriate action to close the record. Sometimes more than one office may need to remedy safety issues related to a single event.
b. Taxonomy. MSAD taxonomy is aligned with the FAA’s Commercial Aviation Safety Team (CAST) and International Civil Aviation Organization (ICAO) common taxonomy team (CICTT). MSAD taxonomy is more in-depth and detailed than CICTT because CICTT terms are currently top-level event descriptors only. MSAD supplements CICTT taxonomy with lower-level event descriptions, part name, and other details. CICTT taxonomy is still evolving. New terms and changes in hierarchy may or may not fit the MSAD application. However, we intend to keep MSAD aligned with CICTT taxonomy as it matures.
c. Event Data Sources. Sources of safety data include FAA databases focusing on event information. We expect industry sources to increase as cooperative data sharing increases. ASEs can manually create an event record from any data source. The ACO COS focal can import event data from external sources via the data portal. Automation will pull data from certain databases and submit those as event records to the cognizant ACOs and directorates based on the make and model. ACO and directorate personnel can manually enter event records from any data source. Manufacturing inspection district office (MIDO) personnel can manually enter event records associated with quality escapes. We anticipate that automation will be modified to add additional data sources.
2-6. Hazard Criteria Analysis (Step 2.0). Hazard criteria are used to automatically or manually filter event data. Filtering reduces the number of events that the senior COS ASE(s) must review for potential safety issues during the preliminary risk assessment. Automatic filtering is based on keyword, key-phrase matches, or other data mining techniques of the event data for the product type. For example, a filter might be one that sends forward any event description that includes the word “fire.” Event data that does not meet the hazard criteria is retained and can be used for trending. If senior COS ASE(s) determine that the event should not
have passed the hazard filtering criteria, then the event should be marked as a “false positive” to assist directorates in refining the hazard criteria.
Note: Automation will not filter § 21.3 reports or other safety data submitted under negotiated agreements since the agreement will typically include the hazard criteria.
a. Product Type Hazard Criteria. Each directorate standards staff must develop and maintain their own hazard criteria to filter event data about their product type.
b. Manual Sampling. Automated hazard criteria should continue to mature over time. The criteria list will be a living document. Directorate-assigned COS representatives must review, at least annually, samplings of events filtered by the hazard criteria to ensure that the criteria stay valid. The rate and frequency of events they sample may vary, based on process maturity and directorate/ACO needs.
2-7. Perform Preliminary Risk Assessment (Step 3.0). The purpose of the preliminary risk assessment is to quickly identify any safety issues needing an emergency AD or immediately adopted rule (IAR) (issued as a “final rule; request for comment”). It also determines whether an event indicates a potential safety issue requiring more investigation through the MSAD process. The assessment should take advantage of whatever data is immediately available. It is preferred to minimize single point safety decisions by doing multi-discipline assessments when manpower permits. This assessment can be performed by a group of senior COS ASEs, however any actual data analysis is usually performed by a single person. If a group assessment is performed, a cognizant senior ASE for the engineering discipline most affected by the event must participate.
a. Review by Other Offices. Here, or at any other point in the MSAD process, the senior COS ASE(s) or assigned ASE may decide that there is a cross-product issue, and that other offices need to review the information. The senior COS ASE(s) or assigned ASE must enter the rationale in the MSAD record and submit it to the appropriate office(s).
b. Questions for the Senior COS ASE(s). As the senior COS ASE(s), you must review the event to answer two key questions:
(1) Is this a potential safety issue?
(2) Does an urgent unsafe condition exist that requires immediate corrective action?
c. Preliminary Risk Assessment Answers. The answers to those two questions determine the remaining steps in the process.
(1) If you, the senior COS ASE(s), determine that there is a potential safety issue, you must define it and submit the resultant action to the assigned ASE to proceed with the risk analysis step (investigation may determine that there is negligible risk and that calculation of the risk is not required). If you are the cognizant engineer for this product, you can assign this potential safety issue to yourself.
(2) If you, the senior COS ASE(s), identify an urgent unsafe condition that requires immediate corrective action, you must define the safety issue and assign it to an ASE to initiate either an emergency AD or IAR (per step 4.0). The assigned ASE can delay the comprehensive risk analysis and causal analysis until after starting the emergency AD or IAR. Once the emergency AD or IAR starts, the assigned ASE must continue analyzing the safety issue in the risk analysis step. You or the assigned ASE may determine that an emergency AD or IAR is necessary later in the MSAD process as new data becomes available.
Note: An emergency AD or IAR can be started based on an FAA estimate that the time in which the action is required is too short to allow the time necessary for public comment. (Reference FAA Manual FAA-IR-M 8040.1, Airworthiness Directives Manual.)
(3) If you, the senior COS ASE(s), determine that no further action is necessary, you must document that determination. The event does not move any further in the event evaluation portion of the MSAD process, but is retained for trending. Prior to closing the MSAD event, you should ensure that MSAD events discovered to be false positives are marked, and that the MSAD record is routed to appropriate offices, as necessary.
2-8. Initiate Immediate Corrective Action (Step 4.0). If a safety issue is urgent, you, the assigned ASE, must start either an emergency AD or IAR per FAA Order 8040.1, Airworthiness Directives; FAA Manual FAA-IR-M 8040.1; and any directorate-specific procedures. This ensures that the risk is mitigated in a timely fashion, without waiting for the remaining MSAD process steps.
2-9. Record Risk Analysis Results (Step 5.0). MSAD risk analysis objectively characterizes hazards for probability and severity, and determines the risk posed by each hazard associated with a given safety issue. Each directorate may have particular risk measures based on their product type. If you are an assigned ASE, you are responsible for the risk analysis, and you must record the total uncorrected fleet risk and the uncorrected individual risk (per flight or per flighthour) and compare them to directorate-defined risk guidelines, as applicable, for issuing ADs or other mandatory corrective actions for the product type. If the safety issue is associated with an appliance, technical standard order (TSO) article or standard part, the safety issue could be associated with multiple product types and the assigned ASE should obtain any available information from the manufacturer to determine product applicability to support the risk assessment. In certain cases where the safety effect is different depending on the installation or when the appliance, TSO article or standard part is installed on multiple product types that use different directorate-defined risk guidelines, the ASE should assign the MSAD records to the cognizant TC or STC oversight office to perform the risk analysis and subsequent follow-on actions. The risk guidelines for AD or other mandatory corrective action assist the CARB in determining if mandatory corrective action is warranted. The risk analysis process (step 5.0) is defined in the next paragraphs, and is illustrated in figure 3.
From perform preliminary risk assessment (figure 2,
page 1, step 3.0)
Additional information (as necessary)
ASE Does safety issue require ASI involvement
(AFS or MIDO**)? Identify No potential
~ --'--.i outcomes*
for hazard
under
study Yes
ASE
Solicit support from cognizant ASI
I 1--'
I I I I I I I I I
I I I I I I I I I
Potential outcomes
If all outcomes
are negligible
~
For each non-negligible
outcome*
ASE
Determine probability of outcome
ASE
Determine severity of
outcome
ASE
Calculate risk values of outcome
Risk information for outcomes
ASE
Proceed to obvious cause
determination (figure 2, page 2,
step 6.0)
ASE
Record risk
values of
issue
Recommend
type of corrective
action to
CARB ____ J
Risk values
of issue
No further
action §)
Direc orate
risk guidelines
*A single event may have multiple undesired outcomes. This should be accounted for with appropriate conditional probabilities. **includes AEG, FSDO, CMO and MIO
Figure 3. Record Risk Analysis Results Flow Diagram
a. Identify ASI Support. If you, the assigned ASE, need manufacturing, maintenance or operations ASI support, you should contact an ASI responsible for the product and ask for information to support the risk analysis. Those ASIs may come from an aircraft evaluation group (AEG), flight standards district office (FSDO), certificate management office (CMO), MIDO and/or manufacturing inspection office (MIO).
Note: When ASEs solicit information from flight standards ASIs in the field (i.e., FSDO or CMO), ASEs should either obtain the information through the cognizant AEG or notify the AEG that information is requested directly from the field.
b. Identify Potential Outcomes. You, the assigned ASE, must identify and document all important potential outcomes for the potential safety issue you are studying for further risk analysis. If you determine that the risk of an outcome is obviously negligible, there is no need to calculate or document it.
c. Calculate the Risk Value of Each Outcome. You, the assigned ASE, must use the directorate-specified product-specific risk analysis method (resulting in units convertible to fatal accidents) to calculate the quantitative probability, severity, and risk value for each important outcome. Calculate and record these risk values:
(1) Total uncorrected fleet risk;
(2) Uncorrected individual risk (per flight or per flight hour); and
(3) Time until control program risk guideline is reached (Not required if (1) and (2)
are both below directorate risk guidelines).
d. See table 1 for detailed descriptions of risk values. You, the assigned ASE, must attach the documents produced that support probabilities, severities and risk values for total uncorrected fleet risk and uncorrected individual risk (per flight or per flight hour) to the MSAD record containing the event information.
Table 1 – Risk Value Definition, Purpose, and Mathematical Basis
Risk Value Definition Purpose Mathematical Basis
Total Predicted risk Provides future risk if no Computed as the product of the uncorrected expected, over corrective action is taken. average severity and average fleet risk remaining life of affected fleet, if no corrective action is taken.
Helps determine if an unsafe condition may exist in future. Used to guide decisions for corrective action.
per-flight (or flight-hour) probability of occurrence, multiplied by the exposure (flights or flight-hours) remaining in affected fleet life. If known voluntary compliance to an existing service document is supported by data, then you can account for the existing control actions by adjusting the affected fleet numbers.
Risk Value Definition Purpose Mathematical Basis
Uncorrected Predicted risk per Needed for cases of low fleet Typically based on averages individual flight or per flightexposure that result in the that apply to the fleet. risk hour. total uncorrected fleet risk, as However, there may be defined above, to be circumstances where you can acceptable while the risk to calculate individual risk an individual aircraft or including risk values for special person is unacceptable. conditions and combinations of Helps determine if an unsafe conditions, or for subsets of the condition may exist in future. fleet, for example by model or Used to guide the decision for usage. corrective action.
If only a subset of the fleet is subject to the risk, include only that portion in the analysis. Evaluate significant variations between identifiable subsets of the fleet (different models, different usages, etc.) as separate populations for the individual risk.
Time until Amount of time from Provides information to assist The time period when risk control when the need for in risk management planning, accumulates in the fleet to a program risk corrective action is i.e., how much time is value that equals the control guideline is determined by the available to determine root program risk guideline (Note: reached CARB to the time cause, develop service Control program risk guideline when the fleet would information, coordinate and is discussed further in section exceed the control process corrective action, and 2-15.c.). program risk incorporate the corrective guideline if no action action in the fleet while was taken. With the staying within the risk agreement of the guideline. See Figure 4 corrective action review board (CARB), this guideline may be exceeded in cases of
On occasion the plan to issue corrective action material by a date that meets this guideline may fail to be achieved (e.g., the planned modification was severe resource availability or service disruption and there is no practical interim measures to reduce risk.
discovered late in the process to not work). In those cases do not shorten compliance times to completely make up for the lost time; it is acceptable to recalculate the time until the program risk guideline is reached and start anew. However, give the issue extra supervision and emphasis to avoid a repeat and work to issue the corrective action material before the new date by as much as can be reasonably accomplished.
-
Figure 4. Corrective Action Timeline
Fleet Safety Corrected Determination Date Date
Control Program AD Published Risk Guideline is reached Safety Root causes determination and determined, all Control program service incorporation time information Corrective action accepted at CARB developed and AD buffer time worksheet submitted
Time Until Control Program Risk Guideline
is Reached
e. Risk Analysis Requirements. You, the assigned ASE, must calculate the safety issue’s risk values as specified in table 1 for each outcome in measures defined by the product directorate and record them in the MSAD record. Additionally, when calculating total uncorrected fleet and uncorrected individual risk, the units must be convertible to the number of fatal accidents for comparative purposes, whether or not an uncorrected risk guideline exists. You must calculate the time until control program risk guideline is reached if either conditions (1) or (2) from paragraph 2-9c are above directorate risk guidelines.
(1) To calculate the risk values of the issue, you must:
Evaluate the risk based on the directorate’s methods and guidance for the particular product type;
Document the assumptions, methods and other supporting information describing how the probability and severity were determined;
Work with the applicable product directorate to gain conditional probability data that the directorate may compile as risk analysis “library data” for its products (e.g., hazard ratios and injury ratios).
(2) Some ACOs have negotiated agreements with certain certificate holders such that the certificate holder performs the risk analysis on behalf of the FAA. In those cases, you
may review the certificate holder risk analysis per the negotiated agreement to verify that the risk analysis meets the objectives of this section.
f. Determine Necessary Action. To assist the CARB in determining the type and/or need for corrective action (mandatory, non-mandatory or no action), you, the assigned ASE, must compare the risk values calculated for the safety issue against the product directorate’s risk guideline(s) for ADs or other mandatory corrective action. (The directorate’s minimum requirement is to provide an uncorrected individual risk guideline). See figure 5, Risk Guideline Diagram.
Figure 5. Risk Guideline Diagram
Optional Corrective
Action
Negligible Benefit as a Result of Action
Safety Improvement
Range (SAIB or Other Non-
mandatory Optional
Corrective Action)
AD Range
Risk Guideline
Airworthiness
Directive
CARB Required Possible Risk Values
2/3 Guideline
2-10. CARB.
a. You, the assigned ASE, must present your recommended action, along with your risk analysis, causal analysis and quantitative evaluation of the risk reduction of corrective actions to the CARB for concurrence. In some cases, a safety issue may go to the CARB several times before a final corrective action is selected. For example, the assigned ASE can elect to bring a given safety issue to the CARB prior to completing all analyses to provide awareness and obtain preliminary feedback. The goal of the CARB is to improve safety through better decision making. This is achieved by:
(1) Improving robustness by reducing the number of single thread safety decisions.
(2) Providing for cross-functional review, allowing others to raise concerns and contribute knowledge about a safety issue and proposed corrective action plan.
(3) Facilitating real-time, open exchange of safety issues across the key lifecycle disciplines among ACO, MIDO and AEG staff.
(4) Providing a forum for the review of the preliminary risk analysis, risk analysis, causal analysis and corrective action(s) for a product type, leading to acceptance, rejection or revision of the proposed corrective action.
(5) Increasing knowledge and experience in the AVS community.
Note: CARBs are designed to precede the AD process, not replace it. CARB actions ensure a complete data package is submitted to the AD process.
b. CARB Applicability. All safety issues with a risk above two-thirds of the risk guidelines for AD or other mandatory corrective action must be reviewed by the CARB. All ADs or other mandatory corrective actions must be reviewed by the CARB regardless of risk. Any unilateral corrective action considered on foreign products and technical no action required (NAR) decisions involving mandatory continuing airworthiness information (MCAI) must always be reviewed by the CARB. Emergency ADs and IARs can initially bypass a CARB review. However, after immediate actions are taken, the safety issue must go through the rest of the MSAD process, including the CARB. In rare cases where a CARB would be significantly delayed by lack of members and alternates, and where we must issue an AD, SAIB or other corrective action without further delay, the applicable ACO may allow a temporary bypass of the CARB. The CARB must review the corrective action during the next meeting. The CARB meeting minutes must be documented, and include attendees, subjects discussed, and decisions made.
c. ACOs Select CARB Participants. The ACO manager is responsible for selecting and assigning representatives to CARBs, and for designating alternates for unavailable representatives. The ACO manager should coordinate with MIDO and AEG management to facilitate MIDO and AEG representation. The CARB should consist of the following personnel:
(1) ACO or responsible office manager.
(2) ASE or pilot assigned and presenting the safety issue.
(3) At least three other ASEs, one with experience in the safety issue and two others that support CARB technical discipline diversity. This can be satisfied using program or branch managers with the appropriate experience.
(4) Representation from the AEG and MIDO/CMO.
(5) Other optional FAA representatives (such as standards staff, chief scientist and technical advisor, etc.) for a range of inputs to the safety issue corrective action.
d. CARB Attributes. The CARB should:
(1) Permit open discussion and not suppress dissenting technical opinions;
(2) Convene regularly as determined by the ACO manager (or delegate) to ensure timely review of safety issues and enable participants to attend;
(3) If necessary, share resources across organizations and offices;
(4) If necessary, use Web cast conferences and teleconferences to support the proper mix of expertise; and
(5) If possible, reach consensus on a safety decision. In cases where consensus cannot be reached, the ACO or responsible office manager has final decision authority.
e. Factors Not Related to Risk Analysis. In rare situations, the ASE or FAA management may, based on factors unrelated to the risk analysis, make recommendations not consistent with risk guidelines for ADs or other mandatory corrective actions. The decision to accept or reject these recommendations is made during the CARB. When this happens, the assigned ASE documents final decisions in the MSAD record. Factors not related to risk analysis must not be used to influence the objectivity of risk analyses as the risk analysis is intended as objective input into the decision.
2-11. Safety Issue Cause(s) (Step 6.0). As shown in figure 2, the MSAD process identifies and mitigates product risk from safety issues. The goal is to determine the product-related cause(s) for the safety issue. If the causes are unclear, you, the assigned ASE, must conduct a structured causal analysis.
a. A “structured” causal analysis uses a tool such as Apollo Root Cause Analysis™ and the companion RealityCharting® Software.”
b. A structured causal analysis is not required for issues with obvious causes and clearly identifiable fleet solution(s) unless directed by the CARB, or in cases where previous corrective action(s) were not effective. Fleet solutions include inspections, re-designs, limitations and/or other product/part corrective actions.
c. If the causes are obvious to you based on engineering expertise and judgment, you must still document the causes based on your non-structured approach. Documenting all of the causes is an important step to support future trending activity and quick identification of systemic problems when causes reoccur.
d. In addition, consider doing a structured causal analysis in all cases for complex and/or high-profile safety issues. The structured approach supports your assumptions and conclusions during the process, and guides the documentation of the cause(s), effect(s) and the causal analysis report.
2-12. Perform Causal Analysis (Step 7.0). Safety issues where causes are not obvious and/or product/part corrective actions are not easily identifiable require a structured method to identify causes. The MSAD process prescribes this structured causal analysis approach. When you perform a structured causal analysis, you trace the chain of events, identify contributing factors and develop a list of candidate solutions.
a. Focus on identifying the part or product causes that can be addressed using a “fleet” corrective action (AD, SAIB or other optional corrective action).
b. You may also identify other causes that contributed to the event. These “contributing factors” may include design, manufacturing, operations and maintenance failures and may have surfaced from “people” and/or “process” issues in a manufacturer, designer or operator’s organization. Causes may also include FAA process shortfalls. You should submit the causes to the appropriate organization for their review and corrective action. You should contact the cognizant ASI when manufacturing, maintenance or operations issues are identified in the causal analysis.
c. Some ACOs have negotiated agreements with certain certificate holders such that the certificate holder performs the causal analysis on behalf of the FAA. In those cases, you must selectively review the causal analysis per the negotiated agreement to verify that it meets the objectives of this section.
2-13. Document the Cause(s) (Step 8.0). Document the causes in the MSAD record using the causal taxonomy. MSAD requires you to document the output of a causal analysis, including at least:
a. A problem statement (may be similar to the defined safety issue);
b. Product/part causes;
c. People/process causes, also called “contributing factors,” if applicable; and
d. A causal analysis report (for structured causal analysis only), which is typically a document produced by a causal analysis tool in a standard format.
2-14. Identifying Causes and Contributing Factors.
a. Causal analyses may identify contributing factors that can influence a part- or system-level failure. Since contributing factors are not always addressed by ADs or SAIBs, you, the assigned ASE, should submit these factors to the appropriate organization for analysis and possible action.
b. If you, the assigned ASE, identify that outside MSAD, an operational, maintenance or manufacturing process is contributing to a safety issue, you should send your analysis results and safety issue information to the appropriate organization for review and action (e.g., AEG,
MIO, MIDO, etc). You should follow-up to ensure that the organization understands and has sufficient information to address the safety issue.
2-15. Evaluate and Select Corrective Action for a Fleet Issue (Step 9.0). You, the assigned ASE, based on the cause(s) identified in step 7.0, must identify candidate corrective action(s) and select the appropriate one(s) to reduce the fleet risk presented by the safety issue.
a. Identify Candidate Corrective Actions (CCAs). Actions can range from initial mitigating to extensive final and terminating. You, the assigned ASE, must evaluate each CCA for its appropriateness and timeliness to mitigate the safety risk. Corrective actions typically are developed by certificate holders. Certificate holders typically submit these to the FAA and the FAA has the option of accepting, rejecting or developing alternative corrective action(s). When a certificate holder does not submit corrective action(s) for a concern, the FAA must develop necessary corrective action(s) to mitigate the risk to an acceptable level. CCAs can include:
(1) Inspections;
(2) Part repairs or replacements;
(3) Modification/kit installations;
(4) Limitations;
(5) Rework; and
(6) Process/procedure changes.
b. Determine the Corrective Action Vehicle.
(1) You, the assigned ASE, must calculate CCA control program fleet and individual risk as defined in table 2 using paragraph 2-9.e.(1).
(2) If new information becomes available late in the control program development that shows that the risk was much greater than first calculated, or if there was a significant delay in issuing the control program due to some unforeseen issue, you may find that the intended control program will now exceed the guidelines. In that case, the control program risk may be evaluated using the exposure from the current point in time (instead of since when the need for the control program was identified in paragraph 2-9 f., as described in table 2 on the next page). Care should be taken to avoid getting into these situations. It should be the norm to use the exposure since the need for the control program was identified in paragraph 2-9 f., and resetting the clock should be an exception.
(3) If you decide not to follow the recommended corrective action, you must document your decision and what you based it on. Attach supporting documentation to the MSAD record.
c. Initiate Corrective Action. AD, SAIB and other corrective action processes are outside MSAD. They are defined in appropriate orders and the quality management system (QMS) process. Once the AD or SAIB is issued, the assigned ASE or other administrative personnel enters the corrective action information (AD number, SAIB number or other applicable information) into the MSAD record.
Table 2 – Control Program Risk Value Definition, Purpose, and Mathematical
Basis
Risk Value Definition Purpose Mathematical Basis
Control program fleet risk
Risk within affected fleet while corrective action is taken (plus any residual risk not remedied by corrective action).
Helps risk managers evaluate candidate corrective actions against a maximum allowable risk value with respect to effectiveness and timeliness.
Computed as the product of the average severity and average per flight or per flight-hour probability of the occurrence, multiplied by the control program exposure (predicted number of flights (or flight-hours) for the fleet during the time taken to accomplish the corrective actions). The start of the control program is normally considered to be when the need for the control program was identified in paragraph 2-9 f., i.e., the risk typically includes the exposure since that time—it includes corrective action preparation time and AD flow time, as applicable. If actual corrective action incorporation rate is unknown, estimate corrective action flights or flighthours by using estimated time for AD issuance plus half the AD compliance time.
Control Predicted risk Needed for cases of low fleet Typically based on averages that program per flight or per exposure that result in the control apply to the fleet during the individual risk flight-hour during the control program.
program fleet risk, as defined above, to be acceptable while the risk to an individual aircraft or person during the control program is unacceptable. Helps risk managers evaluate candidate corrective actions against a maximum allowable risk value with respect to effectiveness and timeliness.
If only a subset of the fleet is subject to the risk, include only that portion in the analysis. Evaluate significant variations between identifiable subsets of the fleet (different models, different usages, etc.) as separate populations for the individual risk.
control program. However, there may be circumstances where you can calculate individual risk including risk values for special conditions and combinations of conditions, or for subsets of the fleet, for example by model or usage.
From causal analysis/ document causes (figure
2, page 2, steps 7 & 8)
r---------------------------------------, I I
Repeated for each candidate corrective action option
ASE Identify individual or bundles of CCAs for
consideration
ASE
Solicit AFS & ,-------i MIDO input for
evaluation (if needed)
ASE Calculate -~-- CCA control
Mandato~
I I I I I 1 ____ 1 __
I I I I AD proposal :
I I I B
CARB = corrective action review board CCA = candidate corrective action CPRG = control program risk guideline
program risk
values
Does CCAoption
meet the
CPRG?
No
ASE
Yes
revised to be
at or below
CPRGor
flagged
Revise, delete, or flag
CCA as needing consideration at the
CARB
•Other = No action, or SAIB, or other optional corrective actions . .. NOTE: May include validation of a single available corrective action or no action.
CCA Evaluation
ASE
Ability to reduce risk
ASE
Material, labor
& opportunity
cost
ASE
Timeliness of implemention
ASE
Complexity of implementation
ASE
Select preferred CCA(s)**
Selected corrective
action
Proceed to AD
or other* process, if applicable
ASE
Conduct CARB***
CARB decision information
••• All safety issues with a risk above two-thirds of the risk guideline or if recommending mandatory action must be reviewed by the CARB.
Figure 6. Evaluate and Select Corrective Action(s) Flow Diagram (expanded
illustration of step 9.0 in figure 2)
c. Evaluate AD CCAs using the Control Program Fleet Risk and Control Program Individual Risk Guidelines (CPRGs). This applies to AD CCAs only. Skip this task if you are proposing non-mandatory corrective action. You, the assigned ASE, should ensure that the CCA (or combination of corrective actions) calculated control program risk is at or below both the fleet and individual CPRG.
(1) During this analysis, consider combined actions of a “bundle” of CCAs as a whole, for example interim action such as a repetitive inspection followed by a final action, such as a part replacement. Evaluate the corrective action plan for the total effect on the risk.
(2) If the risk of a CCA exceeds either fleet or individual CPRG, consider eliminating or revising the candidate by either accelerating the implementation (e.g., replacing at ‘B’ check rather than at ‘C’ check, or “inspect at 100-hr vs. 200-hr intervals”) and/or adding/modifying corrective actions. Use the directorate-established CPRG analysis method to
determine the action’s acceptability and timing by comparing it to the control program fleet and control program individual risk guidelines.
Note: When considering compliance times for mandatory corrective action, do not unnecessarily extend the compliance time even if doing so would keep the control program fleet or control program individual risk below the CPRG. Work within existing maintenance schedules.
d. CCA Evaluation. You, the assigned ASE, must evaluate each candidate corrective action. Ideal candidates are inexpensive, easy to perform, implemented quickly, 100 percent effective at reducing risk and do not introduce substitute risk (risk of unintended consequences). Most situations do not meet these ideals. Therefore, you must conduct a short evaluation of candidate action(s) using effectiveness, cost, timeliness of implementation and complexity.
e. Select Preferred Corrective Action. Once you, the assigned ASE, have evaluated all candidate corrective actions against the attributes in paragraph 2-14.d., select the most appropriate one(s), balancing the attributes. You must document and submit your recommendation with all supporting documentation for review by the CARB.
f. Interim Corrective Action. When issuing interim corrective action, you must add a date and associated comment field to the MSAD record. This date will act as a reminder that the final action still needs to be issued in time to meet the control program risk guideline.
g. Terminating Corrective Action. When the terminating action is defined, you must calculate the control program fleet and control program individual risk to ensure it meets the fleet and individual CPRGs. Proceed through the corrective action selection process as defined in paragraph 2-14.e. The CARB must review terminating actions not previously discussed in the initial CARB.
Note: For some safety issues the time needed to implement the initially proposed control program may exceed the control program incorporation time. This could occur due to the proposed control program requiring unusually extensive engineering and/or testing, unusually long time to produce parts, or unusual shop capacity constraints in installing the proposed corrective action. To effectively manage risk in those cases, an interim action should be taken prior to the initially proposed corrective action (final action). The control program risk of the combination of the interim action and the final action should be within the control program risk guideline.
2-16. Submit to Process Owner for Further Analysis (Step 10.0). As described in figure 2, assigned ASEs discovering causes identified in other AIR business processes (like certification and rulemaking) should communicate those causes to process owners for action.
2-17. Submit Cause to Certificate Oversight Process (Step 11.0). Assigned ASEs who identify causes in the certificate oversight process, like design and production escapes, should communicate them to the certificate oversight representative for action.
2-18. Document and Submit Issue to External Organization (Step 12.0). AFS, air traffic and other non-AIR FAA staffs may receive causal information from the MSAD process that identifies a specific condition in their business process or the certificate holders they oversee. The condition may warrant corrective action, as determined by their business process.
Note: When assigned ASEs submit information to flight standards ASIs in the field (i.e., FSDO/CMO), the assigned ASEs should either submit the information through the cognizant AEG or, at a minimum, notify the AEG that information has been submitted directly to the field.
2-19. Initiate AD, SAIB or other Corrective Action Process (Step 13.0).
a. The CARB should use the risk analysis outputs to guide its decision whether or not to choose an AD, SAIB or other corrective action. If the CARB selects any of these options, the assigned ASE starts the corrective action process.
b. Developing and issuing corrective actions may require exchange of information and further MSAD process analysis. You must use the MSAD process to track changes to the technical decision-making.
2-20. Prepare Internal Feedback to MSAD Process Owner (Optional) (Step 14.0). MSAD is part of the AIR QMS process. AVS MSAD users can submit feedback to the MSAD QMS procedure and associated work instructions by submitting a corrective action request (CAR), preventive action request (PAR) or nonconformance record (NCR) through AVS QMS Information Technology Support (QMITS) system.
2-21. Prepare Lessons Learned (Step 15.0). Either a senior COS ASE or assigned ASE should consider all safety issues to determine if the events, safety issues, risk analyses or corrective action selections are valuable teaching cases. If so, use the “Lessons Learned Nominations” function in the MSAD tool to capture the lessons.
Chapter 3. Follow On and Trending
Chapter 3. Follow On and Trending
3-1. Monitor and Validate. The cognizant ASE must monitor and validate the effects of corrective action in the fleet by monitoring in-service data to ensure that the risk has been properly mitigated (see link B, figure 2). You can do this by identifying repeat and similar events using the MSAD record database. You must also watch for introduction of substitute risk due to unintended consequences of mitigating action.
3-2. Trending.
a. Data trending is defined as collecting and monitoring existing data to identify items that meet specific criteria or exceed established guidelines. Trending data is important because it:
(1) Enables tracking known safety issues to ensure that their rate of occurrence does not cause risk to exceed established guidelines, and is consistent with the intent of the certification assumptions and analyses.
(2) Allows the monitoring of the results of implemented corrective actions. We do this to verify that the implementation and results are as presumed, and that new problems were not introduced by the actions.
(3) Identifies emerging safety issues, which are challenging to implement. Items may appear to be an issue, but very few would likely result in an accident if uncorrected. Therefore, you have to carefully decide on which data to trend and act.
b. Conducting Trending. Trend analysis can be conducted by all AIR personnel responsible for monitoring and addressing product safety risks, as needed. Although MSAD trending is primarily based on fleet level events, it does not prevent us from trending at the certificate management level, looking for trends in people/process causes. MSAD event records can initiate trending, or management can assign it periodically. If the assigned ASE identifies a potential safety issue (see link C, figure 2), the ASE must perform a risk analysis per paragraph 2-9. If an ASI identifies a potential safety issue, the ASI must submit that information to the cognizant ASE, who will assess the fleet risk and take appropriate corrective action, as necessary.
c. Identifying Trends. Trending activities can include:
(1) Identifying items to trend (parts, products, failures, etc.);
(2) Analyzing cross-product trends;
(3) Tracking trends and items of significant interest;
(4) Tracking repeat events – similar failures that have occurred on multiple occasions, including:
Repeat part failures (within makes, models and series or across them);
High part replacement rates (within makes, models and series or across them); and
Repeat safety issues (within makes, models and series or across them).
(5) Identifying causes (during MSAD process);
(6) Identifying most common part category or system failures; and
(7) Identifying patterns or potential correlations (for example, when part A fails and part B fails, then event C occurs).
Note: Not all events necessitate a trend analysis. You should focus on anticipated concerns.
d. Figure 7 illustrates the trending process.
On-demand
trending
ASE
Enter search
criteria
Automation
Search for
potential
trend
Start trending
process
MSAD record
database
Continuous
trending
ASE
Establish trend criteria
Automation
Continuous
search for
trends
-------------------------,
'
Trend criteria
(items of
interest)
Potential
trend information
Trend identified
Trend identified
Potential
trend information
Potential
safety
issue
ASE View and store results
Does trend information indicate No
a potential unsafe
condition?
Yes
' ASE/ASI Submit trend information to MSAD for evaluation (Trend data link C, Figure 2)
Initiate/continue continuous trending?
Recommend different action?
Yes
ASE
Document different
action and pass to
appropriate party
No
No
Yes
Figure 7. Data Trending Process Flow Diagram
Chapter 4. Applying MSAD to Foreign Products
Chapter 4. Applying MSAD to Foreign Products
4-1. Introduction. This chapter describes how MSAD applies to safety issues on products designed and manufactured outside the United States. It covers how to handle events on these products, as well as how to review and disposition MCAI.
4-2. Addressing MCAI. MCAI are documents issued by other State of Design Authorities (SoDA), following ICAO Annex 8, regarding unsafe conditions on products designed or manufactured in other countries. FAA Order 8040.5, Airworthiness Directive Process for Mandatory Continuing Airworthiness Information, offers general guidance on MCAI.
a. If you are responsible for addressing the MCAI, follow the process in Order 8040.5 for all MCAI received. If you determine that unilateral action, as defined by Order 8040.5, may be necessary, enter the information into the MSAD database, creating an initial MSAD record. You then apply the MSAD process steps, beginning with risk analysis per paragraph 2-9. When requesting more technical information from the SoDA, specifically request information needed to perform the MSAD risk analysis step. This also applies when you believe that an MCAI does not represent a safety issue and a corresponding AD need not be issued. This does not apply when no AD required (non-technical NAR) decisions are made for administrative reasons. Typical administrative reasons may include instances when the SoDA has issued a revised or superseding MCAI and the FAA decides no AD is required for the initial or preceding MCAI, and instead writes an AD for the later MCAI.
b. For an illustrated version of how to apply MSAD to MCAIs, see figure 8.
MCAI
Data
START
ASE
Receive
MCAI
ASE
MCAI Process
(Per Order
8040.5)
Unilateral action
considered?
Yes
Proceed to perform risk analysis
(figure 2, page 1, step 5.0)
No
Addressing Events. Some events on foreign products will pass through the MSAD hazard
Figure 8. MSAD Process for MCAI
4-3. criteria filter for a product type, and will need to be reviewed by the responsible FAA office. If you are responsible for a foreign product, review and disposition all events associated with that product using the MSAD process for foreign product data described in this section. Coordinate with the SoDA as needed, ensuring they are aware of any safety issues in order to address them.
a. Hazard Criteria Analysis. Foreign product data will be automatically filtered per paragraph 2-6.
b. Perform Preliminary Risk Assessment. If you have identified a safety issue requiring emergency corrective action or a potential safety issue, coordinate the MSAD event record information, along with the results of the preliminary risk assessment, with the SoDA to determine whether they intend to address the safety issue with an MCAI, are in the process of preparing an MCAI, or have already issued an MCAI. If an FAA AD exists that adequately addresses the safety issue, link the MSAD record item for the event to the FAA AD and reference it for future management of the issue. If you determine that no further action is necessary, close the MSAD record. No further action is required in the MSAD process, but the record is retained for trending. Prior to closing the MSAD event, you should ensure that MSAD events discovered to be false positives are marked. For cases when immediate action is warranted, after you notify the SoDA, you
( __ · _ )
should initiate FAA corrective action without waiting for the SoDA to take action or for the eventual MCAI.
c. If the SoDA notifies us that they are taking no action or you determine that the SoDA action is not adequate, you will need to continue through the MSAD process from “Perform Risk Analysis,” per paragraph 2-9, to determine what further action, if any, is necessary. You may need to take unilateral action following the process described in Order 8040.5.
d. Figure 9 is a depiction of the MSAD process flowchart including other SoDAs.
Figure 9. MSAD Process for Foreign Product Data
Proceed to perform risk analysis
(figure 2, page 1, step 5.0)
START
Automation/ manual entry
Acquire data
Hazard Criteria
ASE
Inform SoDA (that corrective action
may be warranted)
Is SoDA
action adequate?
Yes
Yes
No
Automation
Conduct hazard
criteria analysis
Potential safety issue?
Does an FAA AD address this
issue?
No
END END END
END END END
No Yes
Senior COS ASE
Perform preliminary
risk assessment
Chapter 5. Roles and Responsibilities
5-1. Process Owner Responsibilities. AIR-150 will organize inter-directorate meetings at least annually to review and discuss MSAD information and lessons learned. The agenda may include MSAD process support--make/model lists, hazard criteria, risk analyses methods and risk guidelines--which the directorates are responsible to develop and maintain. The agenda may also include any cross-product issues identified as part of the MSAD process. These meetings can be combined with other meetings, and must be documented in meeting minutes.
Chapter 5. Roles and Responsibilities
5-2. AIR-110 Responsibilities. The Engineering Procedures Office (AIR-110) develops and standardizes regulations, national directives, policy, procedures, and advisory material for COS. AIR-110 will work with AIR-150 to ensure MSAD supports the COS activities in AIR.
5-3. AIR-120 and AIR-130 Responsibilities. The Technical Programs and Continued Airworthiness Branch (AIR-120) and the Avionic Systems Branch (AIR-130) will have access to the MSAD data to determine if the TSO process or specific TSOs need changes to address emerging safety issues in the aviation fleet. Both branches will also participate in the interdirectorate meetings described in paragraph 5-1.
5-4. Directorate Responsibilities. Each directorate is responsible for developing and maintaining the supporting MSAD processes for their product type.
a. Develop Hazard Criteria. Directorates will develop a list of hazard criteria by product type and perform sampling per paragraph 2-6.b.
b. Develop Risk Measures and Risk Guidelines. Directorates will develop risk measures, risk guidelines and specify risk analysis methods.
c. Establish Conditional Probabilities. Directorates will establish for their product type (with assistance from ACOs) conditional probabilities of aircraft-level outcomes given a base event occurrence.
d. MSAD Process Maintenance.
(1) Directorates must support the inter-directorate meetings described in paragraph 5-1.
(2) Directorate COS representatives must review hazard criteria samples (at least annually) and update hazard criteria as needed for their product type concerns. The results of this review should be recorded.
(3) Each directorate, coordinating with industry, may adapt their risk analysis methods and risk guidelines, as needed.
(4) Directorates must determine how often their conditional probability data is reviewed and revised.
5-5. Responsibilities of offices responsible for COS oversight. These offices must:
a. Perform or oversee, per ACO/certificate holder agreement, the MSAD process consistent with the product type.
b. Develop and maintain fleet size and age estimates (cycles or hours) for each model, or review and accept estimates supplied by the cognizant certificate holder.
c. Develop and maintain model utilization rates, or review and accept estimates supplied by the cognizant certificate holder.
d. Develop or review other basic data needed for risk analysis.
e. Support the directorates as necessary in accomplishing their responsibilities.
Chapter 6. Background
Chapter 6. Background
6-1. AIR SMS. The AIR SMS vision states, “AIR manages safety through a comprehensive systems safety approach, maximizing our value to aviation safety through influence and response to the changing aviation environment.”
a. AIR SMS supports and aligns with AVS SMS requirements in VS 8000.367, Aviation Safety (AVS) Safety Management System Requirements, by establishing the MSAD process, which provides a safety risk management and safety assurance approach to AIR's COS mission. To achieve the AVS SMS vision and supporting goals, AIR decision-making processes need to evolve into an approach that is both quantitative and risk-based.
b. A mature AIR SMS will give AIR a holistic approach using risk-based processes to support an enhanced focus on safety. SMS represents the first time AVS and AIR have attempted to manage risk throughout the product life cycle. Before this, much of FAA risk analysis had been an isolated qualitative activity, lacking standard methods for analyzing and managing risk over time.
c. FAA Order 8040.4, Safety Risk Management Policy, establishes our safety risk management policy and requires all FAA lines of business to establish and implement a formal risk management program consistent with their role in the FAA. The order states: “The FAA shall use a formal, disciplined, and documented decision-making process to address safety risks in relation to high-consequence decisions affecting the complete life cycle.”
d. AIR SMS addresses the requirements in Order 8040.4. After setting up an FAA SMS that uses risk-based decision making tools and processes, we expect that industry will develop or coordinate a corresponding SMS compatible with our model and tailored to each company’s needs.
6-2. Industry Interface and Applicability. MSAD effectiveness relies heavily on in-service data from operators, manufacturers and other certificate holders. This data sharing approach is risk-based, reflecting the SMS principle of being proactive by using data-driven analysis.
a. Accordingly, we have to develop MSAD interfaces with industry to foster data sharing. We expect that AIR offices, including headquarters, directorates and ACOs, will promote good working relationships with their respective industry stakeholders.
b. In addition, since certificate holders are routinely responsible for developing corrective actions for product or part hazards in the fleet, we should harmonize industry and FAA processes as much as we can. Harmonized processes promote common understanding of the fundamentals of COS: data analysis, hazard identification, risk analysis methods, risk guidelines, causal analysis and appropriate corrective actions. MSAD is defined so that MSAD process steps, except the decision and issuance of an AD or SAIB, could be performed for us by industry when potential future SMS regulations that are applicable to certificate holders are in place. Some ACOs are working towards, or have in place, negotiated agreements with their certificate holders in which the certificate holder goes beyond statutory requirements and performs data analysis, hazard identification, risk analysis, causal analysis and corrective action
development in accordance with the product directorate’s MSAD objectives. In these cases, the FAA plays a critical oversight safety assurance role, as opposed to a safety risk management role.
6-3. Benefits of the MSAD Process. MSAD and its supporting tools and methods allow the current AIR COS process to evolve into a more risk-based, systemic, decision-making system. MSAD:
a. Identifies safety issues and related causes that determine product-related corrective actions.
b. Submits underlying process and people safety issues to certificate management, rulemaking and other business processes to enable a more complete aircraft life-cycle COS environment.
c. Enables consistent standardized measurable risk-based decision making for COS across AIR.
d. Quickly identifies safety trends using analysis of dependent variables (MSAD database).
e. Builds a COS model that exemplifies what the FAA would like to see from industry in the future.
6-4. MSAD Tool Support.
a. This order defined MSAD minimum process requirements, not tool requirements. The MSAD database, however, is a tool used to store event data according to MSAD-required taxonomy. An assigned ASE conducts the MSAD process using the database and other support tools to:
(1) Access, collect and store safety information;
(2) Guide users through the process;
(3) Track completion of the process steps; and
(4) Support risk, causal and corrective action analyses.
b. The database is intended to store all information input by the user, including calculations, analyses performed and technical decisions made for the MSAD record.
Chapter 7. Exceptions
Chapter 7. Exceptions
7-1. Exceptions to Some Requirements of this Order. This chapter defines some specific situations where exceptions to some of the requirements of this order are authorized, and the extent of those exceptions.
a. When certain higher-level policies require that an AD be written, do not use MSAD risk analysis results in making the decision to write the AD. The policy decision to write ADs in those cases was made during the course of rulemaking or other policy deliberations, and overrides the MSAD risk guidelines. The following are the limited authorized cases of an overriding higherlevel policy:
(1) ADs for mandatory modifications required by the widespread-fatigue-damage rule (75 FR 69746, November 15, 2010, or later revision).
(2) ADs to mandate the incorporation in the airworthiness limitations section of the maintenance manual new or revised damage tolerance inspections or safe-life limits required for § 23.571, 25.571, 27.571, 29.571, 33.14, and 33.70 compliance.
(3) ADs that are needed for Special Federal Aviation Regulation (SFAR) 88 compliance.
(4) ADs required by the aging aircraft program (69 FR 45936, July 30, 2004 and 70 FR 5518, February 2, 2005, or later revisions) for changes to supplemental structural inspection programs and corrosion prevention and control programs (CPCP).
b. ADs are issued to correct an unsafe condition in an aircraft, engine, propeller or appliance (products), and have a defined applicability at the time of issuance. On occasion, after AD issuance, the FAA may discover that the applicability did not include all the affected products, and the AD is superseded to increase the applicability. When the FAA supersedes an AD solely for this reason, do not use MSAD risk analysis results in making the decision to issue a supersedure, as the supersedure will be issued regardless of the risk result to meet ICAO obligations or as general policy.
c. For the situations defined in paragraphs 7-1 a. and 7-1 b., the following exceptions to the requirements of the following paragraphs are authorized:
(1) Paragraph 2-9, Record Risk Analysis Results (Step 5.0).
(2) Paragraph 2-11, Perform Causal Analysis (Step 7.0).
(3) Paragraph 2-12, Document the Causes (Step 8.0).
(4) Paragraph 2-13, Identify Cause and Contributing Factors.
(5) Paragraph 2-14, Evaluate and Select Corrective Action for a Fleet Issue (Step 9.0). (6) Some of the requirements defined in paragraph 2-15 a., CARB. See paragraph 7-1 d. for requirements that remain in effect.
d. When exceptions are made in accordance with this chapter, the following limitations and requirements remain in effect:
(1) The issue and recommended corrective action must be presented to the CARB for concurrence per paragraph 2-15 a.
(2) The CARB decision of the issue must be documented in the CARB meeting minutes per paragraph 2-15 b.
(3) The exception of paragraph 7-1 b. may not be exercised if the unsafe condition of the superseding AD in any way differs from the superseded AD unsafe condition (the supersedure must be solely to add applicability).
10/01/2012 8110.107A Appendix A
Appendix A. Definitions and Acronyms (Continued)
Appendix A. Definitions and Acronyms (Continued)
Appendix A. Definitions and Acronyms
14 CFR Title 14 of the Code of Federal Regulations
ACO Aircraft certification office. The aircraft certification directorate’s engineering operational element. This office administers and secures compliance with agency regulations, programs, standards, and procedures governing the type design of aircraft, aircraft engines, or propellers. It offers certification expertise on investigating and reporting aircraft accidents, incidents, and service difficulties. The term “ACO” refers to the Engine Certification Office (ECO), the Rotorcraft Certification Office (RCO), the Special Certification Office (SCO), Military Certificate Office (MCO) and all other ACOs.
14 CFR Title 14 of the Code of Federal Regulations
AD Airworthiness directive
AEG Aircraft evaluation group
AFS Flight Standards Service
AIR Aircraft Certification Service
ASE Aviation safety engineer
ASI Aviation safety inspector
Assigned ASE Directorate or ACO ASE with COS responsibilities for a specific aircraft or product safety issue
AVS Aviation Safety Organization
CAR Corrective action request
CARB Corrective Action Review Board
CAST Commercial Aviation Safety Team
Causes Underlying circumstances, occurrences, and/or failures that contribute, or could contribute, directly or indirectly, to an event.
CCA Candidate corrective action
CICTT CAST/ICAO Common Taxonomy Team
CMO Certificate management office
Condition See “Safety Issue”
10/01/2012 8110.107A Appendix A
Appendix A. Definitions and Acronyms (Continued)
Appendix A. Definitions and Acronyms (Continued)
CPRG Control program risk guideline. The upper limit of acceptable risk which assists the ASE in determining the adequacy, in terms of risk reduction, of a proposed candidate corrective action. These guidelines are characterized in terms of both fleet risk and individual risk.
Corrected risk Residual risk that remains after corrective action is taken. When highly effective corrective action is taken, residual risk is considered to be zero. See also “Uncorrected Risk.”
Corrective action Any action to mitigate a safety issue. Includes mandatory actions like ADs
and rule changes, to correct an unsafe condition. Includes non-mandatory actions and recommendations like SAIBs and Aviation Alerts. Includes actions that either directly corrects the safety problem and/or mitigates risk with operational limitations or restrictions, like grounding a product from further flight.
COS Continued operational safety
CPCP Corrosion prevention and control program
Cross-product Can be across product lines within a manufacturer, across products from various manufacturers, and/or across product-types, if parts, components or processes are common to other aircraft or engines.
DAB Daily alert bulletin
Event Any individual occurrence involving an aircraft or its components. Described in terms of what is observed (the symptoms) or recorded during the occurrence. Events typically trigger investigations that seek causes of a safety issue. The safety issue (or condition) is then evaluated for safety implications.
FAA Federal Aviation Administration
Fleet Aircraft, engine or propeller products of a type currently in service affected by a certain safety issue.
FSDO Flight standards district office
Hazard Any existing or potential condition that can lead to injury, illness or death to people; damage to or loss of a system, equipment or property or damage to the environment. A hazard is a condition that is a prerequisite to an accident or incident.
IAR Immediately adopted rule
10/01/2012 8110.107A Appendix A
Appendix A. Definitions and Acronyms (Continued)
ICAO International Civil Aviation Organization
MCAI Mandatory continuing airworthiness information
MIDO Manufacturing inspection district office
MIO Manufacturing inspection office
MSAD Monitor Safety/Analyze Data
NAR No action required
NCR Nonconformance record
NPRM Notice of proposed rulemaking
Outcome Result of an event, condition, or failure at aircraft level.
PAR Preventive action request
Preliminary risk An initial assessment of the risk posed by a safety issue, often performed assessment with limited data or qualitative information. This assessment is meant to quickly determine an issue’s potential risk and urgency, and is followed by comprehensive and quantitative analysis as data and circumstances permit, unless the issue is deemed to entail very little risk.
Probability Ratio of the number of actual occurrences to the number of possible occurrences. For example, 1 in 1 million flight hours. Probability is often expressed with the denominator normalized to a single unit; therefore, 1x10-6 per flight hour. Probability can also be evaluated against total exposure of the fleet (or other relevant parameter); as in "40% probability that a failure will occur”, or “an expected number of events, if the hazard is not addressed”.
QMITS Quality Management Information Technology Support
QMS Quality management system
RCA Root cause analysis
RGL Regulatory Guidance Library
Risk analysis Process whereby hazards are objectively characterized for their severity and probability. The process can be either qualitative or quantitative.
Risk guideline The upper limit of acceptable risk which assists the ASE in determining the
10/01/2012 8110.107A Appendix A
Appendix A. Definitions and Acronyms (Continued)
need for AD or other mandatory corrective actions and the adequacy, in terms of risk exposure, of a proposed candidate corrective action.
Risk Expression of the severity and probability of an undesired event. See also "corrected risk” and “uncorrected risk."
Safety issue Cause(s), contributing factor(s), or finding(s) that led to, or could lead to, an unsafe outcome. Safety decisions are rendered on issues/causes, not events. For example, investigation of an uncommanded flight control surface movement--an event--might reveal that the cause was a circuit failure in the autopilot’s computer. Circuit failure is the safety issue/cause to evaluate for safety implications, and to take corrective action against.
SAIB Special airworthiness information bulletin
Senior COS ASE The assigned ASE who performs the preliminary risk assessment which has
experience in COS duties. This person is not equivalent to senior engineers that exist in the ACOs. However, senior engineers may be qualified as senior COS ASEs.
Service bulletin One type of "service document" (see below). In this order, the terms are (SB) synonymous.
Service Publications by a type certificate holder, appliance or component documents manufacturer that offer information on safety, product improvement, economics and operational and/or maintenance practices. Publications include service bulletins, all-operators' letters, service newsletters and service digests or magazines. Not included are flight manuals and certain maintenance manuals required for FAA type certification or approval. (Source: AC 20-114, Manufacturers Service Documents)
Severity The consequence or impact of a hazard in terms of degree or loss or harm.
SMS Safety management system
SoDA State of design authority
STC Supplemental type certificate
Substitute Risk Risk of unintended consequences from implementing corrective action
Taxonomy For the purposes of MSAD, a standard industry language and set of definitions that improve the quality of information and communication within the aviation community.
TC Type certificate
10/01/2012 8110.107A Appendix A
Appendix A. Definitions and Acronyms (Continued)
TSO Technical standard order
Uncorrected risk Risk that accumulates over time in the affected fleet if no corrective action is taken for a certain safety issue. See also "corrected risk."
10/01/2012 8110.107A Appendix B
Appendix B. Administrative Information
1. Distribution. Distribute this order to the Washington headquarters division and branch levels of the Aircraft Certification Service and Flight Standards Service, to the headquarters division and regional divisions of the Flight Standards Service, to aircraft evaluation groups, to all Aircraft Certification Service Directorates and certification offices and branches. Distribute to manufacturing inspection offices (MIO), manufacturing inspection district offices (MIDO), manufacturing inspection satellite offices (MISO), all flight standards district offices (FSDO), aircraft certification and airworthiness branches of the FAA Academy, and the International Policy Office, AIR-40.
Appendix B. Administrative Information
2. Authority to Change This Order. The issuance, revision, or cancellation of the material in this order is the responsibility of the AIR Safety Management Design and Analysis Branch (AIR–150). This branch will accomplish all changes, as required, to carry out the FAA’s responsibility to provide guidance for the Monitor Safety/Analyze Data (MSAD) process.
3. Suggestions for Improvement. If you find deficiencies, need clarification or want to suggest improvements to this order, send FAA Form 1320-19, Directive Feedback Information, (written or electronically) to the Aircraft Certification Service, Administrative Services Branch, AIR-510, Attention: Directives Management Officer. You can also send a copy to the Aircraft Engineering Division, AIR-100, Attention: Comments to Order 8110.107A. If you urgently need an interpretation, contact Safety Management Design and Analysis Branch, AIR–150, at 202-267-8588. Always use Form 1320-19, in appendix C, to follow up each verbal conversation.
4. Records Management. Refer to Orders 0000.1, FAA Standard Subject Classification System; 1350.14, Records Management; and 1350.15, Records, Organization, Transfer, and Destruction Standards; or your office Records Management Officer or Directives Management Officer for guidance regarding retention or disposition of records.
5. Related Federal Regulations and Publications.
FAA-IR-M 8040.1, Airworthiness Directives Manual.
FAA Order 8040.1, Airworthiness Directives.
FAA Order 8040.4, Safety Risk Management.
FAA Order 8040.5, Airworthiness Directive Process for Mandatory Continuing
Airworthiness Information.
FAA Order 8110.100, Special Airworthiness Information Bulletin.
0
10/01/2012 8110.107A Appendix C
U.S. Department of Transportation
Federal Aviation Administration
Appendix C. Directive Feedback Information
Appendix C. Directive Feedback Information
Please submit any written comments or recommendations for improving this directive, or suggest new items or subjects to be added to it. Also if you find an error, please tell us about it.
Subject: Order 8110.107A
To: Directive Management Office, [email protected]
(Please check all appropriate line items)
□ An error (procedural or typographical) has been noted in paragraph … on page …
□ Recommend paragraph … on page … be changed as follows:
(attach separate sheet if necessary)
□ In a future change to this directive, please include coverage on the following subject:
(briefly describe what you want added):
□ Other comments:
□ I would like to discuss the above. Please contact me.
Submitted by: … Date: …
Telephone Number: … Routing symbol: …
FAA Form 1320-19 (10-98)