Back to Top

Educating the Aerospace Industry

Blog Archives

Major Reviews

Test Readiness Review (TRR)

A Test Readiness Review (TRR) is conducted to determine if the system under review is ready to proceed into formal testing by deciding whether the test procedures are complete and verify their compliance with test plans and descriptions. A TRR is normally conducted before each major test configuration item including hardware and software and provides management with the assurance that a system has undergone a thorough test process and is ready for turnover to the next test phase. The Flight Readiness Review (FRR) is a subset test of the TRR.

Checklist: Test Readiness Review (TRR) Risk Assessment

The TRR assesses test objectives, test methods and procedures, scope of tests, and safety and confirms that required test resources have been properly identified and coordinated to support planned tests. The TRR verifies the traceability of planned tests to program requirements and user needs. The TRR also assesses the system under review for development maturity, cost/ schedule effectiveness, and risk to determine readiness to proceed to formal testing.

The TRR should answer the following questions:

  1. Why are we testing?
  2. What is the purpose of the planned test?
  3. Does the planned test verify a requirement that is directly traceable back to a system specification or other program requirement?
  4. What are we testing (subsystem, system, system of systems, other)?
  5. Is the configuration of the system under test sufficiently mature, defined, and representative to accomplish planned test objectives and or support defined program objectives?
  6. Are we ready to begin testing?
  7. Have all planned preliminary, informal, functional, unit level, subsystem, system, and qualification tests been conducted, and are the results satisfactory?
  8. What is the expected result and how can/do the test evaluation results affect the program?
  9. Is the planned test properly resourced (people, test article or articles, facilities, data systems, support equipment, logistics, etc.)?
  10. What are the risks associated with the tests and how are they being mitigated?
  11. What are the hazards and ESOH risks associated with the specific testing?
  12. Have the necessary “Safety Releases” from the Program Manager (PM) been provided to developmental and operational testers prior to any test using personnel?
  13. What is the fall-back plan should a technical issue or potential showstopper arise during testing?

The scope of the TRR is directly related to the risk level associated with performing the planned tests and the importance of the test evaluation results to overall program success. The level of specific risk and associated risk level will vary as a system proceeds from component level, to system level, to systems of systems level testing. Early component level tests may not require the same level of review as the final system level tests. Sound judgment should dictate the scope of a specific test or series of tests.

Typical TRR success criteria including the following:

  • Completed and approved test plans for the system under test,
  • Completed identification and coordination of required test resources,
  • The judgment that previous component, subsystem, and system test results form a satisfactory basis for proceeding into planned tests, and
  • Identified risk level acceptable to the program leadership.

The Program Manager (PM) should address the scope of the TRR in the Systems Engineering Plan (SEP). Test and Evaluation (T&E) is an integral part of the Systems Engineering Processes of Verification and Validation.

AcqLinks and References:

Updated: 7/18/2017

Major Reviews

Initial Technical Review (ITR)

 

The Initial Technical Review (ITR) is a multi-disciplined technical review to support a program’s initial Program Objective Memorandum (POM) submission in the Materiel Solutions Analysis (MSA) Phase . This review ensures a program’s Technical Baseline is sufficiently rigorous to support a valid cost estimate and enable an independent assessment. The ITR assesses the capability needs and materiel solution approach of a proposed program and verifies that the requisite research, development, test and evaluation, engineering, logistics, and programmatic bases for the program reflect the complete spectrum of technical challenges and risks.

Additionally, the ITR ensures the historical and prospective drivers of system Life-Cycle cost (LCC) have been quantified to the maximum extent and that the range of uncertainty in these parameters has been captured and reflected in the program cost estimates. The basic Cost Analysis Requirements Description (CARD)  technical and programmatic guidance, tailored to suit the scope and complexity of the program, should be followed to ensure all pertinent design-related cost drivers are addressed.

Completion of the ITR should provide:

  1. A complete Cost Analysis Requirements Description (CARD) – like document detailing the operational concept, candidate materiel solutions, and, risks,
  2. An assessment of the technical and cost risks of the proposed program, and
  3. An independent assessment of the program’s cost estimate; Independent Cost Estimate (ICE).

Typical ITR success criteria include affirmative answers to the following exit questions:

  1. Does the CARD-like document capture the key program cost drivers, development costs (all aspects of hardware, human integration, and software), production costs, operation and support costs?
  2. Is the CARD-like document complete and thorough?
  3. Are the underlying assumptions used in developing the CARD-like document technically and programmatically sound, executable, and complete?
  4. Have the appropriate technical and programmatic competencies been involved in the CARD-like document development, and have the proper SMEs been involved in its review?
  5. Are the risks known and manageable within the cost estimate?
  6. Is the program, as captured in the CARD-like document, executable?

AcqTips:

  • Independent assessment is conducted by Subject Matter Experts (SMEs)

AcqLinks and References:

Major Reviews

Integrated Baseline Review (IBR)

 

Major Reviews


An Integrated Baseline Review (IBR) is a joint assessment conducted by the government Program Manager (PM) and the contractor to establish a mutual understanding of the Performance Measurement Baseline (PMB). This understanding provides for an agreement on a plan of action to evaluate the risks inherent in the PMB and the management processes that operate during program execution. PM’s are required to conduct IBRs on all cost or incentive contracts that require the implementation of Earned Value Management (EVM) (contracts valued at or greater than $20 million).  IBRs should be used to understand:

  • The scope of the PMB consistent with authorizing documents;
  • Management control processes;
  • Risks in the PMB associated with cost, schedules, and resources; and
  • Corrective actions where necessary.

IBR FigureCompletion of the review should result in the assessment of risk within the PMB and the degree to which the following have been established:

  1. Technical scope of work is fully included and is consistent with authorizing documents,
  2. Key project schedule milestones are identified and supporting schedules reflect a logical flow to accomplish the work,
  3. Resources (budgets, facilities, infrastructure, personnel, skills, etc.) are available and are adequate for the assigned tasks,
  4. Tasks are planned and can be measured objectively relative to the technical progress,
  5. Rationales underlying the PMB are reasonable, and
  6. Management processes support successful execution of the project.

IBRs should be scheduled as early as practicable and the timing of the IBRs should take into consideration the contract period of performance. The process will be conducted not later than 180 calendar days (6 months) after:

  1. Contract award,
  2. Exercise of significant contract options, and
  3. Incorporation of major modifications.

IBRs are also performed at the discretion of the PM or within a reasonable time after the occurrence of major events in the life of a program. These events may be completion of the Preliminary Design Review (PDR), completion of the Critical Design Review (CDR), a significant shift in the content and/or time phasing of the PMB, or when a major milestone such as the start of the production option of a development contract is reached. Continuous assessment of the PMB will identify when a new IBR should be conducted.

AcqTips:

AcqLinks and References:

Major Reviews

Preliminary Design Review (PDR)

 

Major Reviews


The Preliminary Design Review (PDR) is a technical assessment that establishes the physically Allocated Baseline of a system to ensure a system in operationally effective.  This review assesses the allocated design documented in subsystem product specifications for each configuration item in the system and ensures that each function, in the Functional Baseline, has been allocated to one or more system configuration items. The PDR establishes the allocated baseline (hardware, software, human/support systems) and underlying architectures to ensure that the system under review has a reasonable expectation of satisfying the requirements within the currently allocated budget and schedule.

For complex systems, a PDR may be conducted incrementally for each configuration item. These incremental reviews lead to an overall system level PDR. System level performance is supported by compliance with Interface Control Documents (ICD), but not assured. Interface requirements make up each configuration item Allocated Specification.

The Weapons System Acquisition Reform Act of 2009 directed the PDR to: 

  • PDRs before Milestone (MS) B are mandatory for all Major Defense Acquisition Programs (MDAP) and will be reflected in the Technology Development Strategy (TDS) to be approved by the MDA at MS A. Post-PDR assessments will be conducted in association with MS B preparations and will be formally considered by the Milestone Decision Authority (MDA) at the MS B certification review.
  • The timing of PDRs for other than MDAPs will be approved by the DoD Component MDA when consistent with TDS or Acquisition Strategy objectives. When the PDR is conducted before MS B, a post-PDR assessment will be conducted in association with the MS B review and formally considered by the MDA at the MS B review. If the PDR is conducted after MS B, the MDA will conduct a post-PDR assessment at a time reflected in the approved acquisition strategy.
  • PDR before MS B is now a statutory requirement for MDAPs. The post-PDR assessment will be conducted during the MS B review, and prior to the section 2366b certification by the MDA per title 10, Unites States Code.

Completion of the PDR should provide the following:

AcqTips:

  • The Program Manager (PM) should conduct the PDR when all major design issues have been resolved and work can begin on detailed design. The PDR should address and resolve critical, system-wide issues.
  • The PDR should be conducted when the allocated baseline has been achieved, allowing detailed design of hardware and software configuration items to proceed. A rule of thumb is that 10 percent to 25 percent of product drawings and associated instructions should be complete, and that 100 percent of all safety-critical component (Critical Safety Items and Critical Application Items) drawings are complete.

AcqLinks and References:

Major Reviews

Operational Test Readiness Review (OTRR)

 

Major Reviews


The Operational Test Readiness Review (OTRR) assess if a system should proceed into Initial Operational Test and Evaluation (IOT&E). The review addresses and verifies system reliability, maintainability, and supportability performance and determines if the hazards and Environmental, Safety, Occupational and Health (ESOH) residual risks are manageable within the planned testing operations.  This assessment determines if changes are required in planning, resources, or testing necessary to proceed with IOT&E. Of critical importance to this review is the understanding of available system performance to meet the Capability Production Document (CPD) performance threshold values. The OTRR may be conducted in multiple steps to ensure that the “production configuration” system (usually the Low-Rate Initial Production (LRIP) system) can proceed into Operational Test & Evaluation (OT&E) with a high probability of success.

Checklist: Operational Test Readiness Review (OTRR) Checklist

Programs on the Office of Secretary of Defense (OSD) Test and Evaluation (T&E) oversight list are required by DoD policy to establish a Service process for determining and certifying a program’s readiness for IOT&E by the Service Component Acquisition Executive (CAE).  The OTRR is only complete when the CAE evaluates and determines materiel system readiness for IOT&E. The OTRR may be conducted by the Program Manager (PM) or the Operational Test Agency, depending on Service policy.

DoD specifies that the OTRR review shall include:

AcqLinks and References:

Major Reviews

Production Readiness Review (PRR)

 

Major Reviews


The Production Readiness Review (PRR) assesses a program to determine if the design is ready for production. It assesses if the prime contractor and major subcontractors have accomplished adequate production planning without incurring unacceptable risks that will breach thresholds of schedule, performance, cost, or other established criteria. PRRs are normally performed as a series of reviews toward the end of Engineering, Manufacturing and Development (EMD) Phase and should be conducted during System Capability and Manufacturing Process Demonstration to identify and mitigate risks as the design progresses.

The PRR evaluates the full, production-configured system to determine if it correctly and completely implements all system requirements. The review determines whether the traceability of final system requirements to the final production system is maintained. A successful review is predicated on the determination that the system requirements are fully met in the final production configuration and that production capability forms a satisfactory basis for proceeding into Low-Rate Initial Production (LRIP) and Full-Rate Production (FRP).

The PRR should review:

  • The manufacturing readiness process
  • Quality management system
  • Production planning
  • System requirements compliance
  • Inventory management
  • Supplier management

AcqLinks and References: