Back to Top

Educating the Aerospace Industry

Blog Archives

Modeling & Simulation

Prototyping (M&S)

 

A Prototype is a physical or virtual model used to evaluate the technical or manufacturing feasibility or military utility of a particular technology or process, concept, end item, or system. Prototyping is used in an acquisition program as a tool for:

Early prototypes may be built and evaluated during the Technology Development (TD) Phase, or later in the Engineering and Manufacturing Development (EMD) Phase, or is the result of a Joint Capability Technology Demonstration (JCTD) or Advanced Technology Demonstration (ATD), and tested prior to Milestone C decision. Selected prototyping may continue after Milestone C, as required, to identify and resolve specific design or manufacturing risks, or in support of Evolutionary Acquisition (EA).

The Technology Development Strategy (TDS) should include a description of the prototyping purpose and the prototyping strategy at the system and subsystem levels. The TDS should include the number of prototype units that may be produced and employed during the TD Phase. The description should also include a discussion of:

  • How prototype units will be supported
  • Specific performance goals and Exit Criteria that should be met by the employment of the prototypes

Prototypes usually fall into five (5) basic categories:

  1. Proof-of-Principle Prototype: is used to test some aspect of the intended design without attempting to exactly simulate the visual appearance, choice of materials or intended manufacturing process.
  2. Form Study Prototype: type of prototype will allow designers to explore the basic size, look and feel of a product without simulating the actual function or exact visual appearance of the product.
  3. User Experience Prototype: invites active human interaction and is primarily used to support user focused research
  4. Visual Prototype: will capture the intended design aesthetic and simulate the appearance, color and surface textures of the intended product but will not actually embody the function(s) of the final product.
  5. Functional Prototype: attempts to simulate the final design, aesthetics, materials and functionality of the intended design. The functional prototype may be reduced in size (scaled down) in order to reduce costs.

Data Production Prototype
A data prototype is a form of functional or working prototype. The justification for its creation is usually a data migration, data integration or application implementation project and the raw materials used as input are an instance of all the relevant data which exists at the start of the project.

The objectives of data prototyping are to produce:

  • A set of data cleansing and transformation rules which have been seen to produce data which is all fit for purpose.
  • A dataset which is the result of those rules being applied to an instance of the relevant raw (source) data.

AcqLinks and References:

Modeling & Simulation

Modeling Terminology

 

Below is a list of a few common words that may have special meaning when applied to models. [1]

  • Accuracy: The closeness of a measured or modeled/computed value to its “true” value. The “true” value is the value it would have if we had perfect information. We will talk later about various ways to measure accuracy.
  • Algorithm: A set of rules for solving some problem. On a computer, an algorithm is a set of rules in computer code that solve a problem.
  • Calibration: The process of adjusting model parameters within physically defensible ranges until the resulting predictions gives the best possible fit to the observed data.
  • Conceptual Model: A hypothesis regarding the important factors that govern the behavior of an object or process of interest. This can be an interpretation or working description of the characteristics and dynamics of a physical system.
  • Deterministic Model: A model that provides a single solution for the variables being modeled. Because this type of model does not explicitly simulate the effects of data uncertainty or variability, changes in model outputs are solely due to changes in model components.
  • Empirical Model: An empirical model is one where the structure is determined by the observed statistical relationship among experimental data. These models can be used to develop relationships that are useful for forecasting and describing trends in behavior but they are not necessarily mechanistically relevant that is they don’t explain the real causes and mechanisms for the relationships.
  • Federate: an application that may be, or is coupled with other software applications under a Federation Object Model Document Data (FDD) and a runtime infrastructure (RTI).
  • Federation: a named set of federate applications and a common Federation Object Model that are used as a whole to achieve some specific objective.
  • Parameters: Terms in the model that are fixed during a model run or simulation but can be changed in different runs as a method for conducting sensitivity analysis or to achieve calibration goals.
  • Run Time Infrastructure (RTI): The software that provides common interface services during a HLA federation execution for synchronization and data exchange.
  • Sensitivity: The degree to which the model outputs are affected by changes in a selected input parameters.
  • Simulation Object Model (SOM): a specification of the types of information that an individual federate could provide to HLA federations as well as the information a federate could receive from other federates in HLA federations.
  • Statistical Models: Models obtained by fitting observational data to a mathematical function.
  • Stochastic Model: A model that includes variability in model parameters. This variability is a function of:
    • changing environmental conditions,
    • spatial and temporal aggregation within the model framework,
    • random variability.
  • Variable: A measured or estimated quantity which describes an object or can be observed in a system and which is subject to change.
  • Validation: Answers the questions “Is the science valid and does the model use current methods and techniques? Is the numerical model adequate to convey the science principles at the level of the question being asked? Is the model arriving at an acceptably accurate representation of the phenomenon being modeled?”
  • Verification: Does the code for the model run correctly and provide a mathematically correct answer? Do the algorithms being used accurately represent the mathematical function on the computer?

AcqLinks and References:

Modeling & Simulation

M&S Information Management

 

Effective acquisition requires collaboration among multiple Stakeholders. Modeling & Simulation (M&S) tools combined with an information sharing infrastructure (i.e., integrated data environment – IDE) into a distributed collaborative environment can enable cost-effective development and sustainment of systems and systems of systems. [1]

M&S should share the common information base with the rest of Systems Engineering. The IDE should allow data producers to publish their data items and record required metadata. Metadata is critical to the discovery and understanding of functions. It should also allow data consumers, from their desktop, to readily discover (via browsing and searching), access (via proper access controls), understand, and download the data they need and have an archiving capability so that the program can, at any future time, identify the information set that was used to inform a decision. [1]

Metadata
The metadata that accompanies each data item should provide the information needed to understand its structure, lineage, and meaning, including its context and applicability. This same metadata will facilitate Verification, Validation & Accreditation (VV&A) and will allow the data it describes to be transformed into the form needed by the consuming M&S tool. Particular sets of data items will describe the system at a particular point in its development and as such will inform program events and provide the information foundation for program activities, including M&S-based analyses.
[1]

AcqLinks and References:

Modeling & Simulation

DoD Modeling & Simulation Catalog

 

Modeling & Simulation CatelogThe DoD Modeling & Simulation (M&S) Catalog is a web-based search capability that provides a “card catalog” level of detail about M&S tools, data and services available to its users.  It’s designed to guide a user quickly to a manageable set of alternatives to evaluate. The tool has the capability to perform analysis of the characteristics of the search result for a set of resources. The M&S Catalog leverages the DoD Net-centric Vision of all resource descriptions and contact information (metadata) being posted in a defined format on the Global Information Grid (GIG). It can load data from multiple file types as long as the format is known and specifies the metadata elements. Coordination between source sites and the M&S Catalog ensure the metadata is only accessible to authorized users. [1]

Website: DoD M&S Catalog

One of the goals of the DoD Net-centric Vision is establishing visibility into the M&S resources across the DoD. In order to manage and employ M&S capabilities effectively senior leaders and managers must have visibility into the DoD’s M&S portfolio. Organizations supported by M&S need visibility into the tools, data and services that meet their requirements. This visibility is established through a discovery process that has at its core a search capability.

AcqLinks and References:

Modeling & Simulation

M&S Development Diagram

 

VV&A and M&S Development Diagram

See the Verification, Validation & Accreditation Recommended Practice Guide for more detailed information.

AcqLinks and References:

Modeling & Simulation

Modeling & Simulation Support

 

The use of Modeling and Simulation (M&S) in Test and Evaluation (T&E) provides test data to support system concept exploration. It provides valuable information that can increase confidence levels, decrease field test time and costs, and provide data for pre-test prediction and post-test validation.

M&S in T&E can be divided into three (3) categories:

  1. Constructive Simulations: A computer simulation with no hardware
  2. Virtual Simulations: A computer simulation that uses actual hardware
  3. Live Simulations: Use live exercises where troops use actual equipment under actual environmental conditions that approach real life combat.

Validity of M&S
Simulations, manual and computer-designed, can complement and increase the validity of live T&Es by proper selection and application. M&S must be approved for use through verification, validation, and accreditation processes. – – See M&S Verification, Validation and Accreditation

Support to Test Design and Planning
The M&S can assist in the T&E planning process and can reduce the cost of testing. Computer simulations may be used to test the planning for an exercise. By setting up and running the test exercise in a simulation, the timing and scenario may be tested and validated.

Support to Test Execution
Simulations can be useful in test execution and dynamic planning. With funds and other restrictions limiting the number of times that a test may be repeated and each test conducted over several days, it is mandatory that the test director exercises close control over the conduct of the test to ensure the specific types and quantities of data needed to meet the test objectives are being gathered and to ensure adequate safety.

Support to Analysis and Test Reporting
M&S may be used in post-test analysis to extend and generalize results and to extrapolate to other conditions. Simulations can be used to extend test results, save considerable energy (fuel and manpower), and save money by reducing the need to repeat data points to improve the statistical sample or to determine overlooked or directly unmeasured parameters.

Simulation Integration
Simulations are no longer stove-piped tools in distinct areas, but are increasingly crossing disciplines and different uses.

Simulation Planning
With M&S becoming increasingly more complex, more expensive, and more extensive, testers must be thorough in planning their use and subsequent technical confidence. Testers are expected to be involved increasingly earlier, including the Operational Test Agencies (OTA), if the subsequent T&E results are to be accepted. Simulation Support Plans (SSPs) are program documents that span the many simulations, their purpose, and their expected credibility.

AcqLinks and References:

Modeling & Simulation

M&S Body of Knowledge

 

M&S Body of Knowledge

M&S Body of Knowledge

This DoD Modeling & Simulation (M&S) Body of Knowledge (BOK) was published in June 2008. The BOK provides standardized language and associated knowledge base for users, developers, managers and executive-level personnel to effectively apply M&S to DoD requirements. The awareness and application usage levels contained in this BOK are displayed by the services (Army, Navy, Air Force, Marine and Joint) and the following communities: Acquisition, Planning, Test & Evaluation, and Training. The management and executive levels contain a usage level that represents an average usage level across the services and communities.

DoD Modeling &Simulation Body of Knowledge

AcqLinks and References:

Schedule Development

Microsoft Project

 

Microsoft Project PictureMicrosoft Project is a schedule development program by Microsoft that allows program personnel to effectively manage their project. It graphically puts together a schedule that can be utilized by project personnel and easily collaborated with. It can also be used to determine the Critical Path, PERT Analysis, and help with earn value analysis.   The program allows users to:

  • Understand and control project schedules and finances.
  • Communicate and present project information.
  • Organize work and people to make sure that projects are completed on schedule.

Example Project 3

AcqLinks and References:

 

Production, Quality & Manufacturing

Manufacturing Readiness Level (MRL)

 

Manufacturing Readiness Levels (MRLs) are used with assessments and are designed to assess the maturity of a given technology, system, subsystem, or component from a manufacturing prospective. MRLs provide decision makers (at all levels) with a common understanding of the relative maturity (and attendant risks) associated with manufacturing technologies, products, and processes being considered to meet DoD requirements. [1]

Guide: Manufacturing Readiness Level Deskbook v2.4 – Aug 15

Manufacturing readiness and technology readiness go hand-in-hand. MRLs, in conjunction with Technology Readiness Levels (TRL), are key measures that define risk when a technology or process is matured and transitioned to a system. It is quite common for manufacturing readiness to be paced by technology readiness or design stability. Manufacturing processes will not be able to mature until the product technology and product design are stable. [2]

Level Definition DoD MRL Description
1 Basic Manufacturing Implications Identified This is the lowest level of manufacturing readiness. The focus is to address manufacturing shortfalls and opportunities needed to achieve program objectives. Basic research (i.e., funded by budget activity) begins in the form of studies.
2 Manufacturing Concepts Identified This level is characterized by describing the application of new manufacturing concepts. Applied research translates basic research into solutions for broadly defined military needs. Typically this level of readiness includes identification, paper studies and analysis of material and process approaches. An understanding of manufacturing feasibility and risk is emerging.
3 Manufacturing Proof of Concept Developed This level begins the validation of the manufacturing concepts through analytical or laboratory experiments. This level of readiness is typical of technologies in Applied Research and Advanced Development. Materials and/or processes have been characterized for manufacturability and availability but further evaluation and demonstration is required. Experimental hardware models have been developed in a laboratory environment that may possess limited functionality.
4 Capability to produce the technology in a laboratory environment This level of readiness acts as an exit criterion for the Materiel Solution Analysis (MSA) Phase approaching a Milestone A decision. Technologies should have matured to at least TRL 4. This level indicates that the technologies are ready for the Technology Maturation & Risk Reduction Phase of acquisition. At this point, required investments, such as manufacturing technology development, have been identified. Processes to ensure manufacturability, producibility, and quality are in place and are sufficient to produce technology demonstrators. Manufacturing risks have been identified for building prototypes and mitigation plans are in place. Target cost objectives have been established and manufacturing cost drivers have been identified. Producibility assessments of design concepts have been completed. Key design performance parameters have been identified as well as any special tooling, facilities, material handling and skills required.
5 Capability to produce prototype components in a production
relevant environment
This level of maturity is typical of the mid-point in the Technology Maturation & Risk Reduction Phase of acquisition, or in the case of key technologies, near the mid-point of an Advanced Technology Demonstration (ATD) project. Technologies should have matured to at least TRL 5. The industrial base has been assessed to identify potential manufacturing sources. A manufacturing strategy has been refined and integrated with the risk management plan. Identification of enabling/critical technologies and components is complete. Prototype materials, tooling and test equipment, as well as personnel skills have been demonstrated on components in a production relevant environment, but many manufacturing processes and procedures are still in development. Manufacturing technology development efforts have been initiated or are ongoing. Producibility assessments of key technologies and components are ongoing. A cost model has been constructed to assess projected manufacturing cost.
6 Capability to produce a prototype system or subsystem in a
production relevant environment
This MRL is associated with readiness for a Milestone B decision to initiate an acquisition program by entering into the Engineering and Manufacturing Development (EMD) Phase of acquisition. Technologies should have matured to at least TRL 6. It is normally seen as the level of manufacturing readiness that denotes acceptance of a preliminary system design. An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined and characterized, but there are still significant engineering and/or design changes in the system itself. However, preliminary design has been completed and producibility assessments and trade studies of key technologies and components are complete. Prototype manufacturing processes and technologies, materials, tooling and test equipment, as well as personnel skills have been demonstrated on systems and/or subsystems in a production relevant environment. Cost, yield and rate analyses have been performed to assess how prototype data compare to target objectives, and the program has in place appropriate risk reduction to achieve cost requirements or establish a new baseline. This analysis should include design trades. Producibility considerations have shaped system development plans. The Industrial Capabilities Assessment (ICA) for Milestone B has been completed. Long-lead and key supply chain elements have been identified.
7 Capability to produce systems, subsystems, or components in a
production representative environment
This level of manufacturing readiness is typical for the mid-point of the Engineering and Manufacturing Development (EMD) Phase leading to the Post-CDR Assessment. Technologies should be on a path to achieve TRL 7. System detailed design activity is nearing completion. Material specifications have been approved and materials are available to meet the planned pilot line build schedule. Manufacturing processes and procedures have been demonstrated in a production representative environment. Detailed producibility trade studies are completed and producibility enhancements and risk assessments are underway. The cost model has been updated with detailed designs, rolled up to system level, and tracked against allocated targets. Unit cost reduction efforts have been prioritized and are underway. Yield and rate analyses have been updated with production representative data. The supply chain and supplier quality assurance have been assessed and long-lead procurement plans are in place. Manufacturing plans and quality targets have been developed. Production tooling and test equipment design and development have been initiated.
8 Pilot line capability demonstrated; Ready to begin Low Rate Initial
Production
This level is associated with readiness for a Milestone C decision, and entry into Low Rate Initial Production (LRIP). Technologies should have matured to at least TRL 7 or 8. Detailed system design is complete and sufficiently stable to enter low rate production. All materials, manpower, tooling, test equipment and facilities are proven on pilot line and are available to meet the planned low rate production schedule. Manufacturing and quality processes and procedures have been proven in a pilot line environment and are under control and ready for low rate production. Known producibility risks pose no significant challenges for low rate production. Cost model and yield and rate analyses have been updated with pilot line results. Supplier qualification testing and first article inspection have been completed. The Industrial Capabilities Assessment for Milestone C has been completed and shows that the supply chain is established to support LRIP.
9 Low rate production demonstrated; Capability in place to begin
Full Rate Production
At this level, the system, component or item has been previously produced, is in production, or has successfully achieved low rate initial production. Technologies should have matured to TRL 8 or 9. This level of readiness is normally associated with readiness for entry into Full Rate Production (FRP). All systems engineering/design requirements should have been met such that there are minimal system changes. Major system design features are stable and have been proven in test and evaluation. Materials, parts, manpower, tooling, test equipment and facilities are available to meet planned rate production schedules. Manufacturing process capability in a low rate production environment is at an appropriate quality level to meet design key characteristic tolerances. Production risk monitoring is ongoing. LRIP cost targets have been met, and learning curves have been analyzed with actual data. The cost model has been developed for FRP environment and reflects the impact of continuous improvement.
10 Full Rate Production demonstrated and lean production practices
in place
This is the highest level of production readiness. Technologies should have matured to TRL 9. This level of manufacturing is normally associated with the Production or Sustainment phases of the acquisition life cycle. Engineering/design changes are few and generally limited to quality and cost improvements. System, components or items are in full rate production and meet all engineering, performance, quality and reliability requirements. Manufacturing process capability is at the appropriate quality level. All materials, tooling, inspection and test equipment, facilities and manpower are in place and have met full rate production requirements. Rate production unit costs meet goals, and funding is sufficient for production at required rates. Lean practices are well established and continuous process improvements are ongoing.

AcqLinks and References:

Major Reviews

Initial Technical Review (ITR)

 

The Initial Technical Review (ITR) is a multi-disciplined technical review to support a program’s initial Program Objective Memorandum (POM) submission in the Materiel Solutions Analysis (MSA) Phase . This review ensures a program’s Technical Baseline is sufficiently rigorous to support a valid cost estimate and enable an independent assessment. The ITR assesses the capability needs and materiel solution approach of a proposed program and verifies that the requisite research, development, test and evaluation, engineering, logistics, and programmatic bases for the program reflect the complete spectrum of technical challenges and risks.

Additionally, the ITR ensures the historical and prospective drivers of system Life-Cycle cost (LCC) have been quantified to the maximum extent and that the range of uncertainty in these parameters has been captured and reflected in the program cost estimates. The basic Cost Analysis Requirements Description (CARD)  technical and programmatic guidance, tailored to suit the scope and complexity of the program, should be followed to ensure all pertinent design-related cost drivers are addressed.

Completion of the ITR should provide:

  1. A complete Cost Analysis Requirements Description (CARD) – like document detailing the operational concept, candidate materiel solutions, and, risks,
  2. An assessment of the technical and cost risks of the proposed program, and
  3. An independent assessment of the program’s cost estimate; Independent Cost Estimate (ICE).

Typical ITR success criteria include affirmative answers to the following exit questions:

  1. Does the CARD-like document capture the key program cost drivers, development costs (all aspects of hardware, human integration, and software), production costs, operation and support costs?
  2. Is the CARD-like document complete and thorough?
  3. Are the underlying assumptions used in developing the CARD-like document technically and programmatically sound, executable, and complete?
  4. Have the appropriate technical and programmatic competencies been involved in the CARD-like document development, and have the proper SMEs been involved in its review?
  5. Are the risks known and manageable within the cost estimate?
  6. Is the program, as captured in the CARD-like document, executable?

AcqTips:

  • Independent assessment is conducted by Subject Matter Experts (SMEs)

AcqLinks and References: