There is considerable excitement, interest, attention, angst and what have you in Department of Defense (DoD) Acquisitions about new programs’ use of prototypes, prototyping, and rapid prototyping.
There also are some serious workforce misconceptions about what prototypes are and, most important, what they are not. Let us provide some context and expectation management about prototyping and prototypes as increasingly employed in early program stages.
Prototyping Defense Systems
The concept of prototyping defense systems is not new. The Army required delivery of prototype vehicles (called Pilot Models prior to World War II) for a competition that eventually resulted in the venerable “Jeep” vehicle. The prototypes delivered for that program in 1940 — the “1/4 ton, 4×4 utility trucks” — bear only a general resemblance to the Willys MBs and Ford GPWs that were mass-produced as the standard Jeep design during the war. There were significant requirements changes (such as vehicle weight) and redesigning to move the Pilot Models to a producible wartime configuration. Fast forward to a more recent example, the Advanced Tactical Fighter (ATF) program also required delivery of two prototype aircraft from each competitor for a fly-off to decide the winner. Those two prototype aircraft, the YF-22 and YF-23, were evaluated by the Air Force and the YF-22 team was selected and proceeded to design and deliver the present-day F-22A.
So then what exactly is a prototype? Is the Bantam Reconnaissance Car the same as a Jeep, or the YF-22 the same as an F-22A? Not exactly. Prototyping is a part — but not the end state — of the design process. A prototype, or “first type” is the preliminary type, form, or instance of a system or system element that serves as a model for what comes later. Contractors, industry, and research organizations develop prototypes for many reasons. Prototypes are used to reduce technical risk, validate designs or cost estimates, evaluate manufacturing processes, refine requirements, or even explore new operational concepts. Risk reduction prototypes materially decrease Engineering and Manufacturing Development (EMD) risk at an acceptable cost. They can be a system such as the YF-22, or can focus on subsystems or components.
Prototypes may not, and in most cases, do not, have all the features or capabilities of the fully designed system. They may not need to have all the features of the final system. Predominantly they are used to demonstrate the ability to achieve certain critical performance parameters, validate computer models or explore improved manufacturing processes. Prototypes are viewed as only a “temporary” step or to serve a specific purpose in the full design process. After they are fabricated, the design process continues evolving until the completed system is verified. Competitive prototypes are produced by two or more companies or teams competing for a contract award. The enduring problem with competitive prototypes, as many industry officials and financial analysts see it, is that the companies are required to spend huge sums of money upfront without any guarantees that they will be able to recoup their investments. So when they explore prototyping, companies limit their work to fabricating a system with the key features necessary to win a contract to proceed to the next acquisition phase.
Pre-Prototyping
Considerable design work is required before a prototype can even be constructed. The systems engineering process already is fully under way: Requirements generation and analysis, architecture design, functional baseline determinations — all must be done before even a prototype design can be communicated to a group that will fabricate the prototype. This process involves in-depth modeling and simulation in particular or what we now refer to as Digital Engineering. Design changes, particularly to facilitate manufacturing, are a normal outgrowth of the prototyping process. Prototypes basically are things you learn from, not things you go to war with.
In the case of the ATF, the Air Force wanted demonstration of some key features through the prototype delivery and subsequent flight evaluations. Those features were stealth, super-cruise (ability to fly above Mach 1 without using an afterburner), maneuverability, handling qualities, and the ability to serve as test-beds for two different prototype engines. Conversely, there were many features that the prototype ATFs were not demonstrating, such as avionics maturity. These first aircraft were constructed with only enough avionics to ensure safety of flight. They did not incorporate fully mature and integrated avionics, but they didn’t need to do so at that point in the design process. The prototypes delivered to the Air Force were sufficient, as intended, for selecting a design team to proceed into EMD.
Small Subsystems
How does prototyping actually work in practice for smaller subsystems? During my private-sector university research days, I led a team working on a design for a small internal-combustion engine for an unmanned aerial vehicle (UAV) that would run on so-called “heavy fuel” or JP-8. The Army was looking for a heavy-fuel engine to replace the 38-horsepower (hp) gasoline power plant on the Shadow UAV. Under a grant from the Army Research Lab, my research team set about to find such an engine—or see if one could be modified to meet the Army requirements.
When no suitable off-the-shelf replacement engine could be found, we partnered with a small company that had an idea for converting small internal combustion engines from operating on gasoline to heavy fuel but had not yet proven the concept. The initial phase of our research was focused on the key Army requirement of demonstrating heavy fuel operation on an engine in the same power class as the existing engine. We selected an existing 20-hp, two-cylinder aircraft engine that we felt was adequate to use as a prototype testbed to validate the company’s conversion process from gasoline to JP-8. If that initial demonstration succeeded, we planned to scale up the conversion to a larger, four-cylinder prototype version. The conversion parts were designed, fabricated, and installed on the 20-hp version at the same time an engine test stand/test cell was developed and constructed. My team demonstrated JP-8 heavy fuel operation on the 20-hp engine using the conversion design. We then migrated the concept to the four-cylinder, 40-hp prototype and again validated the heavy fuel operation and sufficient engine power for this to be considered as a suitable candidate replacement for the existing Shadow engine.
Although the two prototyping efforts were successful in the two key areas that the Army focused on, there were many other important design features and requirements we were not then able to explore during this early risk-reduction prototyping phase of the program. For example, while operating our test engines, our team experienced the difficulty of trying to tune four independent carburetors on the cylinders and began conceptual design of a fuel injection system that would replace the carburetors.
Limitations
There were other limitations to what we could do with our prototypes. During further engine testing we experienced a failure of the main engine bearings due to the different orientation of the engine on a UAV versus a manned aircraft. We had no opportunity to do any detailed platform integration or installed performance testing with the Shadow itself, which would be an important next step after demonstrating the stand-alone engine performance. Important design considerations such as engine aerodynamic cooling, low-temperature or high-altitude performance could not be tested in our test cell.
We did not do any characterization of the acoustic or infrared signature of the modified engine. During testing, we realized extra warm-up and start time would be required by the use of heavy fuel intended for turbine or compression-ignition engines. Although we were able to achieve the required engine operation time to pass a Federal Aviation Administration qualification test specified by the Army, there was no opportunity to evaluate the long-term reliability and maintainability of the engine and propeller in field conditions. Also no consideration was given to production of the future engine or eventual unit cost.
None of these system features and attributes were settled through the early prototyping, nor was that intended. Instead these requirements could be characterized or optimized only by system-level platform integration and a complete design and verification process. Our engine prototypes did their job as intended — they demonstrated key system capabilities — but they were only an intermediate step to a final design. The prototypes as we operated them would never be accepted as the configuration of the completed engine design. They were truly the first of their type, and much more work lay ahead.
Early Decision-Making
So what are the full-up system articles used for development test and verification at the end of EMD? They are not, in the true sense, “prototypes.” They are, in fact, full-up test articles that have all the features of the system design, but may not necessarily be made using mature production processes. These systems go by several names: qualification test articles, first articles, engineering development models, or EMD test articles. They are much more than prototypes and calling them that is a bit of a misnomer. As shown with previous examples, prototypes were used to demonstrate some, but not all, system requirements. In contrast, these EMD test articles must, by definition, meet all the requirements of the specification(s) in order to verify the final system design. In a perfect world, they would be made on the same line as the production articles, but they need not be. The key condition is that they need to work like production articles to be verified against the specification. The next step after verification is the fabrication of Low Rate Initial Production articles to be used for operational test. Those must, also by definition, be production representative.
Prototyping and competitive prototyping are very useful tools in design and acquisition, particularly for early decision making, but there are many other tools in the system-development process.
The author of this article, which first appeared in the March-April issue of Defense Acquisition magazine, is Brian Duddy, a retired U.S. Air Force lieutenant colonel, now a professor of Program Management at the Defense Acquisition University, where he teaches Program Management and Systems Engineering. The author can be contacted at Brian.Duddy@dau.edu.
