I have some firsthand experience with aerospace programs and dabble in writing the history of them as well. In my opinion, the basic problem with most programs that overrun schedule and budget is that the winning contractor has to way over-commit, usually but not always knowingly, to a combination of performance, capability, schedule, and budget requirements.* A case in point, Northrop-Grumman competing with General Dynamics-McDonnell for the Navy's ATA program: the team that arguably knew the most about the program, N-G, refused to agree with the government's unrealistic expectations, and "lost" the competition for the A-12. The Comanche program is another example of a contractor team agreeing to a nonnegotiable set of fantastic requirements to win a competition. The VH-71 presidential helicopter is another, although my understanding is the requirements got even more ridiculous after contract award. The procuring authorities are complicit in these fiascoes, either from ignorance of the degree of difficulty, the presumption that the contractor will somehow pull a rabbit out of a hat, or the belief that the contract protects them from responsibility for signing up to an undoable program.
There is a Vader aspect to this, both in the competition and the subsequent management of the program. One clue is when your well-reasoned statement based on hard-earned knowledge and experience that something is too hard is met with the response, "A good engineer/manager could get it done; if you say you can't, you must not be a good engineer/manager and I'll replace you with one who says he can."
Some people think McNamara was an idiot. Those who cite the F-111 program as an example conveniently forget that he was also responsible, to pretty much the same degree, for the Vought A-7, which was a joint service program with a fixed price development and production contract. The Corsair II's first flight, development, and qualification were completed ahead of schedule; the cost overrun was insignificant; and the performance guarantees, with only one exception that wasn't critical, met. What was the difference between it and the programs that suffered schedule and cost overruns? For one thing, it was not a great leap forward in speed (in the very beginning, it was to be supersonic) or systems capability (mostly off the shelf stuff) and therefore within the design state of the art. Better is the enemy of good enough.
The optimism of engineers and their drive to meet a challenge can also be a problem. When I was a boy, I read a short science-fiction story that made an impact on me. It concerned two warring worlds that finally agreed to a nonlethal competition to determine which was superior. Each would show the other its most advanced technology. Each would have one year to duplicate and demonstrate the opponent's capability. At the end of the year, both were successful in doing so and also astonished that the other had been able to duplicate their technology, because in both cases their challenge demonstration was faked.
Another trick to a successful program? Don't sign up for one that is over constrained, over managed, and under budgeted by the government. Ed Heinemann made weight on the A3D and A4D programs in part because he had a contract and a working arrangement with BuAer that allowed him a lot of latitude in not meeting every specification requirement. My impression is that some of Kelly Johnson's successes benefited from the same approach.
*Marketers and operators are a dangerous combination in this regard. The former want to sell a program; the latter, the highest possible probability of success in their mission, particularly if it involves getting shot at. Between the two of them, particularly when the marketers from two or more competitors are vying to outdo each other in capability claims, a set of requirements is established independent of the time and money available for the program.
