Does anyone know of a published report of empirical research that claims to have proven the effectiveness of contractual incentives? I'm talking about the predetermined, formula-type incentives described in FAR 16.402, or award-fee incentives.

I am not interested in "success stories" or anecdotal reports about individual procurements, but research based on the analysis of a statistically valid sample of the outcomes of procurements.

I am already aware of the Harvard, Rand, and LMI studies of the 1960s and 1970s. I am also aware of the GAO study of FPI contracts of the early 1980s.

I am looking for reports published during the latter part of the 1980s or during the 1990s.

By Larry Edwards on Saturday, September 30, 2000 - 03:44 pm:

What were the conclusions of the GAO study on FPI?

By Vern Edwards on Saturday, September 30, 2000 - 07:47 pm:

Larry:

At request of Congress, GAO reviewed fixed-price incentive contracts to determine (1) whether they saved the government money and (2) whether they worked as they theoretically should.

They told Congress, "The first of these questions cannot be definitively answered without doing comparative tests. Such tests would involve paired procurements of identical items, at the same time, under identical conditions using a FPI contract and a separate firm fixed-price contract."

In an attempt to answer the second question they tried to determine if there was any statistical correlation between the cost-sharing ratio and achievement of a contract's target price. If incentives work as they should there should be a correlation between the size of the contractor's share and the cost outcome. But GAO reported that it could not find any apparent relationship between them. They concluded that, "These findings do not support that part of incentive theory which holds that as the contractor's share ratio increases the contractor has a greater incentive to meet or underrun target costs."

See: Incentive Contracts: Examination of Fixed-Price Incentive Contracts, GAO/NSIAD-88-36BR, November 1987.

Irving Fisher of Rand Corporation made the same findings in the mid-1960s. He concluded as follows:

"These results are not consistent with the hypothesis that stronger incentives lead to greater efficiency and lower costs. Larger observed underruns seem to be generally unrelated to the profit incentive features of fixed-price contracts. The evidence indicates that these observed underruns originate, instead, from target costs that exceed the contractor's anticipated actual cost."

In other words, the contractors negotiated high cost targets and then underran them. See: A Reappraisal of Incentive Contracting Experience, Rand Corporation Memorandum RF-5700-PR, June 1968.

By bob antonio on Sunday, October 01, 2000 - 08:12 am:

I don't know if FPIF contracts can be evaluated simply. I do not consider any study that simply looks at cost-sharing by itself as being valid.

Here is one example that I posted somewhere and one person commented.

The target cost is $76 million; Target price is $9.7 milion; and ceiling price is $101 million.

Within 15 percent of the target cost, the share ratio is 95/5; outside of the 15 percent zone, the share ratio is 90/10. As can be seen, the ceiling price is 132.8 percent of target cost.

Bear with me on this. I will add more information if this receives responses. Does anyone have any opinions about this FPIF and what it means?

By Vern Edwards on Sunday, October 01, 2000 - 11:14 am:

Bob:

I'm not following you on this. I'm not sure what I'm supposed to see in the data you've posted.

(By the way, you said that the "target price" is $9.7M. Did you mean target profit?)

The Rand, GAO (and other) sharing studies sought to determine whether the FPI contracts functioned as predicted by incentive theory. In what way do you think that those studies fell short? As the GAO said, the only way to determine whether or not the FPI incentive actually saves money would be to conduct controlled experiments. But that would be awfully costly, don't you think?

What kind of study would you construct?

By bob antonio on Sunday, October 01, 2000 - 12:24 pm:

Vern:

Yes, target price should be target profit. I think I made that same error once before.

If a study looks at cost share only, then it makes a great deal of assumptions. Those assumptions can be quite significant and in my opinion make such a study invalid. For example, the study would assume that the target costs for tested contracts were all negotiated with equal skill. By looking at share ratios only, we also ignore the distance betwen target cost and ceiling price. That affects possible profit at various levels of overrun. Another item is the historical profit rate for the company. Perhaps a target profit is set too high during negotiations.

If I did a study of FPIF, I would start with the intent of creating several test groups. This requires an initial review of the negotiation memo and the structure of the FPIF before looking at the results. The groups--and these are my initial thoughts--are weak FPIF structure (90/10 SR, 130% CP above TC; tight FPIF structure 60/40 SR. 120% CP above TC; high quality negotiations; and poor quality negotiations. If I had about 100 FPIF contracts to work with, I might have different groups. However, I think this provides an idea of my concerns.

I am not trying to say that FPIF contracts are effective, I am trying to eliminate or reduce assumptions. If a study provides the results of weak FPIF structures, tight FPIF strucutres; good negotiations, weak negotiations, etc., I would be more interested.

One could say that if enough FPIF contracts were evaluated, chances are that we would get a good mixture of the groups I noted and the assumptions would be balanced out. I think that is being hopeful.

I do not remember reading the Rand or GAO report. So I do not know how either dealt with assumptions.

Using my example, what would one expect if a contractor is looking for a profit rate of 3 percent? This was an actual contract negotiated with a very lean budget and with a contractor that was totally dependent on the government for work. Suppose both the contractor and the government consider 3 percent to be an adequate profit?

By Vern Edwards on Sunday, October 01, 2000 - 12:58 pm:

Bob:

I don't think either the Rand or GAO studies dealt with those assumptions at all. It may be that the researchers thought that by looking at a sufficiently large sample of contracts they could even-out the effects of the factors that you mentioned. GAO looked at 573 FPI contracts.

Your study would be much more comprehensive, but it would cost a lot. I doubt that you'd get anyone to pay for it.

In any event, several serious scholars have concluded that contractual incentives that focus on short-term (i.e., contract-specific) profits miss the boat. They think that contractors focus more on long-term profits, and that contractual incentives are overwhelmed by competing incentives related to long-term outcomes.

I doubt that we'll ever get a definitive empirical study. Some people will continue to believe in predetermined, formula-type incentives (FPI(F), FPI)S), and CPIF), some willl not, and many won't care one way or the other.

Still, if anyone knows of any recent empirical studies that support claims of incentive effectiveness, I would like to know about them.

By bob antonio on Sunday, October 01, 2000 - 01:47 pm:

Vern:

I would tend to agree with the serious scholars.

Another study I would like to do is to look at incentive contracts over a period of decades. I always thought that their structures will vary according to budget availability. I think that in lean budget years for an agency, the FPIF structure may flatten out and take on the structure of a CPFF with a ceiling. Once you get an item half-built, it is more difficult to kill and overruns are endured more easily.

That is what was alleged to be the case in the example that I provided. There were allegations that both the government and contractor were negotiating at around 120 or 125 percent of target cost. These things can be abused and still carry the title of fixed-price contracts. Whenever I run into one, I enjoy analyzing it.

The only other studies I can remember are those DOE did on their performance contracts. They did use forumla incentives on some of them while on others it was a performance fee. However, I believe you are aware of them.

By Ramon Jackson on Sunday, October 01, 2000 - 04:09 pm:

I have some doubt GAO's "paired procurements of identical items" is required to obtain significant information. It would be nice, just as the identical armadillo quads are nice for some medical testing, but science often must deal with nature where such set piece testing is impossible. Instead the information must be painstakingly extracted from statistically significant numbers of subjects.

Is there enough data recorded about contracts and is it preserved to enable such studies to take place in the "contractual wild" rather than in paired procurements? Probably not. It might be worth making an effort though considering how much is spent based on hunch and legend.

I'm unwilling to reject the possibility of incentives being useful. I'm also unwilling to just accept them as effective. What is objectionable is the lack of empirical evidence. I can see where incentives could encourage added effort to do something. I also suspect the prospect of long term relationships might be effective as would many alternative strategies. Sound, effective past performance use would seem to be possibly as effective in this respect as individual contract profit incentives.

It is frustrating that we probably don't spend a fraction of the dollars poured into such schemes to gather, preserve and study the information required to provide empirical evidence of their effectiveness.

By bob antonio on Monday, October 02, 2000 - 08:11 am:

Vern and Ramon:

The cost for the study I noted is not prohibitive. Actually, it would be less expensive than some done in the past. Additionally, there are some add-ons that should be tested in the early stages of design.

The design of the study is relatively easy. I already have the work designed in my head. If such a study was supported by the Department of Defense, the results should be available during the Summer of 2001.

By Vern Edwards on Monday, October 02, 2000 - 08:17 am:

Bob:

So--are you going to pursue it?

By John Ford on Monday, October 02, 2000 - 11:17 am:

This is an interesting discussion. Bob, in your research design, I would hope you would include the impact TINA has on a contractor's abilility to cut proposed costs significantly so as to make an incentive contract a true incentive. One would think that if a contractor has disclosed current, complete and accurate cost or pricing data prior to award, that the opportunity for cost control may be somewhat constrained.

By Vern Edwards on Monday, October 02, 2000 - 12:01 pm:

Bob and John:

This is only tangentially related to what we've been talking about, but if you want to read an interesting discussion about incentives, go to www.findlaw.com and look up the following U.S. Supreme Court decision: United States v. Bethlehem Steel Corporation, 315 U.S. 289 (1942).

The case is about 17 cost-plus-incentive-fee contracts for shipbuilding that were awarded by the Navy during WWI (!). The contracts included a 50-50 share provision and the contractor ended up making an average of 22 percent profit. The Government thought that was too much and tried to get out of paying it, making several arguments, the most interesting of which was that it negotiated the contracts under duress.

The case took 22 years to get to the Supreme Court. The Court found in favor of the contractor. Justice Hugo Black wrote the majority opinion, but don't miss the dissenting opinion by Justice Felix Frankfurter. William O. Douglas's separate opinion, in which he joined the majority, is also interesting from a technical point of view. It discusses the distinction between contracts entire and severable.

This is a pure contract law decision and easy to read, with no esoteric constitutional issues, but with moral and patriotic overtones. (When you read it, think about the fact that it was written slightly more than two months after Pearl Harbor.)

The facts in the case foreshadow some of the Congressional concerns about the use of incentive contracts that resulted in the passage of the Truth in Negotiations Act of 1962.

We could have a heck of a debate at this site about who was right in that case, Black or Frankfurter.

By bob antonio on Monday, October 02, 2000 - 12:17 pm:

Vern:

Unfortunately, I have a full-time job. However, I believe it is doable--complex but doable. I see two initial problems with the analysis. They are both significant but that is where DoD needs to be involved.

First is access to information and that would include several DD 350 runs; access to contract files, and access to contracting staff.

Second is having the right qualified people working on the study. I think experienced procurement analysts from DoD would be fine. They would have to evaluate the quality of at least two negotiations--the one where the FPIF structure is set and the negotiation of final cost. They also would have to think as much alike as possible. Any modest changes that required pricing during the contract probably could be ignored. However, large ones would need to be reviewed. As you can imagine, this is the most time-consuming and complex part of the work.

John:

Yes, that would fall under the quality of negotiations.

By Vern Edwards on Monday, October 02, 2000 - 02:26 pm:

Bob and John:

What do you think about the quality of the negotiations in the U.S. v. Bethlehem Steel Corp case?

Bob:

You're about to get some help on your job. I just read in the September 19 Federal Contracts Report that the number of protests to the GAO has dropped so low that they're going to reassign some of the lawyers from the bid protest to the audit side of the house.

By Fred Weatherill on Monday, October 02, 2000 - 04:28 pm:

Vern,

You might want to read "On contracting for uncertain R&D" by Rajeev K Goel, in Managerial and Decision Economics, Vol 20, issue 2, 1999. The author is with the economics dept at Ill. State U. email rkgoel@ilstu.edu. You can access the abstract at www3.interscience.wiley.com/cgi-bin/abstract. It is a lead that may lead some where. Good luck.

By bob antonio on Tuesday, October 03, 2000 - 08:59 am:

Vern:

I quickly read the case you noted. Years ago, I was evaluating a "cost realism" analysis at a major agency. One of the agency evaluators provided the offeror's basis for its learning curve. The agency evaluator pointed out that the contractor's cost proposal was based on the premise that "the more we learn, the dumber we get." I think that may be an exact quote because it was hard to forget.

It appears that the shipbuilding cost estimate had some of those features. I quit reading it after I saw the following. The cost was based on the building of one hull. Under the contract, at least 19 hulls would be built. Next is the telling quote from the case.

"The next item was the cost of labor, which on hull 253 amounted to 38.75 cents per hour. Brown assumed that wage scales would rise to 50 cents per hour, an increase of 29%, and that efficiency would decrease about 30%."

So they estimated that they will have negative learning as they build more hulls. I like the technical evaluator's comment from my past--the more we learn, the dumber we get.

I think W. C. Field's could provide a simple analysis of this contract. "Never give a sucker an even break."

By Vern Edwards on Tuesday, October 03, 2000 - 09:13 am:

Fred:

I found the article. If you haven't read it, I'll let you know if it says anything interesting.

Vern

By Vern Edwards on Tuesday, October 03, 2000 - 09:27 am:

Bob:

I'm not sure it was a learning curve issue. Keep in mind that in 1917 we had just entered the war and the demand for manpower and materials was going to create shortages, driving prices up. I think that was the main reason that Bethlehem didn't want to accept a lump sum price and made such pessimistic estimates of cost. They were afraid of what would happen to them under a fixed-price contract in a time of rapidly rising prices.

The Navy's mistake was in the deal it negotiated for fee. Indeed, what struck me was how little negotiating the admiral and his colleague did. Did you get to the transcript of the Bethlehem negotiator's recounting of one of the bargaining sessions? A smarter pair of negotiators would have covered the contractor's costs and then kept the fees at rock bottom levels, pointing out that it would not look good if Bethlehem made a lot of profit on war work.

By bob antonio on Tuesday, October 03, 2000 - 09:54 am:

Vern:

I agree. Today, I think we would take the labor and material part of the costs and provide an economic price adjustment. It appears that was included in the cost.

I looked at the first part of Frankfurter's dissenting opinion and saw some of the negotiation discussion. Not much on the government's part but some whining. I did not get to the quotes and will try to do that at lunch. I stopped just above the quotes with the quote I posted. I think the learning curve issue may be a factor. It looked like the basis for cost was one hull--253. If that hull was built by itself and they would have 19 to build under the contracts, I think they could have achieved some efficiencies. However, they proposed inefficiencies. Of course, I read this quickly and may have misread something.

By Linda Koone on Tuesday, October 03, 2000 - 10:29 am:

Vern:

I enjoyed reading the decision.

I agree that the government negotiators, from what can be surmised from the record, weren't effective. We'd be in a fine mess if every government negotiator bent to the initial demands of a sole source contractor.

I have to side with the majority opinion in this case; however, imagine the precedent that would have been established if the Court would have ruled for in favor of the government. What a ruling to fall back on in cases of shoddy negotiating!


What I really find surprising is that either side pursued this as long as and to the level that they did. I would be embarrassed to have this story told, whether I was affiliated with the government or with Bethlehem Steel.

By bob antonio on Tuesday, October 03, 2000 - 12:15 pm:

All:

This case is a must read. Here are some quotes.

"Although there are many cases in which an individual has claimed to be a victim of duress in dealings with the government . . . . This, so far as we know, is the first instance in which government has claimed to be a victim of duress in dealings with an individual." --Justice Black.

-------------------------

"That was a very rough figure. Whatever else I put on I put on because I knew I was going to have to trade the final contract out with [government negotiator] and I knew I would not get what I asked for, and if I did not ask for more than I expected I would not get out where I wanted to be." --Contractor representative.

-------------------------

"Q Now I am asking you whether with respect to those vessels [government negotiator] submitted any price of his own?

A. No, Sir.

Q. Did he ask you how your estimates were made up?

A. No, sir.

Q. Did he ask you to justify your estimate in any way?

A. No, sir.

Q. He just objected to them repeatedly and said they were too high?

A. Yes, sir.

Q. But did not ask you to justify them?

A. No, sir."

---------------------------

"The district court concurred in the proposition that the absence of fraud made the contracts invulnerable. Buts its conclusion is contradicted by its findings: 'The managers for the contractor adopted the famous Rob Roy distinction who admitted he was a robber but proudly proclaimed that he was no thief. The contractor boldly and openly fixed the figures in the estimated cost so high as to give them the promise of large bonus profits. The managers for the [government] knew that the estimate was high and why it was made high and so protested it. The reply of the contractor's manager was. We will take the contract with this promise of bonus profits incorporated in it but not otherwise. You take it or leave it. Whatever wrong there was in this may have been the wrong in a daylight robbery but there was no element of deception in it." --Justice Frankfurter quoting the District Court

The cost contract was proposed with contingencies in the cost estimate. Oh well.

Here is the case U.S. V. BETHLEHEM STEEL CORP

By bob antonio on Tuesday, October 03, 2000 - 12:24 pm:

As Vern noted, the case was argued on December 9, 1941 and decided on February 16, 1942.

By Kennedy How on Tuesday, October 03, 2000 - 12:27 pm:

I read the case, and I found it to be an enjoyable read, mainly because it was on a topic that I understand (though I will say I do like reading Supreme Court decisions!).

I thought that the Government's support staff was poor, and while the dissenting opinion pointed out that the "contracting officer" had no shipbuilding experience, that doesn't necessarily mean the Government cannot strike a good bargain, as we've touched upon in other discussions regarding source selections and negotiations. The last time I did a major systems negotiation, I never pretended to know more than my staff; I had a working knowledge sufficient to understand what they said, but when it came down to negotiations, I let them do their jobs. If their performance was poor, or non-existent, then it'd be shame on me. The fact that a resulting contract is a bad bargain for the Government is no reason to void a contract; we weren't prepared or able, we should suffer the consequences for it.

It's always been difficult to negotiate from a position of weakness, something that's basic in the Government's position in nearly everything we do (at least from my perspective). That's cynical, I know, but I learned early on that if the Government had a need, that need would almost always be there, and the contractor would be in a good position, especially if he was a sole-source. And, on top of that, if the contractor ever figured out that there was a much cheaper way of doing it, we'd want all of the savings, rather than splitting the cost savings with him.

By Ramon Jackson on Tuesday, October 03, 2000 - 02:35 pm:

Kennedy, why "negotiate from a position of weakness, something that's basic in the Government's position in nearly everything we do" when it has so much power and special contract position as demonstrated by termination for convenience?

Could this have anything to do with so often not being well prepared? With not spending the time using its own information resources? Not paying the price to have at its table real experts?

By Kennedy How on Wednesday, October 04, 2000 - 01:01 pm:

Ramon, when I say "position of weakness", I say that because the Government hardly ever walks away from a deal and refuse to procure an item, because it has a bona-fide need for said item. Given that, we're obliged to fulfill that need, even though the smart thing to do is to "walk away". That is what private industry will do, in my opinion, and try to find another way to fill that need. Or, they might come up with a unique way of completing the deal, ways that can't be accomplished due to Government regulations.

I can have the most prepared and best people on my side to solidly present our position, but if the other side isn't willing to be reasonable, we aren't going anywhere. On the flip side, my management is asking me why said item isn't on Contract yet.

I had a small deal once, for an FMS customer. It was for an item that is used on an obsolete vehicle. I was able to find only one business who had it, and the price was relatively outrageous. Nobody within the Government was willing to sign off on the deal, but yet the FMS customer OK'd the price. I couldn't cancel the requirement because the FMS customer had a bona-fide need. I basically sat there for 2 weeks until I decided to provide the FMS customer the name of the business in question so they can contract directly. And, even then, I had to report weekly why the contract action hadn't been completed until the requirement was cancelled internally.

I'll be the first to admit I'm somewhat anal in my preparations for contract actions; I hate making mistakes. That's my nature, and in my career, I've learned to be really prepared and have my bases covered so that I can get things done the first time around.

By Ramon Jackson on Wednesday, October 04, 2000 - 07:02 pm:

Kennedy, that is a good point. Government tends to overvalue its requirements. Many are actually vital and urgent, but many more, objectively viewed, are not. There is a lot of pretense here.

The lack of preparation I was thinking of deals with the fact that as a huge enterprise the government has little in the way of an enterprise wide information system in the field. Each agency often works in its own little cubicle. Even when interested in gathering the necessary intelligence to be in a strong position the teams will find information gathering difficult. There is neither need for this or reason -- just inertia.

By Fred Weatherill on Thursday, October 26, 2000 - 01:06 pm:

Vern,

You may want to read Federal Register: October 24, 2000 (Volume 65, Number 206) Page 63628. It is a call by OFPP for information on the use of incentives as a part of an OFPP effort to "fundamentally examine the manner by which the Government develops and applies incentives to its contractual vehicles..." It may be another lead.

By bob antonio on Thursday, October 26, 2000 - 01:26 pm:

Fred:

The article is on the "Today's News" and "Today's Grants" section from earlier in the week. The daily information is posted every work day.

I added a thread here about it.

By Fred Weatherill on Thursday, October 26, 2000 - 02:51 pm:

Bob,

Thank you. I am still learning to check Wifcon first!

By Anonymous on Thursday, November 02, 2000 - 07:37 pm:

Apparently OFPP is busy looking for new incentives. See Procurement office seeks input on contract incentives in the on-line GovExec.

By JSA on Thursday, November 02, 2000 - 08:31 pm:

Anonymous:

The item was posted to this site on October 24, 2000. It is noted in the post below.

See the "Contracts--General Discussion" thread subject "OFPP Request for Information on Contract Incentives."

It also was posted to the "Today's News" page last week. See the "Last week" section of the "Today's News" page.