Eric:

I believe in the marketplace of ideas. I think that good ideas win through eventually, given time and persistence. I'm not too old, I have a lot of persistence, I can write reasonably well, and I talk to a lot of people in my classes, on the phone, and via e-mail.

Don't assume that the route to "victory" is via a direct approach to the FAR Council. Most procurement innovations start in the field, among working level folks, long before they receive acceptance by policymakers or mention in the regulations. I persuaded people to use oral presentations long before the FAR Council decided to include it in Part 15, so I'll use oral presentations to make my point.

In a February 14, 1994 telephone call, I tried to persuade an Air Force friend of mine, Gail Vranicar, to evaluate oral presentations instead of written technical and management proposals in a source selection. Gail was skeptical, but asked for more info. I wrote a long memo, dated February 25, 1994, explaining the idea in depth. The memo was entitled, "Proposal for the evaluation of oral presentations instead of written technical and management proposals in source selection."

I then asked Ralph Nash to address the question of whether the technique would be legal. He wrote an article in which he concluded that it was. See: "Oral Presentations in Competitive Negotiation: Could They Be Used in Lieu of Written Proposals?" Vol. 8, The Nash & Cibinic Report, ¶ 48 (August 1994).

That article didn't elicit any response from either the DAR Council or the CAA Council, but it did elicit this from OFPP Director Steve Kelman:

"You had a discussion of oral presentation in the August issue of the Report. You asked for reader reactions. This reader reacts by saying that this is a wonderful idea. Let's try to see if we can get some agencies to experiment with this. I will bring it up for discussion at the next meeting of the Federal Procurement Council."

See 8 N&CR ¶ 49 (September 1994).

Ralph and I started preaching the gospel of oral presentations in our classes and got some agencies to try it. Nuclear Regulatory Commission tried it first. They (and their contractors) liked it. (NRC won a Hammer Award for this.) By May 1995 Ralph was able to write: "We have never received as much comment on an article as that received on Oral Presentations in Competitive Negotiation: Could They Be Used in Lieu of Written Proposals?" 9 N&CR ¶ 29.

That summer I persuaded BGen. (then Col.) Craig Weston, Director of the Space Based Infrared Systems Program at USAF's Space and Missile Systems Center, to use oral presentations instead of written technical proposals to award an $80 million development contract. He did it and was very pleased with the result.

In March 1996 the Department of Energy and OFPP published their oral presentation guide, which cited my memo to Gail and one of my articles in N&CR. That same month, on March 4, 1996, BGen. (now Major Gen.) Richard Roellig, then Director of Contracting at HQ AFMC, issued a memo entitled, "Oral Proposals in Lieu of Written Technical Proposals," in which he said: "One of the initiatives holding promise to reduce the time and other resources to conduct formal source selections is the use of oral technical proposals in lieu of written technical proposals... . This initiative, while not mandatory for all acquisitions, should be implemented in those source selections where it makes sense... ."

Finally, in September of 1996, the FAR Council included coverage of oral presentations in its first draft of the FAR Part 15 Rewrite. You know the rest.

I recall many meetings and presentations in which disdainful agency lawyers and some skeptical COs said that oral presentations would result in massive numbers of sustained protests because the GAO would consider oral presentations and Q&A sessions to be discussions. I especially remember a session at the Naval Sea Systems Command in Crystal City. I took a brutal beating from some Navy lawyers. Fortunately, Paul Buonaccorsi, the head of contracts at NAVSEA, was determined to use orals and he pushed it through. NAVSEA was one of the pioneers.

Even my friend Prof. John Cibinic thought the GAO would consider oral presentations to be discussions. But I stuck to my guns. To date, there hasn't been a single sustained protest on that basis, despite the fact that oral presentations have become a standard technique. In fact, there have been very few protests about oral presentations on any basis.

In 1987 I told an audience of DLA contracting officers that they could use best value source selection to award contracts based on nothing more than past performance and price, with no technical or management proposal. They laughed at me. (It seemed a wild idea in those days.) But one of them took me up on it, used the language that I provided for his RFP in the course textbook, and won the ensuing protest: Corvac, Inc., B-244766, Nov. 13, 1991, 91-2 CPD ¶ 454. Chuck Clark at OFPP put the RFP language in the OFPP policy letter on past performance. You can read part of that RFP language, slightly edited, in FAR 42.1501.

Search the GAO decisions for LOCAR or "level of confidence assessment rating," a technique which I described in my 1995 monograph, How to Evaluate Past Performance: A Best Value Approach, 2d ed., pp. 49-56, available from Commerce Clearing House. Also, did you notice that AFFARS 5315.305(a)(2) now refers to "confidence assessments"? Compare the USAF rating definitions in AFFARS 5315.305 to my description of possible adjectival LOCAR scales in my monograph, pp. 49-52.

I believe in what I call the Paul Newman philosophy of bureaucratic politics, spoken by that sage to Joanne Woodward in the movie The Long Hot Summer: "Life is very long and full of salesmanship, Miss Eula. You might buy something yet." Paul and Joanne got married right after they made that movie.

Dee Lee is now Director of Defense Procurement, and I have found her to be an open-minded person. (She reads The Nash & Cibinic Report.) We'll have a new OFPP Director next year, and I have hopes that he or she will be open minded, too.

Eric, I think you give windmills more credit than they deserve. Go ahead and bet on them, but don't bet your house.

Vern

By Eric Ottinger on Thursday, June 1, 2000 - 05:59 pm:

Vern,

Proposal Risk is defined in the Regulation. That is not a guide.

Please clarify. Which term did you find objectionable? “Authoritative?” “Personal opinion?” or “Professoriat?”

I find it interesting that you (and presumably your colleagues) are going to mount-up again. I’m putting my money on the windmill.

I don’t think this has anything to do with simplification or plain English. In practice, it would be an arbitrary limit on the PCO’s discretion.

If our readers want to see the state of the art in streamlining they should read the new AF policy. (This doesn’t mean that I agree with it 100%. There are a couple of excessively innovative ideas in there; which will, I think, go away in time.)

(I know that there are some serious issues here. We will be debating these issues as long as we do procurement. We spent a lot of energy last year pounding this into fine dust. I’m not really motivated to do it all over again.)

Regards,

Eric

By Vern Edwards on Thursday, June 1, 2000 - 12:47 pm:

Christine:

The Air Force does like to write.

A search for "source selection" in the DOD Acquisition Deskbook produced the following results under the heading Mandatory Reference Library Documents:

33 DOD documents,
7 Army documents,
9 Navy documents (common to Navy and Marines),
15 Navy-specific documents,
2 Marine-specific documents, and
61 Air Force documents, and
7 DLA documents.

It produced these results under the heading Discretionary Reference Documents:

74 DOD documents,
80 Army documents,
22 Navy (common) documents,
26 Navy-specific documents,
18 Marine-specific documents,
185 Air Force documents, and
3 DLA documents.

You can't use these statistics to criticize the Air Force. My search wasn't very scientific -- many of the references undoubtedly just mention "source selection." One should judge the quality of the documents, not just their quantity. And the relative lack of regulation and guidance is not necessarily a good thing. But I think that the numbers do suggest differences in what you might call the "acquisition cultures" of the four military services. The Air Force does tend to publish more acquisition regulation and guidance than the other services and seems to do so in order to exercise greater control at higher echelons. Whether that's good or bad is just a matter of preference in management styles.

As another point of interest: In FY99, DOD obligated $135 billion on 5.8 million actions. Of those numbers, the Army obligated $32 billion on 514,727 actions; the Navy obligated $40.6 billion on 537,364 actions (this number includes the Marines); the Air Force obligated $36.4 billion on 908,224 actions; and DLA obligated $10.3 billion on 1,595,431 actions.

Thus, the relative amounts of contracting activity and obligations do not explain the relative amount of regulation and guidance. It's a culture thing.

Vern

By Christine on Thursday, June 1, 2000 - 11:22 am:

Thank you. I have this information at home I just could not remember off the top of my head. We have performed more than one and there were different verbiage in various buys. The AF is updating regularly.

By Vern Edwards on Thursday, June 1, 2000 - 11:17 am:

Christine:

Your answers make sense to me.

The current USAF rating terminology for what it calls a "confidence assessment" is:

Exceptional/High confidence
Very good/Significant confidence
Satisfactory/Confidence
Neutral/Unknown confidence
Marginal/Little confidence
Unsatisfactory/No confidence

Vern

By Christine on Thursday, June 1, 2000 - 10:35 am:

Vern,

I absolutely believe opinions should be used. However I do not think they should be used in our particular setting. In one of the buys the customers had previous experience with three of the contractors. In each case different people had worked with each contractor. No one person dealt with all three. This way we relied on the past performance evaluations performed on the contractor not only from outside references but inside ones as well. Even though it was not required we performed annual past performance evaluations on contracts exceeding as little as $100,000.00. Too often I have seen personality conflicts where they should not be. There was one protest I read a while ago and an employee’s disgruntlement with his CO resulting in a lower performance rating. Completely unacceptable in my mind. In short we had two parts, 1. mostly subjective (the individual performance evaluations) and 2. most objective (the overall evaluation documents, verbiage and spreadsheet).

2. Our technical team was comfortable with this scenario. This enabled them to feel they were making an informed decision without prejudice. I realize we can right now go to the incumbent if the performance evaluation and price are done correctly. I just believe (and shared by the technical team) in the environment of fair and equal competition we needed a way to show how we evaluated fairly without prejudice. Due to the subjective nature of the evaluators a satisfactory rating is sometimes given because they can't think of anything to say. A satisfactory rating will kill an evaluation of past performance. I know it is not supposed to like that but in reality an excellent far outweighs a satisfactory. I sat down with the technical team and they were comfortable with this type of evaluation and did not feel like I was keeping things from them. Their main questions were on size and scope of the contracts to make sure they were rating on relevance. I support this type of buy but it is very emotional for contractors when they lose and I think we owe it to them to treat them as fairly as possible. All their past performance should be known to them and when I can explain this was exactly what they were evaluated on with no further prejudice it strengthens the Government position. I am not a lawyer I was just trying to establish a fair playing arena. I believe it worked.

3. In the first part of the evaluation everyone read the through the censored documents and gave a rating on performance risk. We used the AF's PPTO guide and used the Confidence ratings. I can't remember the exact verbiage because it has changed to many times and I am now trying to work from the other side. I think it went Superior, Very Good, Satisfactory, Neutral, Poor. I think Satisfactory and Neutral were in the same category. After everyone ranked them privately we discussed each offerors strong and weak points and assigned them a group rating. Each proposal assigned a letter and in the second part of the evaluation we show the price ranking for each of the proposals compared to the risk rating and decided which represented the best value to the our needs.

By Vern Edwards on Thursday, June 1, 2000 - 09:59 am:

Christine:

Don't worry, we'll be gentle. Eric and I are mean only to each other. Usually.

Two questions for you: Do you think that its wrong to take personal opinions into account when evaluating an offeror's past performance?

Also, do you think that any members of your team may have resented the fact that you were keeping the offerors' identities secret? Do you think it possible that some persons might interpret that kind of secrecy to be an indication of a lack of trust?

Finally, how did the source selection authority use the rankings when making tradeoffs?

Vern

By Christine on Thursday, June 1, 2000 - 09:03 am:

Janice,

One of the ways I have dealt with the evaluation of non-price factors turned out nicely and was repeated in later evaluations. One of the most important perceptions of the evaluation (in my mind) is the complete perception of objectiveness. We (I was with the AF) of course used the PPTO guide and relied mostly on past performance when purchasing service support. There were not many technical factors. We relied on current relevant information and while the evaluation documents were written, I removed all references to places, people, names of any kind. I contacted most of the references by telephone (all except where the sheer numbers made it impractical) and used specific quotes from references. No personal opinions were included. When all the information was gathered. A technical team met (CO, CA, and customers) and we ranked the performance risk. The only person in the room who knew who the offerors were, was me (the CA) and to some extent the CO. This served two purposes. The ranking were done without personal feelings coming in to play and it made the customers understand the importance of filling out frank honest past performance evaluations. In our procurements we normally did not receive bad performers but we did not always receive excellent ones either. We had budgets to meet and sometimes the excellent service was not worth the price compared to good performance at a better price.

To Vern and Eric,
I am one of the less experienced and I am trying to take part in discussions and hope you will realize this when reading my response. I get lost reading your discussions sometimes and I for the most part enjoy them. They are rather intimidating.

By Vern Edwards on Wednesday, May 31, 2000 - 09:52 pm:

Well, Eric, I can see that you haven't learned to accept an apology graciously.

My initial response to Janice was an explanation, not an argument. I didn't suggest that she do anything in any particular way. I merely tried to explain the concept of risk/price tradeoff analysis in response to her inquiry. (And a very good explanation it was.) The careful, thoughtful reader would have noted that the very last thing I did was to refer her to the Air Force's guidance. (By the way, the Air Force risk assessment guidance is just that -- guidance. The booklet expressly states that it's not mandatory.)

I didn't withdraw my comment about the time you spend in your lawyer's office. But I will if you will withdraw your comment about "the professariat." What I apologized for saying that you got the facts wrong about their recommendations to the FAR Council.

I didn't "agree" that Janice didn't have to identify a specific event, I said so in the first place. My original comments pointed out that while formal treatises on risk analysis required the identification of a specific event, agencies usually don't bother to do so in a source selection.

[By the way, here's how the DSMC handbook, Risk Assessment Techniques (July 1993) defines risk on page II-2: "In this handbook, 'risk' is the term used to denote the probability of an event and its consequence." The handbook points out that formal risk assessment techniques are quantitative. See pages II-1 through II-12.]

With regard to "extraneous baggage," let me remind you that this site is called "The Open Forum." It's a place for sharing ideas. I assume that everyone who reads my comments is intelligent enough to think about them and arrive at their own conclusions or do some independent research. I realize now that for some people any idea besides "ask your lawyer" is "extraneous."

Finally, I certainly do plan to refight my argument about the word proposal now that the FAR Council has announced its intention to rewrite the FAR in "plain language." In fact, I'm looking forward to many more skirmishes.

Speaking of ideas: How would you define "contract action"?

By Eric Ottinger on Wednesday, May 31, 2000 - 05:52 pm:

Vern,

I thank you for graciously withdrawing your gratuitous comment about spending time at the lawyer’s office. If you meant spending time on a Rule 4 file, I should note that I have only had that opportunity once, and it wasn’t my source selection. I was filling in for a colleague who had been hospitalized.

If Janice ignores the guidance in her regulation and uses the guidance that she picks up in the Open Forum in place of the AF policy, she might have some opportunity to spend time working on a Rule 4 file. The Comp. Gen. is open to a variety of approaches, but it does expect contracting people to follow the direction in their agency regulations.

Thanks for the clarifications. Your argument carried a lot of extraneous baggage, and Janice needs to understand that.

I’m glad you agree that Janice is not required to identify “a specific event” or “the [specific] harm." If Janice used this theory to brief her evaluators, she could create some real problems for herself.

If you wish to use the Open Forum to refight old arguments or tilt at windmills, suit yourself. But let’s be careful not to mislead our less experienced readers.

Anyway, thanks for the clarifications.

Eric

By Vern Edwards on Wednesday, May 31, 2000 - 02:26 pm:

Eric and Janice:

I fear that I've had a senior moment and that I have falsely accused Eric of getting his facts wrong.

One of "the professariat" has reminded me that some of of us critics of the first draft of the Rewrite (September 1996) had argued for using proposal throughout the FAR as it was used in FAR 2.101, i.e., as denoting a set of promises.

After it became clear in the second draft of the Rewrite (May 1997) that the FAR Council would not accept our recommendation, we (the professariat and I) then changed our argument to say that proposal should be defined to include an offer, other written information, and any oral presentation, but that discussions pertained only to the offer, i.e., the promises, and not to information about offeror capability.

I thought that only I had argued for universal use of proposal as offer, i.e., as promise, but I'm now sure that "the professariat" did, too. The articles and the copies of the comments to the FAR Council make that clear.

By the way, we still think we were right the first time.

Sorry, Eric.

By Vern Edwards on Wednesday, May 31, 2000 - 12:07 am:

Janice:

I'm not part of "the professariat," but I should add that in my personal opinion FAR Part 15 should use the term proposal the way that it's used in FAR 2.101, as offer, i.e, promise. But since it doesn't, I say that a proposal includes an offer, other written information, and any oral presentation. That is consistent with the use in FAR 15.101 and FAR 15.305(a). However, only the offer includes promises; the rest is capability information.

Eric is right that my view has been rejected by the FAR Council.

By Vern Edwards on Tuesday, May 30, 2000 - 09:36 pm:

Janice:

I see that Eric has decided to take one of his periodic swipes at academics, this time claiming that Air Force and other government policy makers ignored the recommendation of "the professariat" to use the term promise as a synonym for proposal. (I'm surprised that he has time to write these things given the time he spends at his lawyer's office.)

In any event, he's got it wrong. The "professariat's" opinion was exactly the opposite: proposal should not be used as a synonym for promise. Thank God the GAO doesn't use it that way. Unfortunately, the FAR does.

During the development of the FAR Part 15 Rewrite, certain persons urged the FAR Council to draft the Rewrite so as to make a distinction between proposals and promises. They did not make their recommendation to the Air Force, but to Steve Kelman, then the OFPP Administrator, in a private meeting about the FAR Part 15 Rewrite, and in articles that they wrote.

They pointed out that while FAR 2.101 does not define the term proposal, it defines offer and says that proposal is a synonym for offer that is used in negotiated procurements. It still says that. "[R]esponses to requests for proposals are offers called 'proposals'."

Now these members of "the professariat" explained to Dr. Kelman that in the American legal system offers are usually promises or sets of promises. (See Black's Law Dictionary and Restatement of the Law, Second, Contracts, 2d, § 24, Comment a., or any other book about American contract law. Just to be on the safe side with Eric, check Webster's Third, too). Since FAR 2.101 uses proposal as a synonym for offer, and since an offer is a promise, then it follows, even as the day follows the night, that FAR 2.101 is using proposal as a synonym for promise, albeit in a roundabout way.

However, FAR Part 15 uses the term proposal to include not only promises but other statements that are not promissory in nature, such as references for experience and past performance, key personnel resumes, plans of various kinds, etc. See, for example, the first sentence in FAR 15.305(a), which says, "Proposal evaluation is an assessment of the proposal and the offeror's ability to perform the prospective contract successfully." The persons who spoke with Steve Kelman would have preferred something like: "Proposal evaluation is an assessment of the offeror's promises and of the likelihood that it will keep those promises if awarded the contract." Better yet, they would have preferred that the sentence had been tossed out as unnecessary.

The reason that the distinction between promise and proposal was important to the professariat (shouldn't that be professoriat?) was that they wanted the term discussions to apply only to communications about the promises that offerors made that would set the terms of the contract between the Government and the contractor, and not to communications about other things, such as experience, past performance, and other nonpromisory statements of offerors. They would have liked to see FAR define discussions as "negotiations in which the Government asks or allows offerors to change their offers, (i.e., promises)." They thought that definition of discussions was consistent with Congressional intent in passing P.L. 87-653 back in 1962. (That was the statute that first required agencies to conduct discussions with offerors in a competitive range.) Their hope was that this approach would eliminate some of the confusion about what kinds of communications constituted discussions, and that this would eliminate some of the concerns about using oral presentations and the proposal evaluation.

They were eventually told by Dr. Kelman that he was unable to persuade all of the members of the FAR Council to adopt their suggestion, although it made sense to him.

It made sense to others as well. Despite the FAR Council's decision to decline the recommendation, many contracting officers have read the articles written by "the professariat" and have decided that their point makes sense. You can occasionally see the result in a GAO decision, such as Systems Integration & Research, Inc.; Presearch Inc., B-279759.2; B-279759.3, February 16, 1999 (denied) in which the GAO describes the Naval Sea Systems Command's RFP instructions to offerors as follows:

"Offerors were instructed to submit proposals in four separate volumes: offer (volume I); written capability information (volume II); supporting cost data (volume III); and oral presentation (volume IV). Id. § L-3."

You see -- the offer is identified separately from the other stuff.

In any event, I agree with Eric about one thing: You work for the Air Force, so do things their way.

By Eric Ottinger on Tuesday, May 30, 2000 - 05:49 pm:

Janice,

Here is the language out of the AF FAR Sup—

5315.305 -- Proposal Evaluation.

“(ii) Proposal risk assessment focuses on the risks and weaknesses associated with an offeror's proposed approach. Assessment of risk is done at the subfactor (or element, if used) level, and includes potential for disruption of schedule, increased cost, degradation of performance and the need for increased Government oversight as well as the likelihood of unsuccessful contract performance. For any risk identified, the evaluation must address the offeror's proposal for mitigating those risks and why that approach is or is not manageable. Note that if a combination of significant weaknesses leads to unacceptably high proposal risk, this is a deficiency in the proposal. (See FAR 15.301, Definitions.) Proposal risk shall be evaluated using the following ratings:

Rating. Definition

High. Likely to cause significant disruption of schedule, increased cost or degradation of performance. Risk may be unacceptable even with special contractor emphasis and close Government monitoring.
Moderate. Can potentially cause some disruption of schedule, increased cost, or degradation of performance. Special contractor emphasis and close Government monitoring will probably be able to overcome difficulties.
Low. Has little potential to cause disruption of schedule, increased cost or degradation of performance Normal contractor effort and normal Government monitoring will probably be able to overcome difficulties.”

Getting back to your question—

“I can understand a tradeoff for risk for R&D type efforts, but for service support why would you want anything other than a low risk. “

You might have an innovative approach, which you might find attractive. But the innovative approach might be higher risk than the less innovative approach used in the past. The idea sounds good but you are not entirely sure it will work.

You do a “trade-off” whether you quantify the trade-off or merely weigh two factors when you make the decision. In this sense, anything that is evaluated is “traded-off” in the decision process. (Otherwise, you have a go/no-go “definitive responsibility” factor.)

Note that a “Proposal Risk” evaluation is not required at the “Basic” source selection level. If your Counsel isn’t clear on that point, you might want to (tactfully) straighten him/her out on this point.

I don’t think that you will find anything in the AF source selection policy (or the Comp. Gen. cases) to the effect that you must identify “a specific event” or “the [specific] harm that would occur.” I would take the words “significant disruption of schedule, increased cost or degradation of performance” at face value and resist any advice to read these terms narrowly or restrictively.

I don’t think that you will find the term “promise” used formally as a synonym for “proposal” in the FAR, the agency Supplements or in any Comp. Gen. case. Accordingly, I would characterize this as “authoritative personal opinion from the professariat,” which the AF and other policy makers in the Government conspicuously chose to ignore.

I only raise this point because the new AF policy was written to be clear and simple. You will not do yourself any favors interpreting it in light of other theories (unless of course such theories are supported by Comp. Gen. or Court precedents).

Eric

By Vern Edwards on Tuesday, May 30, 2000 - 12:32 pm:

Risk/price may be the most fundamental tradeoff of all.

Risk is important because an offer (proposal) is just a set of promises. If all you evaluate is the value of those promises then you are failing to consider the possibility that the offeror might not be able or willing to actually keep its promises.

There are any number of reasons why an offeror might not keep its promises. It may have promised to do things that it lacks the capability to do. Unforseen events may make the promises impossible to keep. The offeror may be dishonest and have no intention of keeping the promises.

Technically, risk is a measure of the exposure to harm arising from the occurance of a specific event. In order to measure risk you need to identify three things: (1) a specific event, (2) the harm that would occur if the event happened, and (3) the probability that the event will, in fact, occur. But it is sometimes defined more simply as the probability that a harmful event will occur.

Study of source selection plans and RFPs shows that when agencies say they are going to evaluate "risk" what they really mean is that they are going to develop an opinion about the likelihood that an offeror will keep the promises that it made in its offer. In order to develop such an opinion they may consider factors like experience, past performance, understanding of the requirement, personnel qualifications, production capacity, etc. These are sometimes called "capability factors" or "risk factors."

Some agencies simply score such things and then combine the capability scores with the scores for other factors, such as product quality. Other agencies use "risk" scores to adjust the scores assigned to other factors. A search of agency source selection guidance and GAO protest decisions will show you the many different ways that agencies evaluate risk. One technique is the "level of confidence assessment rating," or LOCAR, which is used by some agencies. For an illustration of its use in a source selection read the GAO's decision in the matter of Moore Medical Corporation, B-261758, October 26, 1995.

Whatever method is used to assess risk, the idea is to consider the likelihood that an offeror will actually keep its promises before you make a source selection decision based on the value of those promises.

In order to understand the concept of risk/price tradeoff, consider the following illustration:

Suppose that an agency is going to evaluate offers to provide a service on the basis of quality, risk, and price, with quality and risk being equally important and significantly more important than price. The agency will score "quality" on a 0 to 100 point scale -- the higher the score the better the quality. The agency will score "risk" based on the offerors' experience and past performance on a scale of 0 to 1, e.g., 0, .1, .2, ... .8, .9, 1.0 -- the higher the score the lower the risk. The agency will use an offeror's risk score to adjust its quality score, to reflect the agency's assessment of the likelihood that the offeror will keeps its promises about quality.

Suppose now that the agency evaluates two offerors' technical proposals and gives Offeror A a quality score of 90 points and Offeror B a quality score of 80 points, indicating that Offeror A has promised better quality. These are the promised values of the offers. Suppose further that Offeror A's price is $1,000,000 and Offeror B's price is $1,100,000. Thus, the offeror who promised the better quality also proposed the lower price. At this point, Offeror A looks like the better value.

Now suppose that the agency considers each offeror's experience and past performance and decides that there is only a 50-50 chance that Offeror A will actually deliver on its promises about quality, but that there is a 90 percent chance that Offeror B will deliver on its promises. The agency thus gives Offeror A a risk score of .5 and Offeror B a score of .9. These assessments are entirely subjective.

The agency then determines the expected values of the two offers as follows:

A = Promised Value 90 x .5 = Expected Value 45
B = Promised Value 80 x .9 = Expected Value 72

Thus, although Offeror A promised better quality, Offeror B is expected to deliver better quality. Now comes the tradeoff analysis:

Offeror A: Promised 90, Expected 45, at $1,000,000
Offeror B: Promised 80, Expected 72, at $1,100,000

Which is the better value? That's entirely a judgment call. A decision to award to A could be interpreted as a decision to take a higher risk in the hope of getting better quality at a lower price. The agency would be trading off risk for quality and price. A decision to award to B would thus be a decision to pay $100,000 in exchange for a lower risk, i.e., to tradeoff $100,000 for a greater likelihood of getting good quality, even if not the quality promised by Offeror A.

A decision maker using such a scheme will have to keep in mind that the numbers associated with promised value, risk, and expected value do not represent "reality"; they just symbolize the summary findings and opinions of the evaluators. Also, Offeror A might actually deliver on its promises and Offeror B might not, but there is no way to know that at the time that the source selection decision must be made. That's where judgment comes into play. The risk assessment is one more piece of information to be weighed in making a decision.

The scheme that I've described above has actually been used and found acceptable by the GAO. See the Moore Medical decision, cited above.

No matter what factors and scoring schemes are used, virtually all agency risk assessments and risk/price tradeoffs are conceptually similar to the above illustration. It appears that most agencies use adjectival schemes to describe risk, such as low, moderate, and high, instead of numerical scoring. But that doesn't really matter. The GAO will accept any scheme as long as it is used in a rational manner.

Since your e-mail address indicates that you work at Bolling Air Force Base, you probably already know that Air Force source selection guidance prescribes a scheme for making risk assessments and tradeoffs during source selection.

By Eri Ottinger on Tuesday, May 30, 2000 - 11:49 am:

Janice,

I note from your e-mail address that you appear to be working for the AF. The advice that you are getting from Counsel is a somewhat garbled version of the stadard AF source selection policy.

http://www.safaq.hq.af.mil/contracting/toolkit/part15/

If this doesn't clarify the issue, I can put you in touch with one of the folks who wrote the policy.

There is more than one right answer to the question. But as long as you are working for the AF, I would recommend that you do it the Air Force way.

Keep in mind that this is merely a framework. It is up to you to fill in the details and determine where you want to put the greatest weight.

Good Luck,


Eric

By jkmcconaha on Tuesday, May 30, 2000 - 10:43 am:

FAR 15.101-1 Tradeoff Process This process permits tradeoffs amoung cost or price and non-cost factors and allows the Government to accept other than the lowest priced proposal. I have received legal advise to include in the Source Selection Plan that tradeoffs will be made for the non-factors of mission capability (technical), past performance and risk. I have not read or known of the use of the non-cost factor of risk in tradeoffs. I would like to have discussions and inputs from others that have evaluated and made tradeoffs with risk. What is the value and how do you support/document your tradeoff decision. I can understand a tradeoff for risk for R&D type efforts, but for service support why would you want anything other than a low risk.