Wifcon.Forum Archives

Evaluate against standards in source selection

By Vern Edwards on Thursday, February 06, 2003 - 12:56 am:

Did the professor answer the question? (Do you understand the professor's answer?) If not, can you?

Where is requirement to evaluate against standards in source selection?

Posted to Acquisition Policy on 1/9/2003 by P Jacobs

The Scenario:

Government client is preparing for major source selection. We are helping to develop acquisition strategy. The question came up as to why proposals must be evaluated against fixed standards vice against each other. I have been unable to find a FAR or statute that requires this, but know from experience all the reasons why it is good practice.

The Question:

Is it a regulation or law that DoD proposals must be evaluated against a standard vice against each other? If so, what law, regulation, policy document?

The Answer:

FAR Part 15 directs the contracting officer in preparing the solicitation, and the various provisions included in the solicitation as prescribed by FAR 15 guide the contractor. These provisions describe the procedures (FAR 52.215-1) for the evaluation of the contractor’s proposal when using tradeoff analysis.

Rationale: For those who may not be familiar, the underlying reason for doing it this way is that not only do we want to know who has the best proposal, but how well does it meet our requirements, and if it isn’t “perfect” where does it fall short, and by how much. This allows us to do a proper “tradeoff analysis” when we are trying to decide on what we are willing to give up if a technically perfect proposal is not submitted. Sometimes a perfect proposal is submitted; however its price may exceed what we are willing to pay.

For example a Mercedes may have everything you want in a car, but you are willing to tradeoff a few horsepower in the engine, for a saving of $20,000 by buying a Ford Mustang instead. You did a tradeoff of a few horsepower for $20,000, even though the Mercedes meets all of your technical criteria and the Mustang didn’t.

This is why we first establish the whether Cost is (1) approximately equal (2) significantly more (3) significantly less important than Non-Price (Technical) factors. Then we list the Non-Price factors, and their relative order of importance. We rate the proposals against the factors to see how “perfectly” they meet our requirements. Then if no “perfect” proposal is submitted at the lowest price (not likely) we must do our tradeoff analysis. We look at the best proposal, and its price, then comparing it to other proposals that may be weaker in the non-price (technical) area, but at lower prices. We then make the buying decision.

So to go back to the car analogy, we were willing to “tradeoff” cost savings of $20,000 for the difference in horsepower. Given the same criterion reasonable people can make different decisions, this is what makes source selection so difficult, and why they require so much documentation of the rationale to ensure it is in accordance with the provisions of the solicitation.

By joel hoffman on Thursday, February 06, 2003 - 09:15 am:

The question was very good. The answer didn't answer the question.

I don't think that there is any FAR or DFARS prohibition against using evaluation criteria, which directly compare proposals. There might be agency guidance - I don't know which agency the questioner belongs to.

FAR 15.304 (b)(2) even states that factors must - "support meaningful comparison and discrimination between and among competiting proposals."

As a practical matter, the criteria for most factors should include a description of or a reference to the minimum acceptable level, because an offer should generally conform to the Government's minimum quality criteria. The Government should first determine whether the offer meets the minimum Government criteria.

However, why can't certain criteria use direct comparisons? We have directly compared proposed construction contract durations against each other. happy sails! joel hoffman

By Harley Hartley on Thursday, February 06, 2003 - 09:16 am:

I don't understand this answer--particularly the part that says it would be OK to buy the Mustang even though it did not meet all technical criteria. I think the answer to the question can be found at FAR subpart 15.3.

By Eric Ottinger on Thursday, February 06, 2003 - 10:20 am:

The professor is utterly oblivious to the distinction between criteria and standards.

Eric

By Linda Koone on Thursday, February 06, 2003 - 02:28 pm:

As convoluted as the professor's answer is, the most amusing part of it is that in his/her example, (if you can follow it), the professor does exactly what he/she claims can't be done by comparing the Mercedes to the Mustang and not to the 'standard' in the solicitation!

By Eric Ottinger on Thursday, February 06, 2003 - 02:45 pm:

Linda,

I think the Professor is saying that CTTO exploded the myth that it isn't proper to compare offers directly with each other.

Eric

By Linda Koone on Friday, February 07, 2003 - 02:55 pm:

Eric:

Don't know what CTTO stands for. Sorry for the ignorance.

I read the question and answer again (more carefully) and still don't know what the Professor is actually saying.

However, getting to the question, "Is it a regulation or law that DoD proposals must be evaluated against a standard vice against each other? If so, what law, regulation, policy document?", the Professor should have directed the questioner to FAR Parts 7 and 11 to help understand the need for the Government to state its minimum requirement. Then a discussion on how to evaluate offers against the stated requirement and then against each other should have followed.

It's a terrible answer. That's the bottom line. I feel sorry for the person who asked.

By joel hoffman on Friday, February 07, 2003 - 03:35 pm:

By joel hoffman on Friday, February 07, 2003 - 03:39 pm:

I couldn't find a FAR or DFARS reference which prohibits directly using a comparative rating when evaluating proposals. There may be one in the questioner's agency supplements or policies. Several agencies have "policy guidance" regarding evaluating proposals against criteria, not against each other.

As a practical matter, the RFP usually should include some standard to determine minimum acceptablity of a proposal for most factors. As Linda said, the RFP should state the Government's minimum needs (however, that information is usually stated in the requirements portion of the RFP, rather than in the section that describes the proposal evaluation criteria). happy sails! joel

By Eric Ottinger on Friday, February 07, 2003 - 03:41 pm:

Linda,

Cost/Technical Trade-Off.

I struck me that my post above might ruffle some feathers.

I advise evaluators to just rate and rank, taking each proposal on its own merits. Let the SSA decide who the winner should be.

Of course the SSA compares proposals to other proposals.

The concept of a standard is explained in an attachment to the old AF source selection policy. Even though this is not the current policy, I find the professor's lack of awareness stunning.

Let's say that you and I are Olympic skating judges. How many triple jumps are required to score a 5.5. I think at least three. What do you think.

If evaluators come up with radically different ratings, it is usually because they haven't reached some understanding on the standard to be applied.

Some people think standards are hidden selection factors. They aren't. The old AF policy contemplated that standards should be quantitative and minimum. Quantitative standards are nice, but a lot of standards are qualitative. And we don't really want to buy minimum.

All and all a simple concept mucked up by dogmatic amateurs.

Hope this helps,

Eric

By Eric Ottinger on Friday, February 07, 2003 - 03:45 pm:

Joel,

Sounds like we are on the same sheet of music. The best reference is an old AF Appendix AA or BB.

Eric

By Vern Edwards on Friday, February 07, 2003 - 08:05 pm:

"The Question: Is it a regulation or law that DoD proposals must be evaluated against a standard vice against each other? If so, what law, regulation, policy document?"

In order to answer this question one must be clear about what one means by "evaluate." Ordinarily, there are two steps in every source selection. The first step is to determine how well each proposal performed on the evaluation factors; it is in this step that the evaluators assign ratings or scores. The ratings or scores reflect the value of each proposal in terms of the evaluation factors. Thus, one could say that the first step is the rating or scoring, or evaluation step.

The second step is to determine, based on the ratings or scores assigned in the first step, how well the proposals compare to each other and which is the best value; this is the ranking step. If the govenment has established minimum requirements for the evaluation factors, then direct comparisons should not be used in the performance of the first step, for what I hope are obvious reasons. However, just as obviously, every source selection entails comparison of the proposals to each other in order to determine which is the best value.

The rule to which the questioner refers did not prohibit direct comparisons, but regulated who performed which of the above steps. It appeared in DOD Directive 4105.62, Selection of Contractual Sources for Major Defense Systems, dated September 9, 1985, now cancelled. (An 1965 version was entitled, Proposal Evaluation and Source Selection.) That directive said, in section E.4., Proposal Evaluation, paragraph a:

"Evaluation criteria are used to make an integrated assessment of each offeror's ability to satisfy the requirements of the solicitation. Proposals are evaluated within these criteria. The SSEB [Source Selection Evaluation Board] does not evaluate the relative merits of one proposal as compared to another. The SSEB individually evaluates proposals against the requirements of the solicitation. Only the SSA [Source Selection Authority] and, if requested, the SSAC [Source Selection Advisory Council] will apply judgment regarding relative merits."

In the source selection organization structure prescribed by the directive, the lowest level evaluators comprised the SSEB. The SSAC was a group of senior officials, usually colonels, GS-14s and above. Thus, the rule did not prohibit direct comparisons, but prohibited them by the SSEB during the first, or evaluation, step in source selection.

Air Force source selection regulations long prohibited direct comparisons by the lowest level evaluators during the first step in source selection. For example, Air Force FAR Supplement Appendix AA, Formal Source Selection for Major Acquisitions, now cancelled, said, in paragraph AA-206:

"The SSEB conducts its evaluation by measuring each proposal against objective standards established at the lowest level of [evaluation factor] subdivision. The SSEB shall not compare proposals against each other."

Italics in original. The prohibition was repeated verbatim in paragraph AA-303 and in AFARS Appendix BB, Source Selection Procedures for Other Than Major Acquisitions, also cancelled.

Some old source selection regulations, e.g., AFARS Appendices AA and BB, required that evaluators develop evaluation standards representing the minimum requirement applicable to each evaluation factor -- the level a proposal had to meet in order to be acceptable. Proposals were to be evaluated by comparing them to these standards. FAR and DFARS do not mention standards and I don't think any of the current handbooks or manuals expressly require the development of evaluation standards, although I may be wrong. Arguably, if the government does not have any minimum requirement with respect to an evaluation factor, but just wants to get the best proposal that it can, the evaluators could skip the rating or scoring step and proceed directly to the ranking step based on the information in the proposals.

So the rule to which the questioner refers did not prohibit direct comparisons, per se, which wouldn't have made any sense, but prohibited direct comparisons during the rating or scoring step of source selection by the lowest level evaluators in the source selection organization. It reserved the task of ranking the proposals from best to worst to more senior officials. I don't think the rule survives in any existing publication, but it is still good practice not to rate or score by direct comparison.