|
By
K.C. on Wednesday, September 04,
2002 - 06:56 pm:
I'm currently supporting a program office that is
considering using numerical scores in a source selection. The
contracting officer states that 1) "Numbers always lead to a
protest" and 2) "GAO doesn't like numbers used." I understand
that in general, the sole use of numbers without justification
is not considered appropriate ...
Question A) Does anyone have any empirical data regarding
Numerical Scores vs. Adjectival Ratings in protest situations?
Question B) Does anyone know of a GAO (or other Court decision)
that provides that one shall not ever use numerical scores -- or
the opposite, where GAO found use of numerical scores adequate?
By
joel hoffman on Wednesday,
September 04, 2002 - 07:53 pm:
I don't think that you will find a court or GAO
decision that says not to use points. The problems have been in
the application and execution of the rating system. Scores
should be assigned based on the narrative evaluation, not the
other way around. And scores aren't absolute discriminators,
only indicators of quality. Scores have been taken literally, as
the discriminating factor in the selection process. The
narrative strengths, weaknesses and deficiencies are the real
discriminators.
If this is an Army conducted acquisition, the AFARS, 5115.304 --
Evaluation Factors and Significant Subfactors, prohibits use of
numerical scoring or weighting the relative importance between
the factors and subfactors. If this is an acquisition by the Air
Force, they also prescribe a qualitative rating system in their
supplement, not a numerical rating system. Does this help? happy
sails! joel hoffman
By
Vern Edwards on Wednesday,
September 04, 2002 - 07:59 pm:
K.C.:
I hope that you realize that you have probably started a big
fight among us.
Okay, first, your CO was wrong when he/she said that numbers
always lead to a protest. That's nuts.
Second, it's not true that the GAO does not like numerical
scoring. What the GAO doesn't like is any scoring system that is
not backed up by documentation of evaluation findings concerning
each offeror's performance on the evaluation factors for award.
Such documentation is sometimes referred to as "narrative"
descriptions of "strengths" and "weaknesses." Here is the
closest thing you will get to a policy statement by the GAO:
"Our Office does not independently evaluate proposals, but
instead reviews the agency's evaluation to ensure that it is
reasonable and consistent with the solicitation's stated
evaluation scheme. Westinghouse Gov't and Envtl. Servs.
Co.,Inc., B-280928 et al., Dec. 4, 1998, 99-1 CPD ¶ 3 at 5.
When considering the ratings assigned by an agency to an
offeror's proposal we have consistently taken the position that
evaluation ratings, be they adjectival, numerical or color, are
merely guides for intelligent decision-making in the procurement
process. KBM Group, Inc., B-281919, B-281919.2, May 3,
1999, 99-1 CPD ¶ 118 at 11. The relevant consideration in our
review of an agency's evaluation is whether the record
demonstrates that the agency reasonably assessed the relative
merits of the proposals in accordance with the stated criteria.
Id."
In response to your questions:
A) I doubt that anyone has compiled statistics pertaining to the
number of protests sustained because of errors in numerical
scoring vs. the number sustained because of errors in adjectival
rating. However, you could compile such statistics if you were
willing to put in many, many hours of research in a law library
or at a computer.
B) I am not aware of any protest decision in which a court or
the GAO said that an agency may not ever use numerical scoring.
(Did I understand your question rightly?) However, there have
been many decisions in which a court or the GAO has found that
numerical scoring was adequate.
If you understand the reasons for using "scores" or "ratings"
and the underlying principles of scoring, then you probably
understand why a well-constructed numerical scoring system is
superior to adjectival and color rating systems. But having said
that much, I have come to accept after twenty years of teaching
source selection that most government contracting personnel do
not understand the underlying principles of scoring, are not
competent in the proper design and use of numerical scoring
systems, and therefore should not use them. Not merely because
they would lose a protest, which is actually pretty hard to do,
but because they may make a bad decision.
Finally, if you keep your evaluation plan simple enough, you
shouldn't need any type of scoring system, numerical,
adjectival, or color.
By
joel hoffman on Wednesday,
September 04, 2002 - 08:10 pm:
P.S., I know the leader of the team at Army
Acquisition, which wrote the ~April 2002 change to AFARS,
prohibiting numerical scoring. She explained that it was an
internal decision, not because of any one GAO decision. It was a
decision based on the overall problem I described above.
Many of my Corps of Engineers associates had big heartburn over
the decision and many were misusing the scoring method, as
described in my last post.
Once they realized that the narrative evaluation for a factor
drives the rating, not assigning a score, then developing a
rational to support it, they quickly adapted to the adjectival
rating systems. The most difficult aspect for most of us has
been not being able to assign a "weight" to the relative
importance of the various factors and subfactors. But we've
adapted! happy sails! joel
By
joel hoffman on Wednesday,
September 04, 2002 - 09:01 pm:
Vern, I agree with your preference for a well designed
scoring system as well as your assessment of most people's
comprehension and application of scoring systems in actual
practice. Too many think 1) a total score is a precise
distinguisher, 2) the relative weights between factors or
subfactors are precise, 3) that you assign a score, then develop
the narrative to justify the score, 4) that the difference in
scores is enough to justify the selection, without a trade-off,
etc., etc. happy sails! joel
By
Vern Edwards on Wednesday,
September 04, 2002 - 09:20 pm:
Joel:
AFARS 5115.304(b)(iv) is a perfect example of the ignorance and
incompetence problem:
"Must be qualitative. Numerical weighting (i.e., assigning
points or percentages to evaluation factors and subfactors) is
not an authorized method of expressing the relative importance
of these factors and subfactors. Evaluation factors and
subfactors must be definable in readily understood qualitative
terms (i.e., adjectival, colors, or other indicators, but not
numbers) and represent the key areas of importance to be
considered in the source selection process. The direction of
this subparagraph is not waivable, either on an individual or
class basis, as an AFARS deviation."
Note the phrase: "Numerical weighting (i.e., assigning points or
percentages to evaluation factors and subfactors)... ." When
scoring, you don't assign points to evaluation factors, you
assign them to proposals. Moreover, scoring and weighting are
different things entirely, yet the writer appears to be confused
about this ("i.e., assigning points or percentages").
Also, it's funny that the evaluation factors used to select
service contractors must be "qualitative," while FAR requires
agencies to develop "measurable" performance standards for
services.
By
K.C. on Wednesday, September 04,
2002 - 09:40 pm:
Thanks for the feedback. I support the Navy, which
has, as you likely know, mostly moved to adjectival ratings b/c
people used scores to justify the decision instead of analysis
and documentation to support the scores ... The program office
is in a frustrating environment, where the PCO and lawyers
respond with urban myths ("numbers=protest"), vs. research.
Vern, from what decision did you cut and paste the GAO "policy"
statement? Also, I'm intrigued -- what is a "simple" evaluation
plan with no rating scheme?
Again, REALLY appreciate the thoughtful, researched feedback.
By
joel hoffman on Wednesday,
September 04, 2002 - 09:47 pm:
Vern, agreed.
The language in the AFARS was a last minute revision to the
AFARS rewrite, after the draft document was sent out for review
in the Spring of 2001. When the new Army acquisition executive
came on board, he simply decreed that no more numerical weights
or rating systems would be used and that he didn't intend to
debate anyone about it or listen to any other viewpoint. It was
simply an order. happy sails! joel
By
Vern Edwards on Wednesday,
September 04, 2002 - 10:23 pm:
K.C.:
Tennier Industries, Inc., B-286706.2, 2002 CPD ¶ 75,
March 14, 2001.
By
Vern Edwards on Wednesday,
September 04, 2002 - 10:46 pm:
K.C.:
A simple plan would include the following factors:
1. offer acceptability (offeror's assent to the terms of the
RFP, evaluated on a pass/fail basis)(this factor includes the
socio-economic program evaluation factors);
2. risk (based on offeror experience and past performance); and,
3. price or cost to the government.
You make offer acceptability and risk significantly more
important than price or cost and equal to each other.
Offerors who submit unacceptable offers are eliminated from
further consideration without consideration of their experience,
past performance or price.
You then determine and summarize the experience and past
performance of each offeror who submitted an acceptable offer.
You combine your findings under the rubric of "risk". (The
better the experience and past performance, the lower the risk.)
You then make a risk/price or cost tradeoff analysis in order to
select a winner.
In order to perform a tradeoff analysis you compare the offerors
to each other in a series of pairs. If, in any pairing, one
offeror has better experience and past performance and a
lower price, then it is the better value. If one offeror has the
better experience and past performance and the higher price,
then you subjectively decide whether the lower risk is worth the
higher price. Making this decision entails deciding whether you
want to pay a higher price to avoid the risk, or take the risk
in the hope of paying a lower price. The decision will withstand
any protest as long as it is within the realm of reasonableness.
You continue in this way until you have compared all the
offerors who submitted acceptable offers and determined which is
the best value.
By
bob antonio on Thursday, September
05, 2002 - 05:26 am:
K. C.
You can find Tennier here.
http://www.wifcon.com/pd15_305factors.htm
By
Linda Koone on Thursday, September
05, 2002 - 08:07 am:
K.C.
Also being with the Navy, I thought I'd point out that the Navy
Supplement (NAPS)to the FAR/DFARS does not specifically prohibit
the use of assigning point scores to cost or price, but provides
guidance that generally discourages it at 5215.305(a)(1), which
states, in part . . .
"(1) Cost or price evaluation. Methods of evaluation which
assign a point score to cost or price and combine it with point
scores for other evaluation factors generally should not be
used. Point scores can be helpful in summarizing subjective
evaluation of technical and other factors, but are not needed in
evaluating cost or price and tend to obscure the tradeoff
between cost/price and other factors, rather than clarifying it.
If point scoring of cost/price is utilized, it should be
demonstrated that the value of a cost/price point is comparable,
in value to the Government, to the value of a non-cost/price
point."
By
formerfed on Thursday, September
05, 2002 - 08:45 am:
K.C.,
One further GAO quote that's relevant - "Numerical point scores,
while useful as guides to decision-making, do not of themselves
supply the basis and reason for the award decision."
My old agency had much success on some procurements using an
approach we termed "succinct narrative." It was patterned after
NASA's mid-range streamlined approach, which I believe is in
their regulations.
It requires the development of unique discriminators specific
for each individual procurement. You need to forget about all
the routine factors commonly used for most procurements and get
at the question "what are the distinguishing attributes among
offeror products or services that impact how successful the
product or service is deployed?" Then proposal evaluation
consists of writing concise wording that describes how each
offeror meets the factors. There are no point scores, adjectival
ratings, or summary scores. It's just a summary wording of the
benefits, weaknesses, and deficiencies in very brief terms.
The good thing about this is the award selection decision is
often more readily apparent. It gets the evaluators to focus on
what the procurement is all about and the purpose te product or
service is used for. Finally it really goes to the core of what
separates one offer from another.
By
Anonymous on Thursday, September
05, 2002 - 09:36 am:
Vern,
I like your 3 step/factor evaluation scheme. Coincidentally, it
is similar to the one I used when I proposed to my wife, except
in item 2 where risk was directly proportional to the amount of
prior experience.
Anonymous (for obvious reasons)
By
K.C. on Thursday, September 05,
2002 - 10:05 am:
Thanks again, everyone, for the great advice, links,
and citations.
Vern -- Interesting approach for evaluation (acceptability,
risk, and cost). I'd love to give it a whirl. I am of the
general opinion that a winning proposal has nothing to do with
successful contract execution (with the possible exception of
incumbent sources). At the end of the day, if the Govt
incorporates technical approach etc, undoubtedly it locks in a
solution that was developed by industry without enough
information ... and if they don't incorporate the technical
approach into the contract, why bother listening to it. The
other problem is that in many cases the Government does not
maintain the technical expertise to develop a solid requirements
package ... and therefore how will evaluators know when a
proposal is sound? Rhetorical questions ...
Have you used this simplified approach, or seen it demonstrated
with success? On a major system?
By
Vern Edwards on Friday, September
06, 2002 - 09:03 am:
K.C.:
The approach, with variations, has been used in a number of
source selections. It was used last year by the U.S. Customs
Service to award the multi-billion dollar Customs Modernization
contract. They added "understanding of the requirement" and "key
personnel" as subfactors under risk. They evaluated
"understanding" based on oral presentations. I helped them write
the source selection plan and the RFP proposal preparation
instructions and evaluation factors for award.
By
Vern Edwards on Friday, September
06, 2002 - 09:25 am:
K.C.:
The Customs RFP is still on the Web at
http://www.customs.ustreas.gov/contract/modern/rfp/rfp.pdf.
Read Section M, which you will find at the very end of the RFP.
The evaluation scheme is a little more complicated than the
basic scheme that I described above, but it included only four
main evaluation factors -- offer acceptability, offeror
capability, risk and price -- which is very simple for such a
large procurement. It also included the "contingency procedure"
that I have advocated in past postings.
The oral presentations were longer than I would have liked --
4.5 to 5 hours -- but shorter than they had originally wanted.
They included a one hour formal presentation and allowed 3 to 4
hours for a Q&A session. This was understandable in light of
what was at stake for them. It's a huge task order contract to
develop an enterprise-wide IT system for the agency. The
contract performance period, with options, continues through the
year 2011.
By
K.C. on Friday, September 06, 2002
- 11:42 am:
Vern,
Thanks for the link. Would be interesting to compare this source
selection experience with other enterprise IT system source
selections (Navy Marine Corps Intranet, SOCOM's Enterprise
Information Technology Contract, Army Infostructure, etc etc).
I like the contingency procedure -- puts some teeth into the
statement that's always there about "reserving the right" to
limit for efficiency.
Regarding terms and conditions ... Wonder how the
"acceptability" part would play out in a FAR 12 acquisition --
where negotiation of terms and conditions is generally
acceptable (and expected)?
Also -- what was the point of the professional employees
compensation plan??? To see if customs could rely on key
personnel availability throughout the contract?
Thanks again for the link.
By
Vern Edwards on Friday, September
06, 2002 - 12:05 pm:
K.C.:
Yes, the acceptability criterion could work in a commercial
items procurement, although you might not want to use it if
you're buying supplies.
Professional employee compensation is a mandatory evaluation
factor in certain services acquisitions. See FAR § 22.1103.
Customs was very concerned that the contractor maintain a high
quality IT staff.
|
