Reference SAAL-PP memo, 5 Mar 01, sab.

This memo prohibits the use of numerical weighting to evaluate proposals for Army contracting offices, efffective 5 Mar 01. It states: "Evaluation factors and subfactors must be definable in readily understood qualitative terms (i.e., ajectival, colors, or other indicators, but not numbers."

Does anyone know the reason for this sudden change? Was there a GAO decision and/or court case?

By Anonymous on Wednesday, March 14, 2001 - 07:49 pm:

Nothing on the memo, but there have been some valid objections raised to numerical scores because they tend to give a sense of precision beyond what exists.

I don't see much difference between a short scale of 1-3, 1-4, or even 1-5 and excellent, good, poor; excellent, good, fair, poor; or excellent, good, fair, poor, awful and color scheme equivalents.

When people start thinking they are doing something effective with a 1-10 scale and scores of 7.3 they are fooling themselves and implying levels of detail that just don't exist. Numbers can be an aid in sorting through things. If just left standing as numbers others, if not those who generated them, may conclude they have meaning beyond reality.

By joel hoffman on Wednesday, March 14, 2001 - 09:06 pm:

I don't disagree with you Anon, regarding an individual score for a factor. Some problems I've most commonly noted with scoring are

1) instead of using score as an "indicator", supplementing the narrative evaluation, people tend to use scores as precise differentiators between proposals.

2) Too many people still want to divide the price by the point score and use this as the selection criteria ("$/pt."). This is a fairly meaningless equation.

3) Too many people assign a score to a factor or subfactor, THEN devise narrative statements to justify the score. It' supposed to be done the other way around. You first prepare the narrative evaluation, listing weaknesses, strengths, deficiencies, ambiguities, advantages to the Government, etc. - then select a point score from the scoring criteria guidelines to support the narrative. One could just as easily select adjectival ratings or colors (after preparing the narrative evaluation) for each subfactor. The score or color is still just an indicator.

To me, assigning relative weights to Areas, factors and subfactors is much more useful than scoring the individual subfactors. I can work with adjectival ratings. However, it is necessary to know which factors are more or less important. A green rating in a minor factor/subfactor is not as important as a green in a major factor or subfactor. Weighting helps me keep this in perspective,

Although Bob didn't link this, there is a proposed AFARS change out for public comment. The Corps of Engineers intends to provide comments to the draft. Happy Sails! Joel

By bob antonio on Wednesday, March 14, 2001 - 09:23 pm:

Joel:

If you know the item that I missed, let me know about it so I can add it. I may miss some. In other cases, agencies simply do not publish items in the federal register so the public is left guessing.

By joel hoffman on Wednesday, March 14, 2001 - 10:51 pm:

Bob, I'll e-mail you copies when I get to work, Thursday. One is a letter and the other is a proposed AFARS change. Happy Sails! Joel

By bob antonio on Thursday, March 15, 2001 - 09:04 am:

Joel:

Don't forget to send me the stealth Army proposal.

Thanks

By Anonymous on Thursday, March 15, 2001 - 11:50 am:

Joel points out some specific abuses of what I mentioned in general. I think perhaps the tendency we both note very possibly justifies these policies with one reservation. Though it would be almost impossible to really enforce, it would not be wise to exclude numbers as an aid.

Joel's point about a minor green and a major green and weightings illustrates this. If nothing else, assigning some numeric value for the process is almost necessary. I doubt most people really can think well using "Do three little greens cancel one or two big yellows?" and the cure could be worse than the disease.

By Kennedy How on Thursday, March 15, 2001 - 01:02 pm:

When I was writing my first source selection plan, I was advised to be careful in the area of numerical scoring. The question posed to me was; how will you reply to somebody asking why Offeror A scored a 7.3 and Offeror B scored a 7.8?

Since then, I've taken the tack that numerical scoring is a lot like scoring figure skating or diving. It's too subjective. The criteria may not fit quite right with whatever you're evaluating, so somebody gets subjective. Or something to that effect.

That first go was my one and only attempt to write a source selection plan. That was only because I got reassigned in a reorganization, but I'm pretty sure they used it during the evaluation phase.

Kennedy

By joel hoffman on Thursday, March 15, 2001 - 06:23 pm:

I've used both systems, points and adjectival (colors are a nice visual depiction of the adjectival system), and have made both work.

Our design-build team has been discussing how to lay out a complex layer of factors and subfactors, without using percentage weights. A couple of us decided a good way would be to use the % weighting method to conceptually lay out the various factors and subfactors. Then, we will ditch the % and describe the relative importance, using adjectives. The % will no longer be a consideration - it is just a structured approach to help lay out the layers of factors/subfactors, based on our goals and objectives.

After rating each proposal, build a matrix. List all the factor and subfactors in a vertical hieracy. List all the offerors, horizontally. Fill in the matrix with the ratings. Compare offerors' ratings horizontally, first. Then be sure to indicate which level of advantages/disadvantages is more important. Like Anonymous says, it gets a little dicey when there are three minor advantages from one offeror to compare with a major advantage from another, but that's why they pay us the big bucks! Hey, I embrace change (when there's a reason or need for change). Happy Sails! Joel

By joel hoffman on Friday, March 16, 2001 - 09:07 am:

I spoke with an AFARS manager, this morning. Amazingly enough (why does this surprise me?), the person said that "since this is an internal Army procedure, there will be no public comment on the AFARS rewrite". He also said the numerical weighting ban is a done deal, non-negotiable, no need to comment. I asked him why the change, and he replied that he didn't know, it was simply dictated from above.
Happy Sails! Joel

By formerfed on Friday, March 16, 2001 - 09:23 am:

One of my favorite means of evaluation doesn't involve anything other than concise narrative that summaries the relative benefits of each offer. It doesn't use points, adjectival, colors, etc. - only succinct wording. It gets the evaluators to think in terms that are the most meaningful and relevant.

To work best, requirements are stated in the highest level, purely functional terms such as Statement of Objectives. After the team brainstorms to get requirements at the level where industry is free to propose solutions on an unrestricted basis as possible, they next come up with criteria or "discriminators". The discriminators get far removed from many of the traditional factors and instead focuses on what really separates potential solutions in satisfying the intended outcome.

Once this is done and offers are presented, the team evaluates and keeps summarizing their results and thoughts into key words. The final product is a concise summary of pros and cons of each offer, and clearly shows the respective merits in narrative terms. Doing the ultimate trade-off analysis almost becomes common sense.

By joel hoffman on Friday, March 16, 2001 - 11:21 am:

I totally agree with you, Formerfed! Happy Sails! Joel