When you use color or adjectival rating, how does one come up with factor rating from subfactor level?

For example, assume:

Subfactor 1 (SF1) is greater in importance than Subfactor 2 (SF2) , which is greater in importance than Subfactor 3 (SF3) and 4 (SF4) which are equal in weight.

B = Excellent, G = Good, Y = Satisfactory, R = Unsatisfactory

Ratings of:

KTR A: SF1: B, SF2: R, SF3: R, SF4: G
KTR B: SF1: Y, SF2: B, SF3: B, SF4: B
KTR C: SF1: G, SF2: R, SF3: B, SF4: B

What should the color rating be for each contractor?

If subfactors are equal in weight, I guess you could look at the number of colors to "guestimate" the overall ratiing. However, that method may not be consistent. When the subfactors are weighted differently, as above, it becomes extremely difficult.

The same problem arises when you get to the factor level if the factors are weighted differently.

I guess you could avoid combining the subfactors and factors. However, it will be difficult to ascertain who has the best technical score when there are many subfactors and factors that are weighted differently.

A solution taught by a source selection consultant is to use comparative analysis for each subfactor based on the weight and come up with a ranking for the Factor. Using this method, KTR A, C and B will be ranked 1st, 2nd, and 3rd, respectively. The advantage of this method is that it is consistent.

How does your organization handle subfactors and factors?

By Anonymous on Friday, February 08, 2002 - 07:12 pm:

We still use points and add them up but the SSA decision is on narrative of strenths and weaknesses not the scoring system.

By Vern Edwards on Saturday, February 09, 2002 - 11:34 am:

The difficulty associated with all adjectival and color-rating scoring systems is that there is no built-in way to systematically integrate subfactor scores and weightings or subfactor tradeoffs. If you have only two subfactors, then you can do it intuitively without too much trouble. But as you have shown, with even four subfactors and three contractors the job can be difficult, though certainly not impossible. It seems to me that the method suggested by your consultant ought to work. Any fair, reasonable and consistent method ought to be acceptable to the GAO.

By Dave Barnett on Monday, February 11, 2002 - 10:21 am:

We suport our best value source selections with narrative analysis. The method used for ranking the proposals, whether it be color coding, adjectival, or numerical is not the end all to the source selection, it is merely a tool to assist in the source selection decision.

The Defense Acquisition Deskbook has some decent sources (Army and Navy) on source selection.