By joel hoffman on Tuesday, July 30, 2002 - 12:03 pm:

alexreb, can you advise which organization is conducting the A-E selection? Have you checked their SOP's? happy sails! joel

By alexreb on Tuesday, July 30, 2002 - 01:16 pm:

Joel--

Our engineering section (design and construction) is conducting the selection. Services being contracted in this instance, include surveying (pre-design), design, and QA. Voting board members are all engineers, with specific experience in services being sought. Board chairman is registered engineer. We use one board for both pre-selection and selection functions. The board uses a quantitative evaluation method. Their SOP is sound.

By joel hoffman on Tuesday, July 30, 2002 - 02:35 pm:

Alex, I'll pass your question to our agency A-E contracts proponent; he's out till tomorrow. Please clarify, are you asking "Should firms be asked..." or "Can firms be asked..."?

You don't have to ask firms to submit any more than SF 254/255's. If you mean "Can we ask for more...", I will ask the Guru. Thanks in advance for the clarification. happy sails! joel

By alexreb on Tuesday, July 30, 2002 - 02:54 pm:

Can we ask for more? If we can, would it be beneficial to do so?

Thanks Joel.

alexreb

By joel hoffman on Tuesday, July 30, 2002 - 03:56 pm:

Got it. I e-mailed him your questions. happy sails! joel

By alexreb on Tuesday, July 30, 2002 - 06:42 pm:

Joel,

While you're waiting on that email could you help clarify something I read in Corps Engineering Pamphlet 715-1-7 "Architect-Engineer Contracting"? Paragraph 3-8.d. discusses two A&E evaluation methods, qualitative and quantitative. Under qualitative the board ranks the firms by coming to a consensus through discussions and under quantitative the board ranks the firms through a numerical scoring system.

My question deals with the following statement the pamphlet makes about the quantitative method..."ranking of the firms according to the total scores must represent a consensus; individual scores must not be simply averaged to produce a census score." Could you describe what a board does to reach consensus after each member has individually assigned different scores to each firm?

By joel hoffman on Tuesday, July 30, 2002 - 08:46 pm:

Sure, Alex.

First, let me point out that the Army no longer allows numerical scoring of proposals. The Corps no longer uses that method for A-E, services or construction source selection ratings. However, here is how I think consensus scoring should work and how I used to run my boards.

A scoring system is established, based on an evaluation scheme. Scoring bands are established, using narrative descriptions for each band.

Each member of the selection board evalauates a proposal, writing down a narrative for each evaluation factor. The narrative is that member's initial impression of the strengths, weaknesses, deficiencies or necessary clarifications under that factor. Members should not individually score the factors at this time, as only the consensus numerical rating will be used.

After individually evaluating a proposal or all of the proposals, depending upon board preference, the board meets to discuss the proposals and to develop a consensus rating for each, against the scoring scheme.

Each factor is reviewed, and a narrative, as described above is developed, in a consensus. Once the narrative is developed, the scoring band is chosen to match the narrative. Then the group decides upon an appropriate score from the band - high, low or mid range, depending upon the strength of the proposal. Actually, it works very well. We never had a problem reaching a consensus. The scores simply "fall out", based on the narrative. The narrative is the primary rating, with scores simply reflecting the narrative. Note that scores should only be used as an indicator.

After individual proposals are rated, all proposals are compared against each other, first in the competitive range determination (if discussions are necessary) and later in the trade-off analysis. We use both the score (an indicator) and the relative advantages and disadvantages - the primary differentiator between technical proposals (as well as price differences) to make the comparison.

Here is how it shouldn't be done - and a primary reason why Army banned numerical scoring. Many organizations, including mine, when I first started participating in source selections, used this method. Each member would "score" the proposal and write down (hopefully) their justification. When the group met, everyone would rattle off a score and the group would either add up the scores, average the individual scores, or agree on a consensus score - oh, do we need to write down why??? The score became the primary focus of the rating, the write-up, and the trade-off analysis. When it came time to debrief, nobody could explain why the proposal was scored a certain number of points - because the wrong thing ("the score")was focused on.

Does the make sense to you? It sounds complicated, but really isn't, if done right. Scores should only reflect a strong narrative. Thus, close score differences are probably meaningless, but large differences easily separate proposals. happy sails! joel

By joel hoffman on Wednesday, July 31, 2002 - 08:50 am:

Alexreb, here is the response from my agency's A-E contracting proponent:

"I guess we could get limited work samples, to the extent that they illustrated compliance with one of the selection criteria. But I would only request it from the most highly qualified firms (typically top 3)."

Alexreb, I would say "yes", there could be a benefit to the Government in asking for a sample design or report, similar to the instant project, so that you could sample the firm's products. However, without a detailed review of a design or report, you may not gather much other than a first impression of the firm's presentation skills.

The FAR does allow design competitions, under limited circumstances. However, this would normally be limited to some monumental or highly prominent architectural competition.

The key to remember is that design firms already have high overheads and limited budgets for business development and proposal preparation. We don't want them to incur high expenses to compete, unless absolutely necessary. There is a corollary - as the stakes rise, the possibility of protest also rises. Firms are sometimes content to obtain their "share" of business and not fight non-selection. But, if you require them to make a significant investment to compete, the stakes go up. happy sails! joel

By alexreb on Wednesday, July 31, 2002 - 10:21 am:

Joel, tremendous thanks for your responses. You've really clarified some fuzzy areas for me. Anyone else have any comments?

Alexreb