Joel,

Apologies for any misunderstanding. I wasn't thinking of you when I wrote "we."

It was clear that you were comfortable with the DoD Guide.

Eric

Let me add that I don't have any problems with the DOD Guide and its approaches to collecting and evaluating past performance data. See also page seven, where distinctions between "extent" of experience and "past performance" are very clearly pointed out. Happy Sails!

By Eric Ottinger on Thursday, June 8, 2000 - 06:02 pm:

Bob,

Here is the site for the DoD Guide. The answer to your question is on Page 9.

http://www.acq.osd.mil/ar/doc/ppiguide.pdf

To answer a few question which you may or may not have intended to ask.

Q: Can this evaluation of the relevancy be judgmental?

A: Yes. Note that the Guide provides a broad rather than a restrictive definition of “relevancy.” But you should have some reasonable basis for your evaluation.

Q: Is there any way to select a small contractor who does good work, but has not done exactly the same work before?

A: Yes. The offeror may give you good reasons to think that the performance risk is low. The offeror may hire key personnel (or a subcontractor) with solid experience doing exactly the work that you have in mind.
(It isn’t unheard of for the incumbent’s key staff to jump ship and go with the challenger. In a case like this, who really has the relevant experience? The corporate entity or the team of key managers?)

Q: Do you have to have separate (sub)factors for “Experience” and “Performance?”

A: That is my personal preference, but you can evaluate both under the same factor if you are being pressed to use the minimum number of selection factors.

Joel,

I doubt we will be having the same arguments because we have at least ten times as much case law at this point. I expect we will find that the case law supports the advice in the DoD Guide.

Eric

By bob antonio on Thursday, June 8, 2000 - 11:51 am:

Ramon:

I agree. No harm was done to the protester. The offeror with the relevant past experience received credit for it.

By Ramon Jackson on Thursday, June 8, 2000 - 10:36 am:

Bob A,

Interesting reference. It confirms again that a reasonable, well defined basis for evaluating relevance and nature of past performance stands.

I also found it interesting Navy took the option not to rate the sum of the parts as relevant and poor, but rated it not relevant and neutral. That seems a sane course of action that avoids a great deal of bickering over qualitative aspects of something of questionable relevance. Anyone involved in very large, complex projects knows the sum of the parts is not the whole as many fail on integration at both technical and management levels.

It seems to me that part of Bob M's problem, along with anyone else planning such an exercise, is to carefully scope the relevance envelope so that there is clarity in instructions, procedure and finally in justification for placing an offeror into the no relevant performance and neutral rating category. Lack of thinking upon realistic relevance boundaries, rooted in project requirements and needs, and defendable in the end result tends to lead to much bickering, protest and muddy waters upon review. When selectors try to defend what is really irrelevant performance as "poor" it not only muddies those waters. It creates a more offensive outcome and pours fuel upon protest inclinations.

Ramon

By bob antonio on Thursday, June 8, 2000 - 09:29 am:

A recent decision dealing with this issue.

http://www.gao.gov/decisions/bidpro/284816.htm

By Vern Edwards on Thursday, June 8, 2000 - 09:18 am:

BobM:

What makes past performance relevant to future performance is the nature and scope of the work. In your case it seems likely that there would be significant differences in fabrication and assembly methods. The differences between manufacturing aluminium antenna towers and steel trash cans are likely to be so great that I can only wonder why you doubt that you could consider them.

You are simply wrong to think that you cannot consider differences in relevancy when evaluating past performance. It is well-established in GAO decisions that you can.

Suppose that you are going to choose the contractor on the basis of only past performance and price and that you are comparing your trash can maker to another vendor who has extensive experience making aluminium antenna towers. Suppose further that both have "excellent" reputations with their customers. Finally, suppose that the trash can vendor has proposed a lower price for the towers than the experienced antenna maker. You could consider the aluminium tower maker's excellent past performance more favorably than the trash can maker's excellent past performance because it is more relevant to your procurment, and you could award to the higher priced offeror, assuming that you consider its price to be fair and reasonable and the difference in price to be worth paying in light of the difference in past performance.

There wouldn't be anything controversial about such a decision.

By joel hoffman on Wednesday, June 7, 2000 - 05:21 pm:

At the risk of repetition, we have covered this subject previously and there will be the same arguments again from some persons on this Web Site. The following is my opinion and is not contradicted in the FAR or the "DOD Guide for Collection and Use of Past Performance Information". I don't know of any Protests, sustained for following my practice.

"Past Performance", as defined in the FAR, concerns the QUALITY aspect of a contractor's recent experience. How well did the Contractor perform on recent, relevant projects. The law and the FAR state that we must rate an offeror without any relevant PAST PERFORMANCE record "nuetrally" that is an unknown risk - neither negative or positive.

The OTHER, distinct aspect of "recent, relevant experience" is the "EXTENT" of such recent experience, which can be related to your project through corporate experience or experience by the key personnel involved in your project. You CAN make a separate assessment of "extent of recent, relevant experience" and rate the firm, accordingly. AMEN. Happy Sails! Joel

By Ramon Jackson on Wednesday, June 7, 2000 - 04:02 pm:

Whatever the convention, I think it is important to separate past performance into logical components. Not doing so can lead to muddy and nearly useless results.

First, contract performance should be separated from technical performance, whether building something or cleaning floors. They are two entirely different things and should not be confused. Take your towers. You may have the people who know more about such towers than anyone in the world. They would do an excellent job. One problem, they have an absolutely horrible business staff and can't ever seem to do a job on time in cost. Schedules mean nothing, the guys will do it when they get back from fishing, they forget to order things and stand around waiting -- but one of these days you will sure get a quality tower. In one bin and with one set of questions rate how well the offerer tends to perform on these business matters.

You also don't want to find you have the most professional, even clairvoyant, contract management team who can predict schedule and costs to an astounding degree while having a reputation for meeting cost, schedule and all those other administrative things with some sloppy and corner cutting work.

When you lump these things you may tend to obscure real problems in either. Construct your PP evaluation and analysis to drive out each so that not only can you pretty much eliminate someone truly failing on either, but can also drive out what you might expect to be the bumps in the winner's performance and be forewarned. Then you'll be in the position to address those potential bumps on day one, not after the bounce.

I wholly agree with Kennedy on the difference in trash cans and tower work. Aside from materials used the safety factors are completely different. I believe your technical staff needs to take more than a few moments to determine the envelope of what is relevant work and also be open to a provision for letting an offeror defend relevance. In some fields there can be surprising similarities that are not obvious to even those experienced in one area.

By Kennedy How on Wednesday, June 7, 2000 - 01:47 pm:

Actually, relevancy IS important; in your case, there is a big difference in welding an aluminum antenna tower and fabricating a 50gal trash can. Speaking from personal experience, welding aluminum takes a different skill than welding steel.

My feeling happens to be exactly the opposite than yours is, in this area. Without actually researching the FAR/Regulations, my feeling is that the farther away from the kind of work you are contracting for, the lower you can rate him. I might even suggest using the SIC codes as a potential guide to relevance. (Note I said potential, it's only to give you some idea as to classifying relevant experience.)

Kennedy

I got such good comprehensive answers to my last question, please indulge me by answering one more (by the way, I will ask for missing info in future).

When asking for past performance data from a vendor we ask for "relevant" past performance. In my mind, if I am contracting for antenna towers fabricated and welded using aluminum, I should be able to rate lower (or not even review) an offerer who's past performance is only in the fabrication of small trash cans and other small-sized implements out of carbon steel metal.

However, something tells me I am incorrect. If the vendor who made the trash cans always performed very well in that, then I can neither screen him out nor rate him lower because he has never worked with and welded large aluminum structures. As long as his quality is good and customers love him, he's in as far as past performance goes.

If that assumption is true, then is there a way that I can rate the relevancy of his work in order to place him properly? All other things being equal he should rate lower than an antenna tower manufacturer. I have a feeling there should be another factor called experience.

Bear in mind that most of my contracts are from 50K to 2M, commercial equipment, using SAP and the FAR 13.5 provision. I usually do not use points